Senior Data Engineer
Posted Updated
Summary
Senior Data Engineer building Python ETL/data pipelines for scraping and processing, working with Azure Databricks, Apache Airflow, Postgres SQL, AWS S3 and Azure DevOps CI/CD. Requires around 8 years of Python scripting experience; energy trading domain experience is preferred.
Job Description:
Must have skills:-Python, Postgres SQL, Azure Databricks, AWS(S3), Git, Azure DevOps CICD, Apache Airflow
-Preferred Domain: Energy trading
8+ years of experience in python scripting.
Experience with python packages such as NumPy, pandas, polars beautiful soup, Selenium, Requests etc.
Experience in building native python ETL pipelines for scraping/processing.
Good to have experience in processing big data.
Proficient in SQL programming, Postgres SQL. Spark knowledge is a good to have
Knowledge on DevOps CI/CD like Azure Pipelines, Jenkins, Git.
Experience working with AWS services(S3, lambda,etc) and Azure Databricks.
Have experience in delivering project with Agile and Scrum methodology.
Able to co-ordinate with Teams across multiple locations and time zones
Strong interpersonal and communication skills with an ability to lead a team and keep them motivated.
Python, Postgres SQL, Azure Databricks, AWS(S3), Git, Azure DevOps CICD, Apache Airflow