Data Engineer
Posted Updated
Summary
Data engineer (6-14 years) in Pune designing and maintaining scalable batch and streaming data pipelines using Python and Apache Spark, operationalizing workloads on Databricks and building data solutions across Google Cloud Platform and Microsoft Azure, with data quality and pipeline recovery responsibilities.
Key Skills: Python, GCP, Spark, Databricks, Azure, Bigquery
Roles and Responsibilities:
- Design, develop, and maintain scalable batch and streaming data pipelines using Apache Spark and Python.
- Build and operationalize data engineering workloads on Databricks across cloud environments.
- Develop data solutions using services and platform capabilities across Google Cloud Platform and Microsoft Azure.
- Implement data quality controls, transformation logic, error handling, and pipeline recovery mechanisms.
- Collaborate with data analysts, architects, and application teams to deliver governed and production-ready datasets.
Skills Required:
- 6-14 years of relevant experience in data engineering and cloud data platforms.
- Strong professional experience with Apache Spark and Python for large-scale data processing.
- Hands-on experience designing and implementing solutions on Databricks.
- Practical experience with Google Cloud Platform and Microsoft Azure data environments.
- Proficiency in developing, testing, troubleshooting, and supporting production data pipelines and integrations.
Good to Have:
- Experience with Google BigQuery for analytical data warehousing and query development.
Education: B.Tech or equivalent technical degree.