Azure Data Engineer
Posted Updated
Summary
Designs and maintains batch and near-real-time data pipelines using Apache Spark, Python, and Databricks, building scalable data engineering solutions across Azure and GCP and troubleshooting production pipeline issues. Requires 6-14 years of relevant data engineering experience; Kubernetes is a plus.
Key Skills: Databricks, Python, Spark, Kubernetes
Roles and Responsibilities:
- Design, develop, and maintain batch and near-real-time data pipelines using Apache Spark, Python, and Databricks.
- Build scalable data engineering solutions across Microsoft Azure and Google Cloud Platform environments.
- Develop data transformations, ingestion workflows, and curated datasets for reporting and analytical workloads.
- Troubleshoot data pipeline failures, improve reliability, and resolve data quality and processing issues.
- Collaborate with data architects, analytics teams, and application engineers to deliver production-grade data platforms.
Skills Required:
- 6-14 years of relevant data engineering experience.
- Strong hands-on experience with Apache Spark and Python for distributed data processing.
- Practical experience developing solutions using Databricks.
- Experience designing cloud-based data engineering solutions on Azure or Google Cloud Platform.
- Ability to develop scalable data pipelines, integrations, transformations, and analytical data stores.
Good to Have:
- Experience with Kubernetes and containerized data workloads.
Education: B.Tech, M.Tech, BCA, or B.Sc. in Computer Science, Information Technology, Engineering, or a related discipline.