Azure Data Engineer (Databricks)
Posted 8
views
Summary
Designs and develops scalable data pipelines in Spark SQL/PySpark on Azure Databricks, orchestrating ETL with Azure Data Factory and maintaining a Lakehouse on ADLS. Day-to-day includes pipeline monitoring and root-cause fixes, DevOps collaboration for production deploys, and working with Data Science and BI teams at an MNC.
We are hiring for MNC company
Role & responsibilities
- Design and develop scalable data pipelines using Spark SQL and PySpark on Azure Databricks
- Build and orchestrate ETL workflows using Azure Data Factory (ADF)
- Develop and maintain a modern Lakehouse architecture leveraging ADLS and Databricks
- Perform data preparation tasks including data cleaning, normalization, deduplication, and data type transformations
- Collaborate with DevOps teams to deploy and manage solutions in production environments
- Monitor data pipelines, identify issues, and implement corrective actions, including root cause analysis and resolution
- Actively contribute as part of the global Analytics team, delivering data-driven insights and solutions
- Collaborate with Data Science and Business Intelligence teams to share best practices and drive innovation
- Lead data engineering projects and support cross-functional initiatives led by other team members
- Apply change management practices, including documentation, communication, and training, to support system upgrades and data migrations