Lead Data Engineer
Posted 3
views
Summary
Lead Data Engineer who designs, builds, deploys, and maintains large-scale data pipelines using PySpark on Azure Databricks, collaborating with cross-functional teams and ensuring the scalability, reliability, and performance of the data architecture. Core stack is Azure (Data Factory, Synapse, Databricks) with big data/ETL troubleshooting.
Roles and Responsibilities :
- Design, develop, test, deploy and maintain large-scale data pipelines using PySpark on Azure Data Bricks.
- Collaborate with cross-functional teams to gather requirements and deliver high-quality data solutions.
- Ensure scalability, reliability, security and performance of the data architecture by implementing best practices in design and implementation.
- Troubleshoot complex issues related to big data processing, ETL processes, and data quality.
Job Requirements :
- 10-14 years of experience in a similar role as Lead Data Engineer or equivalent position.
- Strong expertise in Microsoft Azure platform including Data Factory (ADF), Synapse Analytics (SQL DBA), Databricks etc. .
- Proficiency in working with PySpark on Azure Data Bricks for building scalable data pipelines.