Lead Databricks Engineer - ETL/PySpark

Job Summary :

We are looking for an experienced Lead Databricks Engineer with strong expertise in Azure Databricks, PySpark, Python, SQL, Azure Data Factory (ADF), and Delta Lake to develop and optimize scalable data engineering solutions.

Key Responsibilities :

- Design, develop, and maintain ETL/ELT pipelines using Azure Databricks, PySpark, Python, and ADF.

- Build and optimize scalable data processing solutions using Delta Lake.

- Perform SQL query optimization, Spark performance tuning, and pipeline troubleshooting.

- Develop batch and real-time data processing pipelines.

- Lead technical design discussions and contribute to data architecture.

- Implement CI/CD, DevOps, and cloud migration best practices.

- Ensure data quality, security, and reliability.

- Mentor engineers and collaborate with analytics, BI, and business teams.

Mandatory Skills :

- Azure Databricks

- PySpark and Python

- SQL and query optimization

- Azure Data Factory (ADF)

- Delta Lake

- ETL/ELT pipeline development

- Performance tuning and data architecture

- Technical leadership