Azure Databricks Engineer - Python/PySpark

Key Responsibilities :

- Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.

- Build and optimize large-scale data processing solutions using Python and Spark.

- Develop robust ETL/ELT pipelines for batch and near-real-time data processing.

- Work with Azure Data Lake Storage (ADLS) and other Azure data services.

- Implement data transformation, cleansing, validation, and enrichment processes.

- Optimize Spark jobs, Databricks notebooks, clusters, and SQL queries for performance and cost efficiency.

- Develop reusable and maintainable data engineering frameworks and components.

- Implement data quality, governance, security, and monitoring practices.

- Work with structured and unstructured datasets across multiple data sources.

- Collaborate with solution architects and engineering teams to translate business requirements into technical solutions.

- Troubleshoot production issues and perform root-cause analysis.

- Implement CI/CD processes for Databricks code and data pipelines.

- Follow engineering best practices around version control, testing, documentation, and deployment.

- Mentor junior and mid-level data engineers and contribute to technical decision-making.

Required Skills & Experience :

- 8+ years of experience in Data Engineering, Big Data, or related technologies.

- Strong hands-on experience with Azure Databricks.

- Excellent knowledge of PySpark and Apache Spark.

- Strong programming experience in Python.

- Strong experience developing ETL/ELT data pipelines.

- Hands-on experience with Azure Data Lake Storage (ADLS Gen2).

- Good knowledge of Azure data services and cloud architecture.

- Strong SQL skills and experience working with relational and analytical databases.

- Experience with Delta Lake and Delta Tables.

- Strong understanding of data modeling, data warehousing, and distributed data processing.

- Experience with performance tuning and optimization of Spark/Databricks workloads.