Senior Data Engineer - Azure Data Factory

About This Role :

We are seeking a talented and experienced Senior Data Engineer to join our data engineering team. You will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure on cloud platforms. This role requires expertise in modern data engineering tools and a strong foundation in data transformation and optimization.

Key Responsibilities :

- Design and develop robust, scalable data pipelines using Azure Data Factory (ADF).

- Build and optimize data processing workflows using Databricks and Apache Spark.

- Implement declarative pipeline solutions using Lakehouse architecture and best practices.

- Write and optimize complex SQL queries for data transformation and analysis.

- Develop PySpark applications for large-scale data processing.

- Collaborate with data scientists, analysts, and stakeholders to understand data requirements.

- Ensure data quality, security, and compliance across all pipelines.

- Monitor, troubleshoot, and optimize data pipeline performance.

- Document technical solutions and maintain best practices documentation.

- Mentor junior team members and contribute to team knowledge sharing.

Must-Haves :

- 5+ years of hands-on experience in Azure Data Engineering.

- Strong expertise in Azure Data Factory (ADF) for building and managing ETL/ELT pipelines.

- Extensive experience with Azure Databricks, including Apache Spark and PySpark.

- Hands-on experience with Lakeflow Declarative Pipelines.

- Strong proficiency in SQL for complex data transformation and optimization.

- Experience designing and implementing scalable data pipelines and large-scale data processing solutions.

- Solid understanding of data warehousing, data modeling, and performance tuning.

- Experience working with enterprise-scale data platforms in Azure Cloud.