Azure Data Engineer
Summary
Designs and maintains scalable data pipelines on Azure using Databricks, PySpark, and Azure Data Factory to ensure reliable, high-quality data processing and delivery.
We are looking for an experienced Azure Data Engineer with strong expertise in Azure cloud data platforms, Databricks, PySpark, and Azure Data Factory. The ideal candidate should have hands-on experience in building, testing, validating, and optimizing large-scale data pipelines while ensuring data quality, performance, and reliability.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Azure Data Factory (ADF) and Azure Databricks.
- Develop ETL/ELT solutions using PySpark and SQL.
- Perform source-to-target data validation, reconciliation, and data quality checks.
- Build and optimize data ingestion pipelines into Azure Data Lake Storage (ADLS).
- Develop automated data validation scripts using Python/PySpark.
- Validate incremental and full-load data processing.
- Perform schema validation, duplicate checks, null checks, and aggregation testing.
- Monitor and optimize pipeline performance to meet SLA requirements.
- Troubleshoot and resolve ETL and data pipeline issues.
- Collaborate with business stakeholders, data analysts, and development teams.
- Maintain CI/CD practices using Azure DevOps and Git.
Required Skills
- Azure Databricks
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS Gen2)
- PySpark
- Python
- Advanced SQL
- ETL Development
- Data Pipeline Development
- Data Validation & Data Quality
- Source-to-Target Reconciliation
- Data Migration
- Performance Tuning
- Azure DevOps
- Git
Preferred Skills
- Delta Lake
- SSIS
- SQL Server
- Azure Synapse Analytics
- RBAC & Data Security
- Test Automation using Python/PySpark
Qualifications
- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
- Microsoft Certified: Azure Data Engineer Associate (Preferred)
- Databricks Certified Data Engineer Associate (Preferred)
Nice to Have
- Experience with Azure cloud migration projects.
- Experience working in Banking, Logistics, Retail, or Financial Services domains.
- Strong analytical, troubleshooting, and communication skills.