Data Engineer
Summary
Experienced data engineer (5-8 yrs) who designs, builds, and optimizes scalable ETL/ELT pipelines and distributed batch processing on Azure. Day to day involves Spark and Scala/Python development, Databricks notebooks and jobs, Airflow orchestration, Microsoft Fabric workloads, SQL tuning, and ADLS-based data storage.
Job Summary
We are looking for an experienced Data Engineer with strong expertise in Apache Spark, Scala, Python, Databricks, Apache Airflow, Microsoft Fabric, SQL, and Azure Data Lake Storage (ADLS).
The ideal candidate will be responsible for designing, developing, optimizing, and maintaining scalable data engineering solutions across cloud and big-data platforms.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines using Apache Spark, Python, and Scala
- Develop and optimize Spark-based batch and distributed data processing applications
- Build and manage data pipelines and workflows using Apache Airflow
- Develop data engineering solutions using Azure Databricks, including notebooks, jobs, workflows, and Delta Lake
- Work with Microsoft Fabric for data ingestion, transformation, orchestration, and analytics workloads
- Design and manage data storage solutions using Azure Data Lake Storage (ADLS/ADLS Gen2)
- Write complex and optimized SQL queries for data transformation, validation, and analysis
- Implement data quality, validation, monitoring, and error‑handling mechanisms
- Optimize Spark jobs, SQL queries, and data pipelines for performance and scalability
- Collaborate with architects, analysts, developers, and business stakeholders to understand data requirements
- Implement CI/CD and follow development, testing, deployment, and production‑support best practices
- Troubleshoot pipeline failures and resolve data‑processing and performance issues
Required Skills
- Strong hands‑on experience with Apache Spark
- Strong programming experience in Scala and/or Python
- Hands‑on experience with Apache Airflow and workflow orchestration
- Strong experience with Azure Databricks
- Hands‑on experience with Microsoft Fabric
- Strong SQL development and query optimization skills
- Experience with Azure Data Lake Storage (ADLS/ADLS Gen2)
- Good understanding of ETL/ELT concepts and data engineering principles
- Experience working with large‑scale distributed data processing environments
- Good understanding of data pipeline architecture, data quality, and performance optimization
Good to Have
- Experience with Delta Lake and Medallion Architecture
- Experience with Microsoft Fabric Data Factory, Lakehouse, and Warehouse
- Azure cloud services and data engineering ecosystem
- CI/CD using Azure DevOps or GitHub
- Experience with Kafka or other streaming technologies
- Knowledge of data governance and security practices
- Experience5 to 8 years of overall experience in Data Engineering / Big Data Engineering, with strong hands‑on experience in the required technologies
Education
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field
Location - Pan India-Bengaluru,Hyderabad,Delhi / NCR,Chennai,Pune,Kolkata,Ahmedabad,Mumbai