Lead Data Engineer
Summary
Lead Data Engineer building scalable batch, streaming, and near-real-time data platforms on Microsoft Azure using Python, PySpark, SQL, and Kubernetes/AKS with Medallion Architecture.
Role Overview
We are looking for a Lead Data Engineer with strong experience building scalable batch, near-real-time, and streaming data platforms on Microsoft Azure.
The role requires hands-on expertise in Python, PySpark, advanced SQL, Azure data services, Medallion Architecture, and deploying Apache Spark workloads on Kubernetes or Azure Kubernetes Service (AKS).
.Key Responsibilities
- Design and build batch, near-real-time, and streaming data pipelines on Azure.
- Develop Bronze, Silver, and Gold data layers using Medallion Architecture.
- Build and deploy containerized PySpark workloads on Kubernetes or AKS.
- Configure Spark drivers, executors, CPU, memory, scaling, dependencies, and storage access.
- Integrate data from REST APIs, SFTP, databases, files, enterprise systems, and Azure Event Hubs.
- Develop complex transformation, cleansing, enrichment, reconciliation, and validation workflows.
- Implement incremental loads, CDC, watermarking, deduplication, schema evolution, retries, and recovery.
- Optimize Spark jobs, partitioning, shuffles, joins, file sizes, and query performance.
- Implement monitoring, logging, alerting, audit controls, and data-quality checks.
- Build reusable Python, PySpark, and SQL components.
- Create CI/CD pipelines for Spark applications, Docker images, and Kubernetes deployments.
- Review technical designs and support data engineers with implementation standards.
Required Skills
- 6+ years of hands-on data engineering experience.
- Strong experience with Microsoft Azure data platforms.
- Advanced Python, PySpark, and SQL skills.
- Strong hands-on experience with: Apache Spark, Kubernetes and AKS, Docker, Azure Data Lake Storage Gen2, Azure Event Hubs, Azure DevOps and Git
- Experience deploying and operating Spark applications on Kubernetes.
- Strong understanding of Spark drivers, executors, resource allocation, partitioning, caching, broadcast joins, shuffle optimization, and skew handling.
- Experience with batch, streaming, ETL, ELT, and event-driven processing patterns.
- Experience implementing Medallion Architecture.
- Experience with REST APIs, SFTP, JSON, CSV, Parquet, Delta Lake, and relational databases.
- Experience with CDC, incremental processing, schema enforcement, and schema evolution.
- Strong understanding of data modelling, schema design, partitioning, and storage optimization.
- Experience with Kubernetes Jobs, Cron Jobs, Config Maps, Secrets, resource limits, node pools, and autoscaling.
- Experience implementing pipeline observability, data validation, monitoring, alerting, and error recovery.