Lead Data Engineer
Posted
Summary
Recruiting on behalf of a Dubai client, this role leads the design of scalable data pipelines delivering analytics-ready datasets for BI and AI. Day-to-day centers on Databricks, Spark, Delta Lake, dbt, and Azure data services, with data quality, governance, and pipeline monitoring across a hybrid environment.
My client is looking for a Lead Data Engineer to design and build scalable data pipelines that deliver trusted, analytics‑ready datasets for BI, AI, and operational use cases across a hybrid environment.
Key Responsibilities
- Build pipelines across bronze, silver & gold layers (Databricks, Spark, dbt)
- Implement data quality checks, contracts & schema validation
- Apply governance (catalog, lineage, RBAC, metadata)
- Deliver curated datasets, features & embeddings for AI/BI
- Monitor pipeline health, performance & cost to meet SLAs
Tech Stack
- Databricks
- Spark
- Delta Lake
- dbt
- Azure Data Factory
- Kafka/Event Hubs
- CI/CD (Azure DevOps/GitHub)
Governance & Ops
- Enforce data contracts, lineage & cataloging
- Apply masking, tokenisation & access controls (PII/PHI)
- Build observable pipelines with alerts, dashboards & runbooks
- Optimize performance (partitioning, caching, cost efficiency)
Requirements
- 5+ years in Data Engineering
- Strong SQL, data modeling (dimensional/data vault)
- Proficiency in Python
- Hands‑on with Databricks, Spark, Delta Lake & dbt
- Experience with Azure data services (ADF, ADLS, Key Vault)
- Familiarity with CI/CD & container basics (Docker/Kubernetes)
Nice to Have
- Streaming (Kafka/Event Hubs) & CDC (GoldenGate)
- Catalog/lineage tools (Purview, OvalEdge)
- S3‑compatible storage (MinIO, VAST)
- Exposure to BI tools (Power BI)
- Healthcare standards (FHIR/MDR)
Education
Bachelor's in Computer Science, Engineering, or related field.