Data Engineer
Posted
Summary
Designs and builds scalable data pipelines and analytics-ready datasets across a hybrid lakehouse (bronze/silver/gold layers) for BI, AI, and operational use cases. Core stack: Databricks, Spark, Delta Lake, dbt, Azure Data Factory, and Kafka/Event Hubs, with emphasis on governance, data quality, and cost/performance.
Data Engineer (Lakehouse & Hybrid Data Platforms)
My client is looking for a Data Engineer to design and build scalable data pipelines that deliver trusted, analytics‑ready datasets for BI, AI, and operational use cases across a hybrid environment.
Key Responsibilities
- Build pipelines across bronze, silver & gold layers (Databricks, Spark, dbt)
- Implement data quality checks, contracts & schema validation
- Apply governance (catalog, lineage, RBAC, metadata)
- Deliver curated datasets, features & embeddings for AI/BI
- Monitor pipeline health, performance & cost to meet SLAs
Tech Stack
- Databricks, Spark, Delta Lake, dbt
- Azure Data Factory, Kafka/Event Hubs
- CI/CD (Azure DevOps/GitHub)
Governance & Ops
- Enforce data contracts, lineage & cataloging
- Apply masking, tokenisation & access controls (PII/PHI)
- Build observable pipelines with alerts, dashboards & runbooks
- Optimize performance (partitioning, caching, cost efficiency)
Requirements
- 5+ years in Data Engineering
- Strong SQL, data modeling (dimensional/data vault)
- Proficiency in Python
- Hands‑on with Databricks, Spark, Delta Lake & dbt
- Experience with Azure data services (ADF, ADLS, Key Vault)
- Familiarity with CI/CD & container basics (Docker/Kubernetes)
Nice to Have
- Streaming (Kafka/Event Hubs) & CDC (GoldenGate)
- Catalog/lineage tools (Purview, OvalEdge)
- S3‑compatible storage (MinIO, VAST)
- Exposure to BI tools (Power BI) & healthcare standards (FHIR/MDR)
Education
- Bachelor's in Computer Science, Engineering, or related field
Salt is acting as an Employment Agency in relation to this vacancy.