Data Engineer
Summary
Builds and maintains metadata-driven data pipelines in Azure Data Factory and Databricks, migrating large historical datasets to Cloudera Cloud with reconciliation and audit processes.
Mandatory Skills (MustHave)
- Azure Data Factory (ADF) – metadatadriven pipelines
- Basic Cloudera working knowledge
- Cloudera Cloud migration (mandatory)
- Largevolume historical data migration
- Azure Databricks (Spark / SQL)
- Data reconciliation and audit signoff
- Frameworkbased data ingestion / migration
GoodtoHave
- Azure Data Box–based migration experience
- Bulk history extraction using Data Box
- Landing data into ADLS Gen2
- Processing history in Databricks (Delta)
- Orchestrating incrementals via ADF