freehire launches on Product Hunt on 26 August.

Follow →

emagine

Discussion

Data Engineer

Summary

Build and maintain Azure Databricks pipelines and ETL processes using PySpark, Delta Lake, and SQL to migrate and transform data systems.

Job Description: Data Engineer, LDWH Migration Programme

Level: Senior / Mid-Senior

Main Responsibilities:

  • Develop and maintain data pipelines using Azure Databricks.

  • Implement ETL processes, ensuring data quality and integrity.

  • Read and comprehend complex SQL stored procedures.

  • Translate architecture direction into engineering delivery.

  • Conduct data profiling and quality assessments.

  • Define acceptance criteria for data transformation.

  • Collaborate with the programme team to support overall migration efforts.

  • Participate in CI/CD processes for efficient deployment.

Key Requirements:

  • Senior hands-on experience with Azure Databricks (jobs, clusters, Unity Catalog, SQL warehouses).

  • Expertise in PySpark / Python for transformation logic and ETL pipelines.

  • Solid working knowledge of Delta Lake (schema evolution, MERGE, time travel).

  • Advanced SQL skills (complex queries, window functions); ability to comprehend legacy ETL.

  • Familiarity with Databricks Asset Bundles (DAB) and CI/CD practices (Azure DevOps, Git).

  • Working knowledge of Azure Data Lake Storage Gen2.

  • Comfortable understanding legacy ETL processes and source data profiling.

  • Experience with data reconciliation (defining criteria, interpreting discrepancies).

  • Ability to articulate requirements for junior engineers.

  • Fundamental knowledge of Power BI / Tableau.

Nice to Have:

  • Experience in platform engineering and configuration refactoring using YAML.

  • Ability to maintain environment governance across DEV, QAS, and PRD.

  • Skills in source feasibility analysis for new data sources.

  • Capability in transformation development across various data zones using PySpark and Delta Lake.

  • Familiar with implementing acceptance criteria and conducting quality checks.

Other Details:

This position is part of a complex and dynamic migration programme and is pivotal for ensuring the successful transition of data systems. Remote engagement options are available, and collaboration with an internal team is essential.

Show