Data Engineer
Job Overview
We're currently seeking a Data Engineer to build, maintain, and optimise robust data pipelines. This role develops efficient ETL/ELT workflows using both Microsoft Fabric tools and the Databricks Lakehouse Platform to deliver clean, production‑ready data for enterprise analytics.
Job Requirements
- Strong proficiency in PySpark, Databricks SQL, T‑SQL, and handling Delta Lake formats.
- Hands‑on experience building production pipelines inside Databricks workspaces and Fabric capacities.
- Practical knowledge of structural data formats including Parquet, JSON, and CSV.
- Experience with orchestration tools, scheduling dependencies, and error handling in a cloud environment.
- Familiarity with unit testing data pipelines and implementing automated data quality checks.
Job Responsibilities
- Construct scalable data ingestion pipelines using Fabric Data Factory, Dataflow Gen2, and Databricks Workflows.
- Implement Delta Live Tables (DLT) to automate batch and real‑time streaming data processing.
- Build and maintain the Medallion Architecture silver & gold layer across shared storage.
- Write optimized transformations in Fabric Notebooks and Databricks clusters using PySpark and SQL.
- Configure data access, masking, and governance policies within Databricks Unity Catalog.
- Optimize Lakehouse table performance using Z‑Order indexing, liquid clustering, and file compaction techniques.
- Deploy data pipelines using CI/CD frameworks, Git integration, and Databricks Asset Bundles (DABs).
- Prepare finalized gold layer datasets to support real‑time Power BI Direct Lake reporting model.
Benefits
- Salary aligned with your skills and expertise
- 27 days holiday allowance plus statutory public holidays and the option to purchase more
- Birthday leave
- 2 giving back days per year – paid leave to work with local charities or good causes
- Car allowance
- Hybrid working
- Flexible benefit and well‑being schemes
- Enhanced family‑friendly policies, including enhanced maternity pay
- A hard working, fun and professional working environment