Databricks Data Engineer
Posted 11
views
Description
Salary: £75,000 - 95,000 per year
Requirements:- 5+ years of hands-on Databricks experience, including expert-level knowledge of Azure Databricks.
- Advanced PySpark, Spark SQL, and Python programming skills.
- Strong experience developing complex ETL/ELT pipelines and implementing Medallion Architecture (Bronze, Silver, and Gold).
- Deep understanding of Delta Lake, Delta Tables, and Delta Live Tables / Lakeflow Declarative Pipelines.
- Strong knowledge of Apache Spark architecture, optimisation, and performance tuning.
- Hands-on experience with Unity Catalog, access controls, and data governance.
- Experience implementing incremental data loads, CDC, and SCD Type 1/Type 2.
- Strong SQL development and query optimisation skills.
- Experience integrating Azure Data Factory with Databricks and knowledge of Azure Data Lake Storage Gen2.
- Experience with CI/CD pipelines, Git, Azure DevOps, and automated deployments.
- Experience implementing data quality, validation, and reconciliation frameworks.
- Previous experience delivering data engineering projects in UK Government or other public sector organisations, such as Central Government, Local Government, NHS, DWP, HMRC, or the Home Office.
- Understanding of public sector data governance, security, and information management.
- Experience working with sensitive and confidential datasets.
- Knowledge of UK GDPR, the Data Protection Act 2018, and relevant government data security standards.
- Experience working in Agile delivery teams and government digital transformation programmes.
- Eligibility and willingness to undergo SC clearance, government security vetting, and background checks.
- Previous UK Government or Central Government project experience is highly desirable.
- Design and develop scalable Databricks data pipelines.
- Build and maintain enterprise-grade Lakehouse platforms.
- Optimise Spark workloads and Databricks clusters.
- Implement robust data quality and governance controls.
- Collaborate with architects, analysts, and business stakeholders.
- Support production deployments and troubleshoot complex data issues.
- Ensure solutions meet public sector security and compliance requirements.
- Azure
- CI/CD
- Cloud
- Databricks
- DevOps
- ETL
- Git
- Support
- Python
- PySpark
- SQL
- Security
- Spark
- Unity
- GameDev
More:
We are seeking an experienced Senior Databricks Data Engineer to deliver enterprise-scale data engineering solutions using Azure Databricks. The role involves building complex data platforms within UK public sector environments, working with large datasets, sensitive information, and secure cloud-based data architectures.
last updated 46 week of 2026
Skills
Sign in to read and join the discussion.