Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

GIS Recruiter Ltd

NewBe an early applicant

Databricks Data Engineer

Posted 11 views

Description

Salary: £75,000 - 95,000 per year

Requirements:
  • 5+ years of hands-on Databricks experience, including expert-level knowledge of Azure Databricks.
  • Advanced PySpark, Spark SQL, and Python programming skills.
  • Strong experience developing complex ETL/ELT pipelines and implementing Medallion Architecture (Bronze, Silver, and Gold).
  • Deep understanding of Delta Lake, Delta Tables, and Delta Live Tables / Lakeflow Declarative Pipelines.
  • Strong knowledge of Apache Spark architecture, optimisation, and performance tuning.
  • Hands-on experience with Unity Catalog, access controls, and data governance.
  • Experience implementing incremental data loads, CDC, and SCD Type 1/Type 2.
  • Strong SQL development and query optimisation skills.
  • Experience integrating Azure Data Factory with Databricks and knowledge of Azure Data Lake Storage Gen2.
  • Experience with CI/CD pipelines, Git, Azure DevOps, and automated deployments.
  • Experience implementing data quality, validation, and reconciliation frameworks.
  • Previous experience delivering data engineering projects in UK Government or other public sector organisations, such as Central Government, Local Government, NHS, DWP, HMRC, or the Home Office.
  • Understanding of public sector data governance, security, and information management.
  • Experience working with sensitive and confidential datasets.
  • Knowledge of UK GDPR, the Data Protection Act 2018, and relevant government data security standards.
  • Experience working in Agile delivery teams and government digital transformation programmes.
  • Eligibility and willingness to undergo SC clearance, government security vetting, and background checks.
  • Previous UK Government or Central Government project experience is highly desirable.
Responsibilities:
  • Design and develop scalable Databricks data pipelines.
  • Build and maintain enterprise-grade Lakehouse platforms.
  • Optimise Spark workloads and Databricks clusters.
  • Implement robust data quality and governance controls.
  • Collaborate with architects, analysts, and business stakeholders.
  • Support production deployments and troubleshoot complex data issues.
  • Ensure solutions meet public sector security and compliance requirements.
Technologies:
  • Azure
  • CI/CD
  • Cloud
  • Databricks
  • DevOps
  • ETL
  • Git
  • Support
  • Python
  • PySpark
  • SQL
  • Security
  • Spark
  • Unity
  • GameDev

More:

We are seeking an experienced Senior Databricks Data Engineer to deliver enterprise-scale data engineering solutions using Azure Databricks. The role involves building complex data platforms within UK public sector environments, working with large datasets, sensitive information, and secure cloud-based data architectures.

last updated 46 week of 2026

Skills

Apply

See also

Data Engineering jobs by country — openings, pay and top skills →