Data Engineer
Summary
Senior Data Engineer on an initial 6-month inside-IR35 contract, predominantly remote with some travel to a London client site. Designs and optimises production-grade data pipelines and Lakehouse solutions using Databricks, PySpark, Spark SQL and Delta Lake in a secure, regulated environment; must be eligible for UK SC clearance.
Role: Senior Data Engineer - Databricks
Location: Predominantly Remote – some travel to client site in London
Contract: Initial 6 months
Rate: £500 to £540 per day
Clearance: Must be eligible to obtain SC Clearance
This role has been deemed Inside IR35
The Opportunity
We are looking for an experienced Senior Data Engineer to work on complex, large-scale data projects within secure and highly regulated environments.
This is an excellent opportunity for a Data Engineer with strong Databricks, PySpark and modern Lakehouse experience who wants to remain hands-on while taking greater ownership of technical delivery.
You'll work across modern cloud data platforms, designing and delivering scalable data pipelines that transform complex datasets into trusted, reusable data products. Alongside engineering delivery, you'll have the opportunity to influence technical standards, contribute to architectural decisions and mentor other engineers.
The Role
As a Senior Data Engineer, you'll take ownership of the design, development and optimisation of production-grade data solutions using Databricks, Apache Spark and Delta Lake.
Your responsibilities will include:
- Designing, building and optimising scalable ETL/ELT data pipelines using Databricks, PySpark and Spark SQL
- Implementing Medallion Architecture patterns across data layers
- Developing robust data quality, monitoring and alerting processes
- Building and maintaining modern Lakehouse solutions
- Integrating Databricks with cloud services, data lakes, event streams and data warehouses
- Applying Unity Catalogue and wider data governance, security and quality controls
- Establishing engineering standards and patterns across ingestion, transformation, orchestration and monitoring
- Optimising Databricks workloads for performance, reliability and cost
- Supporting CI/CD and DevOps practices across data engineering workflows
- Leading technical delivery across data engineering workstreams
The Person
We're looking for an experienced Data Engineer who combines strong hands-on engineering skills with the ability to take technical ownership of complex data solutions.
You'll ideally have:
- Strong commercial experience with Databricks
- Excellent PySpark, Apache Spark and Spark SQL skills
- Experience building production-grade ETL/ELT pipelines
- Strong knowledge of Delta Lake and Lakehouse architecture
- Experience implementing Medallion/Bronze-Silver-Gold architectures
- Strong experience with Azure and/or AWS data platforms
- Knowledge of Unity Catalogue, data governance, data quality and security
- Experience integrating Databricks with wider cloud and enterprise data services
- An understanding of CI/CD and DevOps practices for data workflows
Experience with streaming or real-time data processing would be advantageous, as would previous experience working within Defence, Central Government, Security, Intelligence or another complex regulated environment.
You don't necessarily need to hold current security clearance, but you must be eligible to obtain SC Clearance.
If you would like to learn more, please apply through the advert and we will be in touch to discuss in more detail.