Lead Data Engineer
Summary
A 6-month inside-IR35 contract with a leading insurance client (via recruiter Stott and May), UK fully remote. The Lead Data Engineer owns the Databricks architecture and roadmap, builds ETL/ELT pipelines with Python, PySpark and SQL, implements governance in a regulated environment, and mentors a team of data engineers.
Lead Data Engineer
Location:UK – Fully Remote
Contract:6-month initial contract
Rate:(Inside IR35)
We are looking for an experienced Lead Data Engineer to join a leading insurance client, providing technical leadership across data engineering initiatives and supporting the development of a high-performing engineering team.
The role is both hands-on and leadership-focused, with Databricks at the centre of the technology stack alongside Python, PySpark and SQL. You will take ownership of the Databricks platform and architecture while working closely with senior stakeholders across Risk, Quantitative Finance, Compliance and Data Science.
Key Responsibilities
- Own the Databricks architecture, strategy and roadmap, ensuring performance, availability and cost efficiency.
- Design, build and operationalise scalable ETL/ELT data pipelines using Databricks, Python and SQL.
- Develop and optimise solutions using Databricks Lakehouse and Delta Lake.
- Implement data governance, lineage, controls and audit frameworks within a regulated environment.
- Lead, mentor and upskill a team of Data Engineers, establishing technical standards and conducting code reviews.
- Provide technical direction across data engineering projects and drive engineering best practices.
- Work closely with Risk, Quantitative Finance, Compliance and Data Science teams to translate business requirements into technical solutions.
- Communicate complex technical concepts and architecture clearly to senior and executive stakeholders.
Required Skills & Experience
- 7+ years' experience in Data Engineering.
- 3+ years' hands-on experience delivering large-scale Databricks implementations within financial services.
- Strong experience within Insurance, Banking, Asset Management or FinTech.
- Advanced Python, PySpark and SQL skills.
- Deep knowledge of Databricks Lakehouse architecture and Delta Lake.
- Experience with at least one major cloud platform: AWS, Azure or GCP.
- Proven experience leading or mentoring Data Engineering teams.
- Strong stakeholder management and communication skills.
- Experience working within regulated environments, with an understanding of data governance and compliance requirements.