Lead Data Engineer
Summary
Designs and maintains a scalable Databricks-based data platform for analytics, AI, and ML, leading a small team and setting technical standards for enterprise data architecture.
Lead Data Engineer (Databricks)
Salary
£65,000 Type
Full time, permanent Hours
37.5 per week Location
Remote or Hybrid UK, with occasional Richmond office attendance (a couple of times per month)
Overview
We are looking for a Lead Data Engineer to join the Retail Insight Data and Engineering team and own the architecture and development of our Databricks environment. This is a senior, hands on role where you will design and build scalable data infrastructure that underpins our products, while guiding a small team of engineers and defining platform standards, data architecture and engineering best practices supporting analytics, machine learning and AI across the business.
Key responsibilities
* Lead the design, development and optimisation of a Databricks based data platform, ensuring reliability, performance and scalability * Design enterprise scale data architectures on Databricks and cloud platforms such as Azure, AWS or GCP * Define lakehouse standards, data modelling approaches and governance frameworks * Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies * Establish security, access control, data quality and compliance standards * Optimise performance, scalability and cloud cost management * Lead and mentor a small team of data engineers, setting technical standards and reviewing code * Partner with data science, analytics and product teams to ensure the platform supports their work * Own data quality, governance and observability across pipelines * Support AI and ML enablement including feature stores and model deployment patterns * Translate business requirements into technical solutions * Contribute to platform roadmap, tooling selection and continuous improvement
Required experience
* Strong experience as a senior or lead data engineer building and scaling production data platforms * Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows * Python and SQL proficiency * Infrastructure as code experience using Terraform, Bicep or CloudFormation * Cloud data platform experience, ideally Azure including Azure Data Factory, ADLS Gen2 and Azure DevOps * Strong understanding of data modelling, ETL and ELT patterns, distributed processing * Experience with security, governance and data platform operations * Experience leading or mentoring engineers, with strong stakeholder management * Rigorous approach to data quality and testing * Databricks certification
Desirable
* Unity Catalog, MLflow and Databricks Asset Bundles * dbt or similar transformation frameworks * Retail or CPG data such as POS, inventory, supply chain * Streaming technologies such as Kafka or Event Hubs
Education
* Degree in computer science, data engineering or related field, or equivalent practical experience
TPBN1_UKTJ
£65,000 Type
Full time, permanent Hours
37.5 per week Location
Remote or Hybrid UK, with occasional Richmond office attendance (a couple of times per month)
Overview
We are looking for a Lead Data Engineer to join the Retail Insight Data and Engineering team and own the architecture and development of our Databricks environment. This is a senior, hands on role where you will design and build scalable data infrastructure that underpins our products, while guiding a small team of engineers and defining platform standards, data architecture and engineering best practices supporting analytics, machine learning and AI across the business.
Key responsibilities
* Lead the design, development and optimisation of a Databricks based data platform, ensuring reliability, performance and scalability * Design enterprise scale data architectures on Databricks and cloud platforms such as Azure, AWS or GCP * Define lakehouse standards, data modelling approaches and governance frameworks * Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies * Establish security, access control, data quality and compliance standards * Optimise performance, scalability and cloud cost management * Lead and mentor a small team of data engineers, setting technical standards and reviewing code * Partner with data science, analytics and product teams to ensure the platform supports their work * Own data quality, governance and observability across pipelines * Support AI and ML enablement including feature stores and model deployment patterns * Translate business requirements into technical solutions * Contribute to platform roadmap, tooling selection and continuous improvement
Required experience
* Strong experience as a senior or lead data engineer building and scaling production data platforms * Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows * Python and SQL proficiency * Infrastructure as code experience using Terraform, Bicep or CloudFormation * Cloud data platform experience, ideally Azure including Azure Data Factory, ADLS Gen2 and Azure DevOps * Strong understanding of data modelling, ETL and ELT patterns, distributed processing * Experience with security, governance and data platform operations * Experience leading or mentoring engineers, with strong stakeholder management * Rigorous approach to data quality and testing * Databricks certification
Desirable
* Unity Catalog, MLflow and Databricks Asset Bundles * dbt or similar transformation frameworks * Retail or CPG data such as POS, inventory, supply chain * Streaming technologies such as Kafka or Event Hubs
Education
* Degree in computer science, data engineering or related field, or equivalent practical experience
TPBN1_UKTJ