Lead Data Engineer
Posted 2
views
Summary
Senior hands-on Lead Data Engineer owning Retail Insight's Databricks-based data platform: designing lakehouse architecture, batch and streaming pipelines, governance and cost optimisation, while mentoring a small team. Core stack includes Databricks, Spark, Delta Lake, Python, SQL, Terraform and Azure.
Lead Data Engineer (Databricks)
Make sure to apply quickly in order to maximise your chances of being considered for an interview Read the complete job description below.
Salary £65,000
Type Full time, permanent
Hours 37.5 per week
Location Remote or Hybrid UK, with occasional Richmond office attendance (a couple of times per month)
Overview
We are looking for a Lead Data Engineer to join the Retail Insight Data and Engineering team and own the architecture and development of our Databricks environment. This is a senior, hands on role where you will design and build scalable data infrastructure that underpins our products, while guiding a small team of engineers and defining platform standards, data architecture and engineering best practices supporting analytics, machine learning and AI across the business.
Key responsibilities
* Lead the design, development and optimisation of a Databricks based data platform, ensuring reliability, performance and scalability
* Design enterprise scale data architectures on Databricks and cloud platforms such as Azure, AWS or GCP
* Define lakehouse standards, data modelling approaches and governance frameworks
* Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies
* Establish security, access control, data quality and compliance standards
* Optimise performance, scalability and cloud cost management
* Lead and mentor a small team of data engineers, setting technical standards and reviewing code
* Partner with data science, analytics and product teams to ensure the platform supports their work
* Own data quality, governance and observability across pipelines
* Support AI and ML enablement including feature stores and model deployment patterns
* Translate business requirements into technical solutions
* Contribute to platform roadmap, tooling selection and continuous improvement
Required experience
* Strong experience as a senior or lead data engineer building and scaling production data platforms
* Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows
* Python and SQL proficiency
* Infrastructure as code experience using Terraform, Bicep or CloudFormation
* Cloud data platform experience, ideally Azure including xwwtmva Azure Data Factory, ADLS Gen2 and Azure DevOps
* Strong understanding of data modelling, ETL and ELT patterns, distributed processing
* Experience with security, governance and data platform operations
* Experience leading or mentoring engineers, with strong stakeholder management
* Rigorous approach to data quality and testing
* Databricks certification
Desirable
* Unity Catalog, MLflow and Databricks Asset Bundles
* dbt or similar transformation frameworks
* Retail or CPG data such as POS, inventory, supply chain
* Streaming technologies such as Kafka or Event Hubs
Education
* Degree in computer science, data engineering or related field, or equivalent practical experience
Make sure to apply quickly in order to maximise your chances of being considered for an interview Read the complete job description below.
Salary £65,000
Type Full time, permanent
Hours 37.5 per week
Location Remote or Hybrid UK, with occasional Richmond office attendance (a couple of times per month)
Overview
We are looking for a Lead Data Engineer to join the Retail Insight Data and Engineering team and own the architecture and development of our Databricks environment. This is a senior, hands on role where you will design and build scalable data infrastructure that underpins our products, while guiding a small team of engineers and defining platform standards, data architecture and engineering best practices supporting analytics, machine learning and AI across the business.
Key responsibilities
* Lead the design, development and optimisation of a Databricks based data platform, ensuring reliability, performance and scalability
* Design enterprise scale data architectures on Databricks and cloud platforms such as Azure, AWS or GCP
* Define lakehouse standards, data modelling approaches and governance frameworks
* Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies
* Establish security, access control, data quality and compliance standards
* Optimise performance, scalability and cloud cost management
* Lead and mentor a small team of data engineers, setting technical standards and reviewing code
* Partner with data science, analytics and product teams to ensure the platform supports their work
* Own data quality, governance and observability across pipelines
* Support AI and ML enablement including feature stores and model deployment patterns
* Translate business requirements into technical solutions
* Contribute to platform roadmap, tooling selection and continuous improvement
Required experience
* Strong experience as a senior or lead data engineer building and scaling production data platforms
* Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows
* Python and SQL proficiency
* Infrastructure as code experience using Terraform, Bicep or CloudFormation
* Cloud data platform experience, ideally Azure including xwwtmva Azure Data Factory, ADLS Gen2 and Azure DevOps
* Strong understanding of data modelling, ETL and ELT patterns, distributed processing
* Experience with security, governance and data platform operations
* Experience leading or mentoring engineers, with strong stakeholder management
* Rigorous approach to data quality and testing
* Databricks certification
Desirable
* Unity Catalog, MLflow and Databricks Asset Bundles
* dbt or similar transformation frameworks
* Retail or CPG data such as POS, inventory, supply chain
* Streaming technologies such as Kafka or Event Hubs
Education
* Degree in computer science, data engineering or related field, or equivalent practical experience
Skills
- AI
- Analytics
- AWS
- Azure
- Azure Data Factory
- Azure DevOps
- Bicep
- Cloud
- CloudFormation
- Data Engineering
- Data Modeling
- Data Quality
- Data Science
- Databricks
- dbt
- Delta Lake
- DevOps
- ELT
- ETL
- GCP
- Infrastructure as Code
- Kafka
- Lakehouse
- Machine Learning
- MLflow
- Model Deployment
- Observability
- Python
- Spark
- SQL
- Stakeholder Management
- Terraform
- Unity