Lead Data Engineer

As Lead Data Engineer, you will own the Databricks-based data platform for the Retail Insight Data and Engineering team. You will design scalable data infrastructure, guide a small engineering team, and set platform standards for data architecture and best practices that support analytics, ML, and AI across the business. You’ll translate business needs into robust technical solutions while partnering with data science, analytics and product teams. This is a senior, hands-on role with a clear impact on performance, security and governance at scale.

• Lead design, development and optimisation of a Databricks data platform for reliability, performance and scalability
• Design enterprise-scale data architectures on Databricks and cloud platforms (Azure, AWS, GCP)
• Define lakehouse standards, data modelling approaches and governance frameworks
• Lead batch and streaming platform design using Delta Lake and Spark
• Establish security, access control, data quality and compliance standards
• Optimise performance, scalability and cloud cost management
• Mentor a small team of data engineers and set technical standards
• Partner with data science, analytics and product teams to ensure platform support
• Own data quality, governance and observability across pipelines
• Support AI/ML enablement including feature stores and model deployment patterns
• Translate business requirements into technical solutions
• Contribute to platform roadmap and tooling selection

• Senior or lead data engineer with production data platform experience
• Deep hands-on Databricks expertise (Spark, Delta Lake, Databricks Workflows)
• Python and SQL proficiency
• Infrastructure as code (Terraform, Bicep or CloudFormation)
• Cloud data platform experience, ideally Azure (ADLS Gen2, Azure Data Factory, Azure DevOps)
• Strong data modelling, ETL/ELT, and distributed processing knowledge
• Security, governance and data platform operations experience
• Experience mentoring engineers and stakeholder management
• Rigorous data quality and testing approach
• Databricks certification

• Stakeholder management
• Mentoring and coaching
• Collaborative cross-functional partnership
• Databricks (Spark, Delta Lake, Databricks Workflows)
• Python
• SQL