Lead Data Engineer
Summary
Senior/Lead Data Engineer at Howden, an insurance group, leading design and delivery of scalable, metadata-driven data pipelines and orchestration frameworks on Databricks and Azure (ADF, ADLS, Azure Functions). Sets CI/CD, DevOps, and data governance best practices and mentors junior engineers.
As a Senior Data Engineer at Howden, you will lead the design and delivery of scalable, metadata-driven data solutions using Databricks and Azure. You will own metadata ingestion and orchestration frameworks, shaping data pipelines to enable business insights and operational efficiencies. You’ll drive best practices, CI/CD for data, and mentor junior engineers while collaborating with stakeholders across disciplines. This role places you at the heart of a modern Azure-based data ecosystem, contributing to Howden’s data-driven transformation and cross-functional impact.
Pay / Benefits- flexible hours
- hybrid working
- reasonable adjustments
- diversity and inclusion
- career development
- global team culture
- Lead metadata ingestion frameworks and orchestration processes
- Design and implement scalable, reusable metadata-driven data services
- Architect and maintain Databricks/Azure-based data solutions (Autoloader, Spark Declarative Pipelines, ADF, ADLS, Azure Functions)
- Establish and enforce data engineering, CI/CD, and DevOps best practices
- Collaborate with stakeholders, data scientists and analysts to translate data needs into solutions
- Optimize pipelines for performance, reliability, and scalability
- Lead data governance and security best practices
- Mentor junior engineers and promote continuous improvement
- 8+ years in data engineering focusing on Databricks (SQL & PySpark)
- Extensive Azure data services experience (ADF, ADLS, Azure Functions)
- Track record of metadata-driven pipeline design
- Expertise in orchestration and data workflow automation (Airflow, DBT)
- Strong CI/CD practices for data engineering
- Infrastructure as code experience (Terraform, Bicep, or ARM templates) preferred
- Solid software engineering standards and design discussions
- Strong stakeholder management and ability to translate requirements into data solutions
- Knowledge of data governance, security, and compliance
- leadership
- communication
- collaboration
- Databricks (SQL & PySpark)
- Azure data services (ADF, ADLS, Azure Functions)
- metadata ingestion and governance