Head of Data Engineering

Camden AI is a London-based data and AI consultancy. We help clients across retail, ecommerce, higher education, media, insurance and hospitality design, build and run the data foundations needed to automate business processes, support AI solutions and deliver actionable insight.

From strategy through to hands-on implementation, we solve complex data challenges and turn them into reliable, production-ready solutions.

About Camden AI

Camden AI is a London-based data and AI consultancy. We help clients across retail, ecommerce, higher education, media, insurance and hospitality design, build and run the data foundations needed to automate business processes, support AI solutions and deliver actionable insight.

From strategy through to hands-on implementation, we solve complex data challenges and turn them into reliable, production-ready solutions.

The Role

We are looking for a senior, hands-on data engineering leader to drive project delivery and client engagement across our portfolio, while helping to develop new customer partnerships.

This is not a purely advisory or governance role. You will spend a significant part of your time working directly in the technology: writing and reviewing SQL, designing data models, building and debugging pipelines, and resolving performance and data‑quality issues across Azure, Databricks and Microsoft Fabric.

You will combine deep technical expertise with commercial and business understanding. With a background in consultancy or professional services, you will be comfortable owning delivery across multiple clients, industries and technical environments.

Key Responsibilities

• Lead the hands-on design, build and delivery of modern cloud data platforms, primarily using Azure, Databricks and Microsoft Fabric.
• Write and review complex SQL for data transformation, integration and reporting, including query optimisation, performance tuning and troubleshooting.
• Design, build, test and optimise production ETL/ELT pipelines, with appropriate orchestration, monitoring, error handling and recovery.
• Integrate data from operational databases, APIs, files, SaaS applications and third‑party systems, resolving inconsistent or duplicated data at source.
• Design and review dimensional, warehouse and lakehouse data models for scalability, accuracy, maintainability and performance.
• Investigate and resolve data‑quality, pipeline reliability and platform performance issues in development and production environments.
• Set practical engineering standards for version control, testing, deployment, monitoring and technical documentation, and mentor delivery team members.
• Work with Solution Architects and delivery teams to translate business needs into practical technical designs and take them through to production.
• Support new‑business opportunities through client conversations, discovery and the design of credible technical solutions and delivery approaches.
• Apply proportionate controls for data quality, lineage, ownership and access as part of engineering delivery, without creating unnecessary process.
• Help shape practical AI and agentic solutions using tools such as OpenAI, Copilot and Claude, ensuring they are supported by reliable data foundations.

What You’ll Bring

• Significant hands‑on data engineering experience, ideally within consultancy or professional services and across multiple client environments.
• Advanced SQL skills, including complex transformations, query optimisation, performance tuning, debugging and code review.
• Strong experience designing, building and supporting production ETL/ELT pipelines and integrating data from multiple source systems.
• Hands‑on experience with Microsoft Azure, Databricks, Microsoft Fabric or comparable cloud data platforms.
• Strong data‑modelling experience across dimensional, warehouse and lakehouse patterns, with a focus on scalable and maintainable solutions.
• Practical Python or PySpark experience for data processing, automation and pipeline development.
• Experience with engineering practices such as Git, CI/CD, automated testing, monitoring and infrastructure as code.
• Strong client‑facing communication and commercial awareness, with the ability to translate business requirements into solutions that can actually be built.
• Exposure to AI/ML and agentic solution architecture is useful, but the core of this role is hands‑on data engineering and technical delivery.

Technical Environment

• Platforms: Microsoft Azure, Azure Data Factory, Azure Synapse Analytics, Microsoft Fabric, Databricks and Delta Lake.
• Data engineering: SQL, Python/PySpark, Spark, notebooks, APIs, ETL/ELT and workflow orchestration.
• Architecture and modelling: Dimensional modelling, Data Vault, data warehousing and lakehouse architecture.
• Engineering and AI tools: Git/GitLab, CI/CD, testing, monitoring and Terraform or similar infrastructure-as-code tooling; OpenAI, Copilot, Claude, LLMs and agentic architecture.