Senior Data Engineer
Summary
Contract Senior Data Engineer in London leading technical direction on an Azure-based data platform: designing engineering patterns in Azure Databricks and Data Factory, optimizing large-scale pipelines, improving DevOps/CI-CD practices, and setting standards with data, analytics and platform teams.
Senior Data Engineer contract in London leading technical direction across Azure-based data platform. You’ll improve patterns, ensure quality and performance, and drive DevOps enhancements while collaborating with data, analytics and platform teams. The role focuses on optimizing a large data estate, establishing standards, and enabling scalable data pipelines. You’ll work autonomously to challenge approaches and deliver rapid impact in a global organization.
Responsibilities- Design and improve engineering patterns across Azure Databricks and Azure Data Factory
- Drive performance optimization of data pipelines (caching, indexing, partitioning, cost)
- Review code and act as a technical quality gate
- Improve DevOps practices (CI/CD, environment promotion, branching, release management)
- Identify opportunities to automate repetitive engineering tasks
- Work with advanced SQL within established patterns
- Support data quality, governance and orchestration
- Collaborate with data engineering, analytics, data science and platform teams
- Establish consistent engineering standards across the service
- Azure Databricks
- Azure Data Factory
- Spark / PySpark
- Advanced SQL and performance optimisation
- CI/CD and Azure DevOps
- Delta Lake and medallion architecture
- Unity Catalog
- Data governance and orchestration
- API-based data ingestion
- Experience with high-volume operational, IoT or telemetry data is highly valuable
- Autonomous and self-driven
- Collaborative cross-functional teamwork
- Strong problem-solving and critical-thinking
- Azure Databricks
- Azure Data Factory
- Spark / PySpark