Lead Data Engineer
Summary
Lead cloud-based data platform design, automation, and optimization in GCP, mentoring engineers and ensuring scalable, secure data solutions for enterprise transformation.
Lead Data Engineer
Key Points
- Location: London Or Reading, Hybrid
- Duration: 12 months
- Rate: Competitive Day Rate, Outside IR35
Overview
This is an opportunity to join a leading organisation undertaking a major cloud transformation and data modernisation programme, delivering enterprise-scale data solutions across a complex technology landscape.
As a Lead Data Engineer, you will provide technical leadership across the design, development, automation, and optimisation of cloud-based data platforms, ensuring secure, scalable, and high-performing solutions are delivered to support business and operational objectives.
Working closely with architects, stakeholders, analysts, and engineering teams, you will drive best practice across data engineering, infrastructure automation, CI/CD, and cloud technologies while mentoring and leading other engineers within the delivery team.
Key Responsibilities
- Lead the design and delivery of enterprise-scale data engineering solutions within Google Cloud Platform (GCP).
- Develop and maintain scalable ETL/ELT pipelines, data models, and integration frameworks.
- Design and manage Infrastructure as Code (IaC) solutions using Terraform.
- Build and support automated deployment processes through GitLab CI/CD pipelines.
- Design, optimise, and maintain BigQuery datasets, schemas, and analytical workloads.
- Develop and manage Cloud Storage, Cloud SQL, and Cloud Composer solutions.
- Develop Python-based ETL applications, automation tools, and data services.
- Implement real-time data integration and Change Data Capture (CDC) solutions using Striim.
- Develop and support ETL workflows and reusable components using Ab Initio.
- Ensure data quality, governance, security, and compliance standards are maintained.
- Monitor platform performance, reliability, and operational efficiency across cloud environments.
- Lead code reviews, mentor engineers, and establish engineering best practices.
- Collaborate with cross-functional teams to deliver secure, reliable, and scalable data solutions.
- Support architecture design, technical planning, estimation, and continuous improvement initiatives.
What they ask for
Required
- Experience with Google Cloud Platform (GCP)
- ETL/ELT pipeline development
- Infrastructure as Code (IaC) using Terraform
- GitLab CI/CD pipeline automation
- BigQuery dataset design and optimization
- Python-based ETL applications and automation
- Real-time data integration and CDC using Striim
- ETL workflows and reusable components using Ab Initio
- Data quality, governance, security, and compliance
- Cloud Storage, Cloud SQL, and Cloud Composer solutions
Preferred
- Mentoring and leading engineers
- Collaboration with cross-functional teams