Senior Data Engineer | GCP | Kafka | BigQuery | Python
Summary
A Senior Data Engineer will design, build, and operate production batch and real-time data pipelines on GCP (BigQuery, Airflow/Cloud Composer, GCS) using Python, SQL, Kafka, and dbt for a global FTSE 100 fintech, supporting analytics, compliance, and AI/ML use cases.
A leading FTSE 100 fintech is looking for a Senior Data Engineer to join its global Data Engineering function and contribute to a major transformation of its data platform.
The team is consolidating a complex data estate onto GCP, creating a scalable and trusted platform that supports analytics, compliance, Data Science and increasingly sophisticated AI/ML use cases.
This is a hands‑on role suited to an experienced engineer who enjoys solving complex data problems and wants to have a meaningful influence on engineering standards and platform design.
The role
Working within a Data Engineering squad, you will design, build and operate production‑grade data pipelines across both batch and real‑time environments.
You will work across:
- GCP – BigQuery, Cloud Composer/Airflow, GCS and Cloud Run
- Apache Kafka – event‑driven ingestion and real‑time streaming
- Python and SQL – building reliable, production‑grade data solutions
- dbt – transformation and data modelling
- Medallion architecture – Bronze, Silver and Gold layers
- Data quality and observability – contracts, lineage, monitoring and SLOs
Role responsibilities
- Designing and building scalable batch and real‑time data pipelines
- Developing Kafka‑based event streams and Airflow‑orchestrated workflows
- Building transformations and data models in BigQuery
- Developing and maintaining ingestion integrations and connectors
- Implementing data quality controls, observability and pipeline monitoring
- Contributing to engineering standards, code reviews and CI/CD practices
- Supporting Data Science and AI‑ready data initiatives
- Mentoring junior engineers and contributing to the development of the wider engineering team
Role Requirements
- Strong commercial experience building and operating production data pipelines on GCP
- Strong SQL and Python skills
- Solid experience with Apache Kafka or comparable streaming technologies
- Experience with BigQuery, Airflow/Cloud Composer and GCS
- Experience with dbt and modern data transformation approaches
- Understanding of Medallion architecture and data platform best practices
- Experience with data contracts, lineage, observability and pipeline SLOs
- Experience working within a structured or regulated environment
This is an opportunity to work on large‑scale data engineering challenges within a global fintech, with the technical scope to make a real impact on the platform and how data is used across the organisation.