Senior Data Engineer
Summary
Senior Data Engineer on an agile team building and optimizing scalable ETL/ELT pipelines and medallion-style Lakehouse architectures across a multi-cloud platform. Day-to-day work centers on Python, SQL, Spark/PySpark, Airflow, and dbt, plus CI/CD, data quality frameworks, and AWS tooling (S3, Athena, Redshift).
A hands-on engineering role building and optimising scalable ETL/ELT pipelines and Lakehouse architectures across a modern multi-cloud platform using Python, Spark, Airflow, and dbt.
Requires 5+ years of data engineering experience with strong expertise in SQL, CI/CD, data quality frameworks, and cloud technologies (S3, Athena, Redshift), with a plus for streaming and AI tooling experience.
Overview
A Senior Data Engineer position within an agile team, focused on building scalable data pipelines and modernizing a multi-cloud data platform. The role involves working with streaming technologies, Lakehouse architectures, and leveraging automation/AI to improve efficiency.
What you will do
- Pipeline Development: Build ETL/ELT pipelines (Python, SQL), optimize Spark/PySpark jobs, manage Airflow DAGs, and implement dbt transformations.
- Lakehouse Architecture: Design medallion-style architectures using S3 and modern table formats (Iceberg, Delta Lake) with optimized layouts.
- Platform Infrastructure: Create reusable frameworks, enforce data contracts, build CI/CD pipelines, and establish observability practices.
- Data Quality: Implement validation frameworks, data contracts, and SLIs/SLOs for critical pipelines.
Your experience
Required
- 5+ years in data engineering
- Strong Python, advanced SQL, Airflow, Spark/PySpark, dbt, and CI/CD proficiency
- Experience with Lakehouse architectures, dimensional modeling, and cloud data warehouses (S3, Athena, Redshift)
- Data quality frameworks, monitoring, and schema evolution expertise
Nice-to-Haves
- AI coding assistants (Cursor, Claude Code, Copilot)
#SDELONDON #TECHLONDON #seniordataengineerUK #Amazonwebservices