Data Engineer - SC Cleared - AWS
Summary
Data Engineer (active SC clearance required) supporting a large-scale UK central government data and analytics programme in London. Day to day you design, build and support secure AWS data pipelines and Kafka-based streaming services using Python and SQL, with Terraform-managed infrastructure, CI/CD and production support.
Data Engineer - AWS / Python / Kafka - SC Cleared
We are looking for experienced Data Engineers to support a large-scale central government data and analytics programme.
You will design, build and support secure, scalable data pipelines and services, working across cloud infrastructure, streaming technologies and production data environments.
Due to the nature of the roles, active SC is required.
Key responsibilities
- Design, build and maintain data pipelines and data-processing services.
- Develop production-quality solutions using Python and SQL.
- Build and support Kafka-based streaming and event-driven architectures.
- Work with AWS services and cloud-native data platforms.
- Manage infrastructure through Terraform / Infrastructure as Code.
- Support data profiling, mapping and transformation.
- Re-engineer existing pipelines as upstream systems or requirements change.
- Contribute to monitoring, incident investigation and production support.
- Work with engineering, architecture and delivery teams to improve data services.
Essential experience
- Strong Python engineering experience.
- Strong AWS experience.
- Apache Kafka / streaming data.
- Terraform.
- Strong SQL.
- ETL / ELT and data pipeline development.
- CI/CD and automated deployment.
- Experience supporting production data services.