Data Engineer
Posted Updated
Summary
On-site data engineer in London who builds and maintains ETL/ELT data pipelines and real-time streaming with Apache Kafka, runs containerised data services on Kubernetes, and writes Python/Java automation, with CI/CD, data quality, and Agile delivery responsibilities.
- Develop, test, deploy, and maintain data pipelines and ETL/ELT processes.
- Build and support real-time data streaming solutions using Apache Kafka.
- Design and implement containerised applications and data services using Kubernetes.
- Develop software components and automation scripts using Python and Java.
- Integrate data from multiple internal and external sources into cloud and on-premise platforms.
- Monitor, troubleshoot, and optimise data processing workflows.
- Implement data quality checks, validation rules, and data governance controls.
- Support CI/CD pipelines and DevOps practices to enable automated deployments.
- Collaborate with Solution Architects, Data Architects, Business Analysts, and stakeholders to understand requirements.
- Produce technical documentation and maintain operational procedures.
- Participate in Agile ceremonies, including sprint planning, stand-ups, reviews, and retrospectives.
- Support incident investigation and resolution activities within agreed SLAs.