Data Engineer
Summary
Design, build and operate scalable batch and streaming data pipelines and data models that turn diverse sources into analytics-ready datasets for ML and analytics, across Azure, edge and on-prem environments. Core stack: Python, SQL, Azure, Docker, Kubernetes, Spark/Flink.
In this role, you design, build and operate scalable data pipelines and models that convert diverse sources into analytics-ready datasets for ML and analysis. You’ll support batch, streaming and interactive workflows across Azure, edge and on-prem environments, collaborating with data scientists, analysts and engineers in an agile setting. You will develop reusable platform components and ensure data quality, reliability and performance. This position offers hands-on impact shaping data products for mission-critical applications and advanced analytics.
Pay / Benefits- Bonus scheme
- Private healthcare + cash plan
- Life assurance (4x salary)
- Share options
- Pension
- Flexibility (Hybrid working)
- Build and maintain batch and streaming data pipelines (ingestion, transformation, orchestration)
- Monitor production data services and resolve data quality and performance issues
- Develop reusable platform components and tooling to boost engineering productivity
- Implement automated tests, validation checks and integration plans for data pipelines
- Design data models and storage patterns to support analytics, ML features, embeddings and conversational querying
- Document data flows, schemas and technical decisions
- Strong Python for production-grade data engineering code
- Strong SQL and data modeling experience
- Experience with structured/time-series/high-volume storage (SQL databases, InfluxDB or similar)
- Experience building batch and event-driven/streaming pipelines
- ~3+ years in a data engineering/data platform team
- Familiarity with distributed processing (Spark, Flink, or similar)
- Orchestration tooling to schedule/monitor/recover pipelines
- Understanding of event-driven architecture, message brokers and stream-processing patterns
- Hands-on with Azure data services and cloud data platform patterns
- Docker antibodies; comfortable building/debugging Docker containers
- Experience deploying/operating data workloads on Kubernetes (cloud/edge/on-prem)
- Experience developing/consuming APIs for ingestion, access and integration
- Strong attention to detail; able to prioritise independently; solid communicator
- Strong communication
- Independent prioritisation
- Collaboration in cross-functional teams
- Python
- SQL
- SQL databases