Data Engineer
Posted
Summary
A data engineer who owns critical payment-lifecycle streaming pipelines: migrating dbt/SQL business logic into Apache Flink jobs (Java/Python), monitoring them in Datadog, deploying via ArgoCD/Terraform, and managing long-lived state with Apache Iceberg. Hybrid role in London (3 days/week in office).
Responsibilities
- Shift left processing by migrating existing business logic from dbt/SQL into Flink stream-processing jobs (Java/Python).
- Take the lead on the Payment Lifecycle Flink jobs, moving them from Data Platform ownership into the Analytics domain.
- Be responsible for the health, monitoring (Datadog), and deployment (ArgoCD/Terraform) of these critical pipelines.
- Work with Apache Iceberg to manage state and lookups for long payment lifecycles.
- Take initiative to improve and optimise engineering workflows and platforms.
Key Requirements
- Proven delivery experience in data or software engineering.
- Comfortable writing and debugging production-grade Java or Python.
- Hands‑on experience with Apache Flink (or similar engines like Spark Streaming/Kafka Streams) and Apache Kafka.
- Understanding of checkpoints, watermarking, and state management.
- Familiarity with the modern DevOps stack: Docker, Kubernetes, Terraform, and CI/CD principles.
- Strong focus on data integrity.
Benefits
Hybrid working model offering flexibility, with three days per week in the office.