Lead Data Engineer
Summary
Lead data engineer at an early-stage AI product company in London: you own the data architecture and technical direction, building batch and real-time pipelines plus vector search and ML data infrastructure. Core stack includes Spark, Airflow, Kafka, and Elasticsearch/OpenSearch.
Want to own the data architecture behind an AI product while the foundations are still being built?
This is an early-stage lead role where you’ll shape the systems powering real-time data, AI workflows, vector search and analytics — with the freedom to make the key technical decisions.
What’s in it for you?
- Own the data architecture and technical direction
- Build the data backbone of an AI-first product from the ground up
- Work closely with AI, backend and product engineers
- Solve problems across streaming, vector search and ML data infrastructure
- Help define tooling, standards and the future data team
- Competitive salary + equity
- Join early enough to have genuine influence over how things are built
What you’ll be doing
- Architecting scalable batch and real-time data pipelines
- Designing ingestion, transformation and storage systems
- Building infrastructure for vector search, retrieval and ML workflows
- Improving data quality, observability and reliability
- Optimising large-scale datasets and query performance
- Supporting AI training, inference and product features
- Making long-term architecture and tooling decisions
- Helping grow and mentor the data engineering function
What you’ll bring
- Experience designing scalable pipelines and distributed data systems
- Experience with relational and NoSQL databases
- Hands‑on experience with vector databases
- Good understanding of data modelling, performance and storage architecture
- Experience with technologies such as Spark, Airflow, Kafka or Elasticsearch/OpenSearch
Experience with AI/ML platforms, cloud infrastructure or event-driven architectures would be useful, but you don’t need everything on the list.