Lead Data Engineer
Summary
Lead Data Engineer (first senior data hire) at an AI start-up in London building an intelligent automation platform for B2C services. Day to day: design batch and real-time data pipelines, vector search and ML data infrastructure using Python, PostgreSQL/NoSQL, Kafka, Spark, and Airflow, while mentoring engineers and shaping data architecture.
Salary: £80,000 - 115,000 per year
Requirements
- 7+ years of experience in Data Engineering or Backend Engineering.
- Strong Python development experience.
- Experience building distributed data platforms and scalable pipelines.
- Excellent knowledge of PostgreSQL or similar relational databases, and NoSQL databases.
- Experience with vector databases such as Qdrant, Milvus, or pgvector.
- Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas, and Polars.
- Strong understanding of data modelling, storage architecture, and performance optimisation.
- Desirable: experience working on AI or Machine Learning platforms.
- Desirable: knowledge of cloud platforms such as AWS, Azure, or GCP.
- Desirable: JavaScript/Node.js experience.
- Desirable: event-driven architecture and stream processing experience.
- Desirable: previous experience within a high-growth start-up environment.
Responsibilities
- Design and build scalable batch and real-time data pipelines.
- Develop data ingestion, transformation, and storage solutions for operational and AI workloads.
- Build and optimise vector search, retrieval systems, and ML data pipelines.
- Ensure data reliability, governance, monitoring, and observability across the platform.
- Work closely with AI, backend, and product teams to support model training, inference, and product development.
- Optimise large-scale datasets and database performance.
- Help define engineering standards and mentor future data engineers.
Technologies
- AI
- Airflow
- AWS
- OpenSearch
- Azure
- Backend
- Cloud
- ElasticSearch
- GCP
- Support
- JavaScript
- Kafka
- Machine Learning
- NoSQL
- PostgreSQL
- Python
- Spark
- pandas
- NodeJS
- C#
- Embedded
- FPGA
- Golang
- Linux
More
We are an exciting AI start-up developing an intelligent automation platform that is transforming the B2C service. This is a hybrid role based in London with a salary of £80,000 to £115,000. As one of our first senior data hires, you will have the opportunity to shape our data architecture, work directly alongside our founders and senior engineers, and enjoy a high level of ownership, autonomy, and technical influence. We offer competitive salary with equity options and the chance to build technology that is transforming an entire industry.