Principal AI Data Engineer
Principal AI Data Engineer
Certain Advantage is hiring for a Principal AI Data Engineer based in London. This role is an initial contract until the end of the year and is hybrid, requiring 3 days per week onsite in London.
You’ll be responsible for:
- Developing AI, GenAI and Agentic AI prototypes using Azure AI Foundry, Copilot Studio, Databricks Mosaic AI and MLflow.
- Building, optimising and evaluating Retrieval-Augmented Generation (RAG) solutions using prompt engineering and embedding models.
- Designing and deploying AI agents using frameworks including LangChain, LangGraph and AutoGen.
- Delivering production-ready GenAI applications for commercial teams within Trading & Supply.
- Building scalable AI workflows on Databricks using Mosaic AI, MLflow, Genie and AgentBricks.
- Applying DevOps best practices through CI/CD pipelines, GitHub workflows and automated testing.
- Designing evaluation frameworks to benchmark, test and improve GenAI system performance and reliability.
The Individual
- Strong expertise with Databricks, including Delta Lake, Unity Catalog, DLT and MLflow.
- Deep knowledge of Large Language Models including GPT, Llama, Claude and Mistral.
- Experience building and deploying GenAI, RAG and Agentic AI solutions in enterprise environments.
- Strong programming skills in Python, SQL and/or Scala.
- Experience with Azure cloud technologies including Azure OpenAI and Azure AI Foundry.
- Knowledge of Spark Structured Streaming, Autoloader and modern data engineering practices.
- Experience with ETL/ELT, data modelling and scalable data architectures.
- Familiarity with Azure Data Factory, Qlik Replicate and data ingestion pipelines.