Staff Engineer - AI Data Engineering

Prudential's purpose is to be partners for every life and protectors for every future. Our purpose encourages everything we do by creating a culture in which diversity is celebrated and inclusion assured, for our people, customers, and partners. We provide a platform for our people to do their best work and make an impact to the business, and we support our people's career ambitions. We pledge to make Prudential a place where you can Connect, Grow, and Succeed.

We are looking for an experienced AI Data Engineering Lead to define and lead the engineering of enterprise-scale data platforms and data products that power Artificial Intelligence, Machine Learning, Generative AI and advanced analytics.

This individual will build the organisation's AI-ready data foundation by bringing together modern data engineering, cloud platforms, batch and real-time processing, unstructured data, vector and semantic technologies, data governance, and AI/ML engineering practices.

The role requires a senior technology leader with 15+ years of experience, strong hands-on technical depth, architecture capability, and a proven ability to lead engineering teams and deliver complex data platforms at enterprise scale.

Key Responsibilities

AI Data Engineering Strategy & Architecture

• Define the enterprise AI Data Engineering strategy, architecture and engineering standards.
• Design scalable data architectures supporting AI/ML, GenAI, LLM, analytics and enterprise data products.
• Establish reusable architectural patterns for structured, semi-structured and unstructured data.
• Define patterns for data lakes, lakehouse architectures, data warehouses, streaming platforms, vector databases and AI knowledge layers.
• Drive architectural decisions covering performance, scalability, resiliency, security and cost.

AI-Ready Data Platforms

• Lead engineering of data platforms providing trusted and reusable data for ML models, LLMs, AI agents and GenAI applications.
• Build ingestion and transformation frameworks for data from databases, APIs, documents, events, logs and enterprise applications.
• Establish pipelines for batch, near-real-time and streaming data processing.
• Drive automated data validation, quality controls, metadata management and lineage.
• Establish engineering approaches for preparing text, documents, images and other unstructured data for AI consumption.

GenAI & LLM Data Engineering

• Provide technical leadership for Retrieval-Augmented Generation (RAG), document ingestion, parsing, chunking and enrichment.
• Lead embedding generation and management, vector and hybrid search, semantic retrieval and reranking.
• Design knowledge repositories, metadata and access-control filtering, and enterprise search integration.
• Create data pipelines for LLM evaluation, AI grounding, authorised retrieval, fine-tuning where applicable, and AI agents.
• Understand how poor retrieval, incomplete or stale data, weak metadata and inappropriate chunking affect enterprise AI reliability.

Modern Data Engineering

• Provide leadership across ETL/ELT, distributed processing, lakehouse and data warehouse architectures, streaming, event-driven design, data APIs and data services.
• Drive data modelling, data contracts, observability, quality engineering, lineage and metadata management.
• Embed Infrastructure-as-Code, CI/CD, DevOps and DataOps practices in the engineering lifecycle.
• Improve reusability, automation, reliability and developer productivity through common frameworks and standards.

Cloud & Technology Platforms

• Demonstrate significant expertise across modern cloud and data platforms such as Microsoft Azure, Azure Databricks, Microsoft Fabric, Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, Snowflake and Databricks.
• Bring strong capability in Python, SQL, Apache Spark/PySpark, Kafka, Delta Lake, Airflow and dbt.
• Apply AI and GenAI data technologies including Azure AI Search, vector databases, embedding models, semantic search, RAG frameworks, MLflow and feature stores.
• Experience with Azure OpenAI, enterprise LLM platforms and agent architectures is highly valuable. Deep expertise is expected across a relevant enterprise-scale stack, rather than every technology listed.

Data Governance, Security & Responsible AI

• Embed data governance, privacy, security and regulatory controls into data engineering architecture.
• Implement appropriate access controls for sensitive and confidential information.
• Ensure lineage, traceability and auditability across AI data pipelines.
• Ensure AI solutions retrieve and process only appropriately authorised data.
• Partner with Cybersecurity, Risk, Compliance, Data Governance and AI Governance teams, while promoting Responsible AI engineering practices.
• Experience handling PII, sensitive customer information, data residency and regulatory requirements is highly valuable.

Data Quality & Observability

• Define enterprise standards for AI data quality and automated