Senior Data Engineer - Microsoft Fabric and Microsoft Foundry
Summary
Build and maintain data pipelines in Microsoft Fabric and Azure to feed analytics, AI, and agent-based solutions via Microsoft Foundry for a contract role with potential to convert to permanent.
We are seeking an experienced Senior Data Engineer for an initial three-month contract, with strong long-term potential and the opportunity to transition into a permanent position for the right person.
This position will suit a highly capable Data Engineer who combines strong Microsoft Azure and Fabric experience with an understanding of how reliable, governed data is prepared and delivered to analytics, artificial intelligence and agent-based solutions through Microsoft Foundry.
The opportunity:
You will work closely with architects, developers, analysts and business stakeholders to build scalable data pipelines, lakehouse and warehouse solutions, and trusted enterprise data products. A major focus will be integrating structured and unstructured organisational data with Microsoft Foundry to support generative AI, retrieval-augmented generation and intelligent agent solutions.
The initial contract will provide both parties with the opportunity to establish a successful working relationship, with a genuine pathway into an ongoing permanent position.
Key responsibilities:
This position will suit a highly capable Data Engineer who combines strong Microsoft Azure and Fabric experience with an understanding of how reliable, governed data is prepared and delivered to analytics, artificial intelligence and agent-based solutions through Microsoft Foundry.
The opportunity:
You will work closely with architects, developers, analysts and business stakeholders to build scalable data pipelines, lakehouse and warehouse solutions, and trusted enterprise data products. A major focus will be integrating structured and unstructured organisational data with Microsoft Foundry to support generative AI, retrieval-augmented generation and intelligent agent solutions.
The initial contract will provide both parties with the opportunity to establish a successful working relationship, with a genuine pathway into an ongoing permanent position.
Key responsibilities:
- Design, develop and maintain scalable data pipelines across Microsoft Azure and Microsoft Fabric.
- Build modern lakehouse, warehouse and data integration solutions using Fabric, OneLake, Azure Data Factory, Azure Data Lake and related Microsoft technologies.
- Develop robust data transformation processes using SQL, Python and PySpark.
- Prepare and optimise structured and unstructured data for analytics, generative AI and Microsoft Foundry solutions.
- Integrate enterprise data with Microsoft Foundry models, agents, knowledge sources and APIs.
- Support retrieval-augmented generation solutions using Azure AI Search, vector indexing, embeddings and enterprise knowledge repositories.
- Implement data quality, lineage, security, monitoring and governance controls.
- Work with Microsoft Purview, Entra ID and role-based access controls to protect sensitive information.
- Establish automated deployment and testing processes using Azure DevOps, Git and CI/CD.
- Troubleshoot performance, reliability and data-quality issues across complex production environments.
- Translate business requirements into practical, scalable and supportable technical solutions.
- Extensive commercial experience as a Data Engineer within Microsoft-based enterprise environments.
- Strong hands-on experience with Microsoft Fabric, Azure Data Factory, OneLake, Azure Data Lake, Synapse Analytics or Azure Databricks.
- Advanced SQL skills and commercial experience with Python or PySpark.
- Experience designing data pipelines, lakehouse architectures, data warehouses and dimensional models.
- Practical experience with Microsoft Foundry, formerly Azure AI Foundry, or closely related Azure AI services.
- Understanding of generative AI, AI agents, RAG, vector databases, embeddings and enterprise search.
- Experience integrating data platforms with APIs, applications and cloud-based services.
- Strong knowledge of data security, governance, privacy and production support practices.
- Excellent communication skills and the ability to work effectively with technical teams and business stakeholders.
- Experience with Microsoft Foundry Agent Service and Foundry IQ.
- Microsoft Fabric, Azure Data Engineering or Azure AI certifications.
- Experience with Power BI, Microsoft 365 Copilot, Power Platform or Microsoft Purview.
- Knowledge of infrastructure as code using Bicep or Terraform.
- Experience delivering AI-enabled data solutions within a large, regulated or complex organisation.