Principal Data Engineer (Databricks)
Summary
Contract Principal Data Engineer leading transformation of a global energy trading firm's Data Exchange platform: owning data/platform architecture and delivering the Azure Databricks platform (batch and streaming ingestion, data modelling, Azure integration) while setting engineering standards and landing AI/automation opportunities. Initial 6-month contract, hybrid in London.
Programme: Data Exchange Platform
Industry: Commodities / Energy Trading
Location: London (Hybrid)
Contract: Initial 6-month contract (long-term transformation programme)
X4 Technology are partnered with a global energy trading firm on a major transformation of its Data Exchange platform - the technical data backbone supporting physical commodities trading on a global scale.
The platform automates, reconciles and standardises transaction workflows across physical crude oil and refined petroleum products, giving traders a single, trusted source of truth across the entire ecosystem.
Joining as contract Principal Data Engineer, you'll play a pivotal role in the transformation - combining deep Azure Databricks expertise with genuine technical ownership across data architecture, streaming, integration and platform engineering.
Responsibilities for the Principal Data Engineer (Databricks)
- Own end-to-end data and platform architecture, shaping scalable, secure and high-performance solutions across multiple workstreams
- Lead design/delivery of the Databricks data platform - batch and real-time ingestion, streaming, data modelling, integration across Azure
- Set technical direction and engineering standards, driving best practice on performance, scalability, governance, security, testing and cost optimisation
- Be the go-to across Architects, Engineers, Programme Managers, Business Analysts and Trading stakeholders
- Spot and land pportunities for AI and intelligent automation - MLflow, Azure AI Foundry, Agent Bricks and Microsoft Copilot
Requirements for the Principal Data Engineer (Databricks)
- Extensive hands-on Azure Databricks expertise with a strong track record designing and delivering enterprise-scale data platforms (PySpark, Python, SQL, Delta Lake, Structured Streaming, DLT)
- Proven ownership of data platform architecture - data modelling, integration, governance, security, performance and cost optimisation
- Experience leading complex, greenfield or large-scale data transformation programmes across multiple workstreams and teams
- Strong real-time data and integration experience, including streaming architectures and complex data flows (Kafka beneficial)
- Excellent technical stakeholder management and influencing
High-profile seat at the heart of a global trading data ecosystem. You'll combine hands-on data engineering with real technical influence, shaping a highly secure, cloud-native data platform that connects trading desks, physical assets and external partners
Initial 6-month contract with multi-year programme scope, £850-950/day (inside IR35), hybrid in London. Interviews are being scheduled immediately, with a view to onboarding as soon as possible.