Data Engineer (Databricks)
Summary
Data Engineer building a scalable Azure Databricks platform for a global energy trading firm. The role involves migrating legacy Oracle systems to a cloud-native environment using PySpark, SQL, Delta Lake, and Structured Streaming to support real-time analytics and risk reporting.
Contract: Initial 6-month contract (long-term transformation programme)
X4 Technology are partnered with a leading global energy trading organisation on a major tech transformation - migrating from a legacy ETRM platform onto Endur, and modernising its enterprise data platform with Azure Databricks.
As part of this multi-year programme, Oracle-based data platforms are being rebuilt into a scalable, cloud-native Azure Databricks platform supporting Endur, global reporting, analytics and risk - cutting critical data latency from 10–15 minutes down to near real-time.
They're after an experienced Data Engineer to help design and build that platform hands-on - combining deep engineering with real technical ownership, shaping standards, performance and future architecture along the way.
Responsibilities of the Data Engineer (Azure Databricks)
- Design, build and optimise scalable Azure Databricks pipelines using PySpark, SQL, Delta Lake and Structured Streaming
- Lead the migration of legacy Oracle platforms into Azure Databricks, delivering high-performance solutions for trading, reporting and analytics
- Drive engineering best practice across performance, scalability, CI/CD, testing, governance and cloud cost
- Work closely with architects, platform teams and stakeholders to shape solutions, mentor engineers and influence platform direction
Requirements for the Data Engineer (Azure Databricks)
- Extensive hands-on experience delivering enterprise-scale Azure Databricks platforms - strong PySpark, Python, SQL, Delta Lake and Structured Streaming (Delta Live Tables a bonus)
- Strong grasp of modern cloud data architecture, modelling, performance optimisation, governance and best practice
- Experience on large-scale cloud migration or platform modernisation programmes, with the ability to influence direction and mentor others
- Energy trading, commodities, capital markets or financial services background - a plus, not a must
A brilliant chance to work on one of the biggest Azure Databricks transformations in energy trading, helping build a modern cloud platform underpinning trading, risk, reporting and analytics - with real influence over technical decisions while staying hands-on in a high-performing team.
Interviews moving fast, with a view to onboarding ASAP.