Senior Data Engineer : Databricks & Delta
Summary
Contract Senior Data Engineer role in London for a leading energy trading company, designing scalable batch and real-time data pipelines using Databricks, Delta Lake/DLT, and streaming, plus SAP data ingestion via ADF and Qlik Replicate to power analytics and reporting at scale.
In this role you will design and implement scalable data pipelines and enable real-time and batch analytics for a leading energy trading company. You will work with cross-functional teams to build robust data solutions that support decision-making and reporting at scale. The position focuses on Databricks, Delta Lake, streaming architectures, and SAP data ingestion, offering an opportunity to impact data-driven decisions across the organization. This contract role combines hands-on engineering with shaping data platforms to support energy trading operations.
Responsibilities- Design and implement scalable data pipelines using Databricks and Delta Live Tables (DLT)
- Develop and manage streaming data solutions for real-time analytics
- Ingest data from SAP systems using Qlik Replicate and ADF to ensure data quality
- Perform data modelling and optimize data structures for analytics and reporting
- Collaborate with cross-functional teams to understand data requirements and deliver robust solutions
- Monitor, troubleshoot, and enhance data workflows for performance and reliability
- Maintain technical documentation and contribute to data engineering best practices
- Strong experience with Databricks- DLT, Delta Lake concepts, UC governance
- Solid understanding of streaming technologies (e.g., Spark Structured Streaming, Autoloader)
- Programming skills in Python, SQL, or Scala
- Proficiency in data modelling, ETL/ELT processes, and data architecture
- Experience in MS Copilot, AI Foundry, Databricks (MosiacAI, MLflow, Agentbricks, Genie)
- Strong analytical background with problem-solving skills
- Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing
- Hands-on expertise in ADF, and Qlik Replicate for data ingestion and replication
- Experience working in Azure cloud environments
- collaboration with cross-functional teams
- problem-solving and analytical thinking
- attention to data quality and governance
- Databricks (DLT, Delta Lake)
- Streaming technologies (Spark Structured Streaming, Autoloader)
- Python, SQL, or Scala