Senior Data Engineer: Databricks & Delta
Summary
Contract role designing scalable data pipelines on Databricks and Delta Lake for an energy trading company in London — building real-time streaming analytics, ingesting SAP data via Qlik Replicate and ADF, and optimizing data models on Azure for reporting and decision-making.
Overview
In this role you will design and implement scalable data pipelines and enable real-time and batch analytics for a leading energy trading company. You will work with cross-functional teams to build robust data solutions that support decision-making and reporting at scale. The position focuses on Databricks, Delta Lake, streaming architectures, and SAP data ingestion, offering an opportunity to impact data-driven decisions across the organization. This contract role combines hands-on engineering with shaping data platforms to support energy trading operations.
Responsibilities
- Design and implement scalable data pipelines using Databricks and Delta Live Tables (DLT)
- Develop and manage streaming data solutions for real-time analytics
- Ingest data from SAP systems using Qlik Replicate and ADF to ensure data quality
- Perform data modelling and optimize data structures for analytics and reporting
- Collaborate with cross-functional teams to understand data requirements and deliver robust solutions
- Monitor, troubleshoot, and enhance data workflows for performance and reliability
- Maintain technical documentation and contribute to data engineering best practices
Key requirements
- Strong experience with Databricks- DLT, Delta Lake concepts, UC governance
- Solid understanding of streaming technologies (e.g., Spark Structured Streaming, Autoloader)
- Programming skills in Python, SQL, or Scala
- Proficiency in data modelling, ETL/ELT processes, and data architecture
- Experience in MS Copilot, AI Foundry, Databricks (MosiacAI, MLflow, Agentbricks, Genie)
- Strong analytical background with problem-solving skills
- Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing
- Hands-on expertise in ADF, and Qlik Replicate for data ingestion and replication
- Experience working in Azure cloud environments
- collaboration with cross-functional teams
- problem-solving and analytical thinking
- attention to data quality and governance
- Databricks (DLT, Delta Lake)
- Streaming technologies (Spark Structured Streaming, Autoloader)
- Python, SQL, or Scala