Senior Data Engineer : Databricks & Delta
Summary
Senior Data Engineer at a world-leading energy trading company, designing scalable pipelines with Databricks and Delta Live Tables, building streaming and batch analytics, and ingesting SAP data via Qlik Replicate and ADF in Azure.
This is an exciting role for a Senior Data Engineer with strong Databricks and Delta to join a world-leading Energy Trading company.
Role Description:
We are looking for a highly skilledSenior Data Engineerwith strong analytical capabilities and hands-on experience in building scalable data solutions.
The ideal candidate will have expertise inDatabricks,streaming architectures, data ingestion frameworks, anddashboarding, along with experience working withSAP data sources.
You will be responsible for designing and implementing scalable data pipelines and enabling real-time and batch analytics across the organization.
Key Responsibilities:
- Design and implement scalable data pipelines usingDatabricks and Delta Live Tables (DLT)
- Develop and managestreaming data solutionsfor real-time analytics and decision-making
- Ingest data from various sourcesincludingSAP systems using Qlik ReplicateandADFto ingest data from various sources ensuring data quality and consistency
- Performdata modellingand optimize data structures for analytics and reporting
- Collaborate with cross-functional teams to understand data requirements and deliver robust solutions
- Monitor, troubleshoot, and enhance data workflows for performance and reliability
- Maintain technical documentation and contribute to data engineering best practices
Required Skills & Qualifications:
- Strong experience withDatabricks- DLT, Delta Lake concepts, UC governance
- Solid understanding ofstreaming technologies(e.g., Spark Structured Streaming, Autoloader)
- Programming skills inPython,SQL, orScala
- Proficiency indata modelling, ETL/ELT processes, and data architecture
- Experience in MS Copilot, AI Foundry , Databricks (MosiacAI, MLflow, Agentbricks, Genie)
- Strong analytical background with problem-solving skills
- Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing
- Hands-on expertise inADF, and Qlik Replicatefor data ingestion and replication
- Experience working inAzure cloud environments