Senior Data Engineer - Azure Databricks

Senior Data Engineer - Azure Databricks & Microsoft Fabric

Role Overview:

We are looking for an experienced Senior Data Engineer with strong hands-on expertise in Python, PySpark, SQL, Azure Databricks, and Microsoft Fabric. The candidate will be responsible for designing and maintaining scalable batch and real-time data pipelines, developing lakehouse architectures, and enabling reliable data solutions for analytics and AI/ML applications.

Key Responsibilities:

Data Pipeline & Lakehouse Engineering:

- Design, implement, and maintain high-performance batch and real-time streaming data pipelines using Python, PySpark, and Spark SQL. Build multi-tier Medallion architectures (Bronze, Silver, and Gold) using Delta Lake principles.

Data Platform Delivery:

- Leverage OneLake, Direct Lake, Lakehouses, and Data Factory Gen2 to streamline data ingestion, optimise semantic models, and reduce redundant ETL processes.

API & Systems Integration:

- Build resilient API connectors, Change Data Capture (CDC) workflows, and data integration pipelines to ingest data from multiple heterogeneous sources.

AI & Analytics Enablement:

- Structure, clean, and optimise lakehouse data layers for Power BI reporting and AI/ML applications, ensuring data is validated and structured for AI grounding.

Data Quality & Governance:

- Implement data validation, access controls, data lineage, and metadata management using Unity Catalog, platform governance capabilities, and automated testing frameworks.

Technical Mentorship & CI/CD:

- Conduct code reviews, enforce engineering best practices, optimise query performance, and manage automated deployment pipelines.

Required Technical Skills:

- Advanced hands-on proficiency in Python, PySpark, and SQL, including Spark SQL and T-SQL.

- Strong experience with Azure Databricks, including Delta Lake, Unity Catalog, Delta Live Tables, Auto Loader, and Genie.

- Demonstrated experience developing data pipelines and workspace components within Microsoft Fabric.

- Expertise in Azure cloud services, including ADLS Gen2, Azure Data Factory (ADF), Azure Key Vault, and Azure SQL / Azure Synapse.

- Strong understanding of data engineering requirements for AI/ML workloads, including knowledge of vector embeddings, RAG data ingestion, and feature stores.

Education:

- Bachelor's or Master's degree in Computer Science, Information Technology, or a related quantitative field.