Databricks engineer (Azure)

Azure Databricks

Mandate skills

Databricks + Azure ( Databricks + Azure Data Engineer, with strong hands-on expertise in Databricks, PySpark/SQL, and Azure data services)

Role Overview

We are looking for a hands-on Databricks + Azure Data Engineer to design, develop and optimise scalable data pipelines and data processing solutions. The ideal candidate should have strong expertise in Azure Databricks, PySpark, SQL and Azure data services, with experience working on large-scale data engineering and transformation workloads.

Key Responsibilities

• Design, build and maintain scalable data pipelines using Azure Databricks.
• Develop complex data transformations using PySpark and SQL.
• Build and optimise ETL/ELT pipelines for high-volume datasets.
• Work with Azure data services such as Azure Data Factory (ADF), ADLS Gen2, Azure SQL and Synapse Analytics.
• Implement and manage Delta Lake / Delta tables and efficient data processing patterns.
• Perform performance tuning and optimisation of Databricks jobs, Spark workloads and SQL queries.
• Integrate data from multiple structured and semi-structured sources.
• Implement data quality, validation, monitoring and error-handling mechanisms.
• Support production deployments, troubleshoot pipeline failures and resolve data-related issues.
• Collaborate with architects, analysts and business teams to translate requirements into scalable data solutions.

Mandatory Skills

• Strong hands-on experience with Azure Databricks for large-scale data processing and engineering.
• Proficiency in PySpark and SQL, including development of complex data transformations and optimised queries.
• Strong experience in building and maintaining ETL/ELT data pipelines.
• Hands-on expertise with key Azure Data Services, particularly Azure Data Factory (ADF) and Azure Data Lake Storage Gen2 (ADLS Gen2).
• Practical experience with Apache Spark and Delta Lake, including Delta tables and data processing frameworks.
• Strong understanding of data warehousing, data modelling and data integration concepts.
• Experience in performance tuning, troubleshooting and optimisation of Databricks/Spark workloads.
• Ability to independently design, develop, test and support production-grade data pipelines.