Microsoft Data Engineer / Consultant
Summary
Design and build scalable data pipelines on Databricks and Azure Fabric, implementing governance, IaC, and CI/CD for secure, high-quality data solutions.
DATABRICKS CERTIFICATION IS ESSENTIAL & REQUIRED FOR THIS ROLE!!
IT IS ABSOLUTLEY ESSENTIAL THAT YOU HAVE SOLID CLIENT FACING DATABRICKS & DLT
We are looking for 3 Senior Azure candidates with expertise within Databricks to join our clients teams.
We have open Databricks roles with 3 different organisations.
We have 2 roles both on-site and 1 role which is hybrid.
Assignments Summary
Design, develop, and maintain data platforms and governance frameworks with a strong focus on Databricks and Fabric, ensuring robust and secure data pipelines.
Main Responsibilities
- Drive improvements of the Databricks platform setup for scalable and secure data solutions.
- Design and implement governance models, catalogs, and metadata management.
- Work with Databricks and related services including Delta Live Tables and Lakeflow connections.
- Apply data modelling principles for analytical and operational workloads, including star and snowflake schemas.
- Implement and manage Infrastructure as Code (IaC) solutions for Databricks.
- Collaborate with stakeholders to translate requirements into governed data products.
Key Requirements
- Bachelor's or Master's in Computer Science, Data Engineering, or equivalent experience.
- Proficiency in Python, SQL, and Terraform (or equivalent IaC tools).
- Fabric an advantage
- Deep expertise in the Databricks platform, including Delta Lake and Unity Catalog.
- Experience in Infrastructure as Code, automation, and CI/CD pipelines.
- Knowledge of cloud technologies, preferably AWS.
- Strong background in modeling structured and unstructured data.
- A Grenadier attitude.
Infrastructure and DevOps
- Lead and enforce the use of source control systems such as Git or Azure DevOps in line with company standards.
- Design, implement, and maintain CI/CD pipelines and environment management processes for data engineering solutions.
- Implement and maintain automated testing and data validation for pipelines.
- Lead performance testing and optimisation activities for data engineering solutions.
Testing and Quality
- Design, write, and review unit and integration tests to ensure solution quality and reliability.
- Document issues and resolutions accurately for review•
- Lead and oversee testing processes to validate data accuracy, performance, and quality.
- Ensure solutions are delivered in line with quality and documentation standards.
- Support knowledge sharing and improvement of internal frameworks and templates.
Nice to Have
- Familiarity with high-performance data processing pipelines.
- Experience with data visualisation tools.