Senior Data Engineer | Azure & Data Forensics
Summary
Senior Data Engineer builds and optimizes Azure-based data pipelines for a NSW government department, reverse-engineering legacy systems and cleaning messy data using Medallion Architecture and Power BI.
Our client is a large NSW Government department delivering extensive education service. This opportunity has come up due to additional project scope and a critical need for extra technical data engineering resourcing.
Role Description:
The Senior Data Engineer will operate within a complex data integration environment where the source data has known structural and quality issues. This is not a standard greenfield development role; you will act as a \"data detective\"—looking under the hood of legacy systems, systematically diagnosing problems within undocumented extracts, and building clean modern pipelines.
Responsibilities:
- Diagnose and reverse-engineer complex data logic across partially defective, undocumented source datasets.
- Hands‑on pipeline development and notebook authoring (not just tool configuration) across the Azure cloud ecosystem.
- Build and optimize structured data layers utilizing the Medallion Architecture (Bronze, Silver, Gold).
- Design robust semantic layers, security parameters, and visual developments in Microsoft Power BI.
- Document technical findings and data logic clearly to a standard that withstands director‑level scrutiny.
- Facilitate stakeholder engagement, ensuring clear communication of technical issues to non‑technical stakeholders and the tech team (BAs, Warehousing, and Engineers).
Skills and Experience:
- 5+ years of dedicated Data Engineering experience.
- Core Azure Stack: Strong hands‑on experience with Azure Synapse Analytics, Microsoft Fabric, Azure SQL Server, Azure Data Lake Storage (ADLS), and Azure Key Vault.
- Expert Coding: Master‑level SQL (complex transformations and data analysis) combined with strong Python and PySpark notebook authoring skills.
- Data Modelling: Complete understanding of dimensional modelling techniques (facts/dimensions) and how to implement them.
- Power BI: Solid understanding of semantic layers, row‑level security, and DAX.
- Data Privacy: Proven knowledge of managing PII (Personally Identifiable Information) data within a large corporate environment. Prior government experience is not required.
- Working knowledge of Microsoft Dynamics CRM and Dataverse (GUID generation patterns and common CRM data quality issues) is highly regarded.
Keywords: Azure Synapse, Microsoft Fabric, PySpark, Python, Data Engineer, Senior Data Engineer, Government, SQL, Power BI, DAX, Medallion Architecture, Dimensional Modelling, Data Forensics, Dataverse, Dynamics CRM.