Data Engineer MS Fabrics
Posted Updated 3
views
Summary
Sydney-based data engineer role building end-to-end pipelines in Microsoft Fabric (Data Factory, Lakehouse/OneLake, Warehouse) using PySpark, SQL/T-SQL, with Power BI semantic models, Purview governance, and Azure DevOps/Git CI/CD.
- Design, build, and maintain end-to-end data pipelines using Microsoft Fabric Data Engineering and Data Factory.
- Develop and manage Lakehouse and One Lake architectures for structured and unstructured data.
- Ingest data from multiple sources (databases, APIs, files, streaming sources) into Fabric.
- Implement ETL/ELT processes using Spark (Pyspark) and SQL.
- Design and optimize data models to support analytics and reporting workloads.
- Develop and maintain Fabric Data Warehouse solutions using T-SQL.
- Ensure data quality, reliability, and consistency through validation and monitoring.
- Optimize pipeline performance, storage, and capacity utilization in Fabric.
- Implement security, access control (RBAC), and data sensitivity labels.
- Enable and support Power BI semantic models and downstream reporting use cases.
- Monitor, troubleshoot, and resolve data pipeline and performance issues.
- Implement data governance, lineage, and cataloguing using Microsoft Purview.
- Support CI/CD and DevOps practices for analytics and data engineering workloads.
- Collaborate with architects, analysts, and stakeholders to translate business requirements into scalable data solutions.
Skill Experience requirement:
- Design enterprise data architecture leveraging Fabric components (Lakehouse, Warehouse, Dataflows Gen2, Pipelines, OneLake).
- Build and optimise Fabric Lakehouse tables, Warehouses, Pipelines, and Dataflows Gen2.
- Define data modelling standards, governance frameworks, and security patterns.
- Develop reusable ETL/ELT frameworks using PySpark, SQL, and Fabric notebooks.
- Manage CI/CD workflows via Azure DevOps and Git.
- Architect scalable, cost-efficient data solutions integrating Fabric with Azure/enterprise systems.
- Apply best practices for performance tuning, cost optimisation, and data quality.
- Provide technical leadership and guidance to engineering teams.
- Expertise in Delta Lake, Fabric Data Engineering, ADLS, PySpark/SQL.
- Strong understanding of data modelling and integration with Power BI datasets.
- Strong understanding of metadata management, Purview, access control, and FinOps.
- Certifications: DP-203, Fabric Analytics Engineer (preferred).