Data Engineer - Databricks & Microsoft FabricData Engineer - Databricks & Microsoft Fabric
Summary
Build and maintain Microsoft Fabric Lakehouse solutions and scalable data pipelines using Azure Databricks and Delta Lake. Responsibilities include developing ELT workflows, optimizing Spark jobs, integrating Power BI, and implementing governance and CI/CD processes.
- Build Microsoft Fabric Lakehouse solutions
- Build lakehouse Bronze Silver Gold data models
- Create runbooks and deployment guides
- Define data quality rules null duplicate referential cross field
- Design scalable data pipelines
- Develop Azure Databricks notebooks jobs workflows
- Develop Fabric SQL semantic models
- Develop Fabric data pipelines
- Develop batch ingestion pipelines
- Develop incremental ingestion pipelines
- Implement CI CD for notebooks pipelines SQL scripts
- Implement Delta Lake features
- Implement ELT/ETL workflows
- Implement data quality validation reconciliation exception handling
- Implement governance and access controls
- Implement partitioning caching indexing
- Integrate Microsoft Purview
- Integrate Power BI
- Maintain data lineage metadata documentation
- Migrate data platforms to Microsoft Fabric
- Monitor production jobs and resolve incidents
- Optimize Spark jobs
- Orchestrate data pipelines in Fabric
- Prepare support documentation
- Troubleshoot pipeline failures and bottlenecks
- Use Git version control and deployment automation