Senior Data Engineer

Position Summary We are seeking a Senior Microsoft Purview Developer / Data Governance Engineer with 5-10 years of experience in Data Governance, Metadata Management, Data Engineering, and Enterprise Data Catalog implementations. This role will be responsible for the implementation of Microsoft Purview as part of an enterprise-wide migration from Collibra to Microsoft Purview. The candidate will design and implement metadata ingestion, governance workflows, lineage solutions, custom integrations, automation frameworks, and business metadata migration across Azure, Databricks, BigQuery, D365, Workday, Fabric, Power BI, and other enterprise platforms. Required Experience: -5-10 years of experience in Data Engineering, Data Governance, Metadata Management, or Data Architecture. -3+ years of hands-on Microsoft Purview implementation experience. -Experience implementing enterprise data catalog and governance platforms. -Experience supporting large-scale metadata management and governance initiatives. -Experience working in Azure-based data platforms and cloud environments. -Strong stakeholder engagement skills with Data Architects, Data Engineers, Business Stewards, and Domain Data Owners. Microsoft Purview Skills: Purview Data Map: -Source registration and scan configuration. -Metadata ingestion and technical metadata management. -Classification and sensitivity labeling. -Collection hierarchy design and administration. -Metadata relationship management. -Scan scheduling and automation. -Lineage configuration and management. -Purview administration and performance optimization. Purview Unified Catalog: -Business Glossary implementation. -Governance Domains and ownership models. -Data Products. -Asset Certification. -Stewardship enablement. -Data Curation. -Business metadata management. Workflow Automation: Experience configuring and supporting governance workflows including: -Business Term Approval -Acronym Approval -Critical Data Element (CDE) Approval -Data Product Approval -Asset Certification -Data Access Request Approval -Metadata Change Requests -Stewardship Task Management Data Engineering Skills: -Strong understanding of modern Data Lake and Lakehouse architectures. -Experience with Azure Data Lake Storage Gen2 (ADLS). -Strong knowledge of Medallion Architecture (Bronze, Silver, Gold layers). -Hands-on experience with Databricks and Unity Catalog. -Experience with BigQuery metadata and governance integration. -Experience with Microsoft Fabric and Power BI semantic models. -Strong SQL, data modeling, and metadata management skills. -Understanding of ETL/ELT pipelines and data transformation processes. -Ability to trace end-to-end lineage across source systems, data platforms, semantic models, and reporting layers. -Familiarity with Azure Data Factory, Fabric Data Factory, or similar orchestration platforms. Metadata Migration & Governance -Experience migrating metadata from Collibra or similar governance platforms. -Experience working with: -Business Terms -Acronyms -Critical Data Elements (CDEs) -KPI Catalogs -Data Products -Report Catalogs -Policies and Governance Artifacts -Experience creating metadata mapping and migration frameworks. -Experience performing migration reconciliation and validation. -Understanding of Apache Atlas metadata models and relationships. Integration & Lineage Engineering Experience integrating and governing metadata across: -Azure Data Lake Gen2 -Databricks Unity Catalog -BigQuery -Microsoft Fabric -Power BI -Dynamics 365 ERP -Dynamics 365 CE -Workday -Braze Strong experience with: -Apache Atlas APIs -Microsoft Purview APIs -Metadata relationship modeling -Lineage engineering -Custom lineage ingestion -End-to-end lineage design and validation Programming & Automation Strong hands-on experience with: -Python -Requests -Pandas -PySpark -JSON processing -API integration frameworks Automation -PowerShell -REST APIs -OAuth2 Authentication -Microsoft Graph APIs -Purview APIs -Apache Atlas APIs Ability to develop: -Metadata ingestion utilities -Migration automation scripts -API wrappers -Bulk metadata loading frameworks -Reconciliation and validation scripts Azure & Security Experience with: -Microsoft Entra ID -RBAC and Access Management -Managed Identities -Service Principals -Azure Key Vault -Azure Networking Concepts -Private Endpoints -Security and Compliance Best Practices Reporting & Dashboarding Experience building governance dashboards using: -Microsoft Fabric -Power BI Metrics may include: -Asset Inventory -Stewardship Metrics -Workflow Status -Certification Status -Governance KPIs -Metadata Completeness -Domain Health Metrics Preferred Certifications -Microsoft Certified: Azure Data Engineer Associate -Microsoft Certified: Azure Administrator Associate -Microsoft Certified: Fabric Analytics Engineer Associate -Microsoft Purview Certification (Preferred) -Collibra Certification (Preferred) Key Responsibilities -Implement and configure Microsoft Purview Data Map and Unified Catalog. -Configure metadata scans across ADLS, Databricks, BigQuery, D365, Fabric, Power BI, Workday, Dynamics CE, and Braze. -Develop custom metadata ingestion and integration solutions where native connectors are unavailable. -Migrate business and technical metadata from Collibra to Purview. -Configure governance workflows, stewardship processes, and approval mechanisms. -Enable and validate end-to-end lineage across enterprise data platforms. -Develop automation scripts, APIs, and metadata migration utilities. -Build governance dashboards and operational reporting. -Support testing, reconciliation, go-live, and post-production stabilization. -Create technical documentation, operational procedures, and knowledge transfer materials.