IN_Senior Associate_Data Engineer_Digital Integration_Advisory_Kolkata

<p></p><table><tbody><tr><td><p><strong>Job Position Title:</strong></p></td><td><p>Senior Associate</p></td></tr><tr><td><p><strong>Responsibilities:</strong></p><p> </p></td><td><ul><li><span style="font-size:inherit;">Architecture & Platform Design</span><ul><li><span style="font-size:inherit;">Define end-to-end solution architecture for enterprise analytics platforms on Microsoft Fabric and Azure - ingestion, storage, compute, serving, and consumption layers — with clear rationale and documented trade-offs.</span></li><li><span style="font-size:inherit;">Design workspace topology, capacity strategy, and environment separation (dev/test/prod), including naming standards, item organization, and promotion paths.</span></li><li><span style="font-size:inherit;">Establish platform-level design patterns: medallion layering standards, hot-path vs. cold-path (Lambda-style) architectures, metadata/control-table-driven orchestration frameworks, and reusable pipeline templates.</span></li><li><span style="font-size:inherit;">Make and defend technology selection decisions (e.g., Warehouse vs. Lakehouse, Eventhouse vs. Warehouse for a workload, ADF vs. Fabric pipelines, native vs. custom alerting) based on cost, latency, governance, and skills fit.</span></li><li><span style="font-size:inherit;">Design for security and compliance by default: Entra ID RBAC models, workspace/item-level permissions, service-principal boundaries, network and data-residency considerations, and Purview-integrated governance.</span></li><li><span style="font-size:inherit;">Define non-functional requirements and design against them — scalability, capacity/CU consumption, refresh SLAs, disaster recovery, and observability/monitoring architecture.</span></li><li><span style="font-size:inherit;">Produce and maintain architecture artifacts: high-level and detailed design documents, data-flow diagrams, decision records (ADRs), and review-board presentations.</span></li><li><span style="font-size:inherit;">Conduct design and code reviews, set engineering standards for the team, and guide build teams through implementation of the approved architecture.</span></li></ul></li></ul><p style="margin-left:1in;"> </p><p style="margin-left:1in;"> </p><ul><li><span style="font-size:inherit;">Data Platform Engineering (Microsoft Fabric)</span><ul><li><span style="font-size:inherit;">Design and implement medallion architectures (Bronze/Silver/Gold) using Fabric Lakehouses, Data Warehouses, and OneLake with Delta/Parquet as the storage foundation.</span></li><li><span style="font-size:inherit;">Build and orchestrate Fabric Data Pipelines and Notebooks (PySpark/Python) for batch ELT, including metadata-driven/control-table-based orchestration frameworks.</span></li><li><span style="font-size:inherit;">Develop Real-Time Intelligence (RTI) solutions: Eventstreams for streaming ingestion, Eventhouse/KQL databases for hot-path analytics, materialized views, and update policies.</span></li><li><span style="font-size:inherit;">Implement Data Activator (Activator) rules and alerting — threshold, anomaly, and spike detection with actions into Microsoft Teams, email, and downstream systems.</span></li><li><span style="font-size:inherit;">Write performant T-SQL against Fabric Warehouses and SQL analytics endpoints, and KQL against Eventhouses.</span></li></ul></li></ul><p style="margin-left:1in;"> </p><ul><li><span style="font-size:inherit;">Semantic Modeling & BI</span><ul><li><span style="font-size:inherit;">Design and maintain enterprise semantic models (star schemas, DAX measures, incremental refresh, RLS) consumed by Power BI reports and dashboards.</span></li><li><span style="font-size:inherit;">Build business-facing reports (consumption analytics, operational monitoring, KPI dashboards) and manage their lifecycle across dev/test/prod workspaces.</span></li><li><span style="font-size:inherit;">Prevent and remediate model/report drift through source-controlled definitions (TMDL/PBIP).</span></li></ul></li></ul><p style="margin-left:1in;"> </p><ul><li><span style="font-size:inherit;">Azure Integration & Migration</span><ul><li><span style="font-size:inherit;">Integrate Fabric with the wider Azure estate: Azure Data Factory (including ADF-to-Fabric pipeline migration), Azure Data Lake Storage, Azure Key Vault, Azure Functions, and Azure OpenAI for AI-enriched data workloads.</span></li><li><span style="font-size:inherit;">Migrate legacy ETL frameworks (ADF, SSIS, Synapse) into Fabric-native equivalents with parity validation.</span></li></ul></li></ul><p style="margin-left:1in;"> </p><ul><li><span style="font-size:inherit;">AI & Microsoft Foundry</span><ul><li><span style="font-size:inherit;">Build AI-enabled data solutions with Microsoft Foundry (Azure AI Foundry): model catalog and deployments (GPT-4o and other frontier models), prompt flow, and evaluation of model outputs for quality and safety.</span></li><li><span style="font-size:inherit;">Develop agentic and RAG (retrieval-augmented generation) workloads using Foundry Agent Service / Azure OpenAI with enterprise data sources — Fabric lakehouses, Azure AI Search indexes, and vector stores (pgvector, Eventhouse).</span></li><li><span style="font-size:inherit;">Ground AI applications in governed data: connect Foundry projects to OneLake/Fabric data, apply responsible AI practices (content filters, evaluations, red-team findings), and monitor deployed models for cost, latency, and drift.</span></li><li><span style="font-size:inherit;">Operationalize AI workloads alongside data pipelines — embedding generation, enrichment/classification steps in ELT, and LLM-as-judge evaluation harnesses — with the same CI/CD and observability rigor as the rest of the platform.</span></li></ul></li></ul><p style="margin-left:1in;"> </p><ul><li><span style="font-size:inherit;">DevOps, CI/CD & Automation</span><ul><li><span style="font-size:inherit;">Implement Git integration for Fabric workspaces and manage promotion via deployment pipelines and the Fabric REST APIs (item CRUD, service-principal automation).</span></li><li><span style="font-size:inherit;">Build headless deployment tooling in Python (e.g., workspace provisioning, Variable Library configuration, notebook/pipeline deployment) and CI/CD in Azure Pipelines or GitHub Actions.</span></li><li><span style="font-size:inherit;">Authenticate and automate using Entra ID service principals, managed identities, and least-privilege RBAC.</span></li></ul></li></ul><p style="margin-left:1in;"> </p><ul><li><span style="font-size:inherit;">Governance, Quality & Operations</span><ul><li><span style="font-size:inherit;">Implement data governance with Microsoft Purview: cataloging, lineage, classification, sensitivity labels, and glossary seeding via API/automation.</span></li><li><span style="font-size:inherit;">Establish data-quality checks, reconciliation (hot vs. cold path), monitoring, and runbooks; verify end-to-end flows with automated smoke/E2E tests.</span></li><li><span style="font-size:inherit;">Document architecture, control tables, and operational procedures to an audit-ready standard.</span></li></ul></li></ul></td></tr><tr><td><p><strong>Mandatory skill sets:</strong></p><p> </p></td><td><ul><li><span style="font-size:inherit;">5+ years of professional data engineering experience on the Microsoft data stack (Azure and/or Fabric), with at least 1–2 years hands-on with Microsoft Fabric in real project delivery.</span></li><li><span style="font-size:inherit;">Demonstrated architecture and platform design experience: designing end-to-end data platform solutions (workspace/environment topology, medallion layering, hot/cold path patterns, security and RBAC models) and producing design documents and decision records that guide build teams.</span></li><li><span style="font-size:inherit;">Strong programming skills in Python (including PySpark) for data processing and platform automation.</span></li><li><span style="font-size:inherit;">Advanced T-SQL: dimensional modeling, stored procedures, performance tuning, warehouse workload patterns.</span></li><li><span style="font-size:inherit;">Working proficiency in KQL (Kusto) for streaming/telemetry analytics.</span></li><li><span style="font-size:inherit;">Proven experience with lakehouse/medallion architecture, Delta Lake, and Parquet.</span></li><li><span style="font-size:inherit;">Hands-on Power BI / semantic model experience: star schemas, DAX, row-level security, deployment across environments.</span></li><li><span style="font-size:inherit;">Experience with Azure Data Factory or Fabric Data Pipelines for orchestration, including parameterized, metadata-driven designs.</span></li><li><span style="font-size:inherit;">CI/CD experience with Azure DevOps Pipelines or GitHub Actions, plus Git-based source control for data artifacts.</span></li><li><span style="font-size:inherit;">Experience with Entra ID (Azure AD) authentication patterns — service principals, OAuth2/MSAL, managed identities — and secure secret management (Key Vault).</span></li><li><span style="font-size:inherit;">Strong debugging and verification discipline: ability to prove a pipeline works end-to-end, reconcile data across layers, and document evidence.</span></li></ul></td></tr><tr><td><p><strong>Preferred skill sets:</strong></p><p> </p></td><td><ul><li><span style="font-size:inherit;">Azure Databricks and its associated tech stack: Spark on Databricks (batch and Structured Streaming), Delta Lake / Delta Live Tables, Unity Catalog for governance, Databricks Workflows/Jobs, notebooks and Repos, cluster/compute management, MLflow for experiment tracking, and Databricks SQL warehouses. Experience integrating or migrating between Databricks and Fabric (e.g., shared Delta tables via OneLake shortcuts, Unity Catalog mirroring) is especially valuable.</span></li><li><span style="font-size:inherit;">Real-Time Intelligence depth: Eventstream, Eventhouse, update policies, materialized views, Activator alert rules.</span></li><li><span style="font-size:inherit;">Microsoft Purview implementation experience (catalog, lineage, APIs).</span></li><li><span style="font-size:inherit;">Experience calling Fabric REST APIs for headless/programmatic deployment at scale.</span></li><li><span style="font-size:inherit;">Hands-on Microsoft Foundry (Azure AI Foundry) experience: model deployments, prompt flow, Foundry Agent Service, AI evaluations, and integrating Foundry projects with enterprise data platforms.</span></li><li><span style="font-size:inherit;">Exposure to Azure OpenAI / AI-augmented data workloads (RAG, embeddings, pgvector, or GraphRAG-style knowledge graphs) and Azure AI Search.</span></li><li><span style="font-size:inherit;">Full-stack awareness: Next.js/React/TypeScript apps consuming data APIs, MSAL browser auth.</span></li><li><span style="font-size:inherit;">PostgreSQL (pgvector, graph extensions) or other polyglot persistence experience.</span></li><li><span style="font-size:inherit;">Domain experience in energy & utilities (AMI/smart-meter data, grid events, consumption billing) or another regulated industry.</span></li><li><span style="font-size:inherit;">Testing tooling: pytest, Playwright, or similar automated verification frameworks.</span></li></ul><p style="margin-left:1in;"> </p><p><strong>Certifications (Good to Have)</strong></p><ul><li><span style="font-size:inherit;">Microsoft Fabric & Data</span><ul><li><span style="font-size:inherit;">DP-600 — Fabric Analytics Engineer Associate</span></li><li><span style="font-size:inherit;">DP-700 — Fabric Data Engineer Associate</span></li><li><span style="font-size:inherit;">DP-203 — Azure Data Engineer Associate</span></li><li><span style="font-size:inherit;">PL-300 — Power BI Data Analyst Associate</span></li></ul></li><li><span style="font-size:inherit;">Azure Architecture & AI</span><ul><li><span style="font-size:inherit;">AZ-305 — Azure Solutions Architect Expert</span></li><li><span style="font-size:inherit;">AI-102 — Azure AI Engineer Associate (Azure OpenAI / Foundry workloads)</span></li><li><span style="font-size:inherit;">AZ-204 — Azure Developer Associate</span></li><li><span style="font-size:inherit;">AZ-400 — DevOps Engineer Expert</span></li></ul></li><li><span style="font-size:inherit;">Databricks</span><ul><li><span style="font-size:inherit;">Databricks Certified Data Engineer Associate / Professional</span></li><li><span style="font-size:inherit;">Databricks Certified Associate Developer for Apache Spark</span></li><li><span style="font-size:inherit;">Databricks Certified Machine Learning Associate</span></li></ul></li><li><span style="font-size:inherit;">Fabric certifications (DP-600/DP-700) carry the most weight given the platform focus; the rest strengthen the profile but are not screening criteria.</span></li></ul></td></tr><tr><td><p><strong>Years of experience required:</strong></p></td><td><p>5 to 7 Years</p></td></tr><tr><td><p><strong>Education qualification:</strong></p></td><td><p>Bachelor's degree in Computer Science, IT, or a related field.</p></td></tr><tr><td><p><strong>Location</strong></p></td><td><p>Kolkata</p></td></tr></tbody></table>