Associate Delivery Manager - Data Engineer
Posted Updated
Description
Job Summary
To effectively design, develop, and manage data solutions using ETL technologies such as Azure Databricks (ADB), Azure Data Factory (ADF) and SQL along with leading 3 to 5 members of developers
Duties and Responsibilities - Lead the end-to-end design, development, and delivery of scalable data engineering solutions using Azure Databricks, ADF, PySpark, SQL, and Delta Lake.
- Convert business requirements into robust technical designs, architecture documents, data models, and implementation plans.
- Own technical delivery of data integration, ETL, lakehouse, semantic layer, and AI/BI enablement initiatives.
- Guide and mentor data engineers on coding standards, design best practices, performance optimization, and reusable framework development.
- Review technical designs, code, pipelines, and deployment plans to ensure quality, scalability, maintainability, and compliance.
- Drive architecture decisions for batch and near-real-time data pipelines across Bronze, Silver, and Gold layers.
- Ensure data quality, reconciliation, anomaly detection, and timely resolution of production issues through effective RCA and permanent fixes.
- Optimize data pipelines, Databricks jobs, SQL queries, and storage usage to improve performance and reduce cost.
- Implement CI/CD practices, version control, automated deployments, and environment management across Dev, QA, and Production.
- Collaborate with PMO, business stakeholders, BI teams, InfoSec, DevOps, and external partners for smooth project execution.
- Establish SOPs, engineering standards, reusable components, monitoring frameworks, and documentation practices.
- Track delivery progress, manage technical dependencies, prioritize work, and ensure timely closure of project milestones.
- Support adoption of modern data platforms, semantic modeling, metrics layer design, and GenAI/BI capabilities.
- Ensure compliance with data governance, security, access control, audit, and enterprise data management standards.
- Act as the technical escalation point for critical issues, complex solutioning, and cross-team dependency resolution.
- Define semantic layer design and metric definitions
- Prioritize data vs AI optimization trade-offs
- Handle production issues with RCA and long-term fixes
- Drive architectural decisions for lakehouse + Data integration
- Ensuring Data Delivery within TAT
- Driving adoption of GenAI-based BI over traditional dashboards
- Balancing performance, cost, and scalability
- Managing dependencies across data engineering, AI, and business teams
Must Have
- Azure Databricks - PySpark, SQL, Delta Lake
- Strong experience in Semantic Modeling & Metrics Layer design
- Hands-on with Databricks workflows
- Pyspark (Pandas, PySpark, FastAPI)
- Azure Data Factory (ADF) for ETL pipelines
- Strong SQL and data modeling skills
Good to Have
- Cosmos DB / MongoDB (NoSQL concepts)
- Azure Data Explorer (KQL)
- Databricks Lakehouse - PySpark, SQL, Delta Lake
- AI for BI - Databricks Genie, Genie Rooms, Instructions, Agents
- ETL & Orchestration - Azure Data Factory
- Programming - Pyspark
- Cloud Platform - Azure (Preferred)
- DevOps - CI/CD Pipelines, Git