IN_Senior Associate_Data Engineer + AI_Digital Integration_Advisory_Kolkata

<p><br /></p> <p style="font-family:'Segoe UI';margin-bottom:1rem;color:rgb(0,0,0);">We are seeking an <span style="font-family:DarwinSans, 'Noto Sans Variable';font-weight:bolder;">Azure Databricks Data Engineer</span> to design, develop, and optimize scalable data platforms and analytics solutions on Microsoft Azure. The ideal candidate will have strong expertise in Azure Databricks, PySpark, Delta Lake, and data engineering best practices to build efficient, reliable, and high-performance data pipelines.</p><p style="font-family:'Segoe UI';margin-bottom:1rem;color:rgb(0,0,0);">The role involves collaborating with business stakeholders, data architects, and analytics teams to deliver enterprise-grade data solutions while ensuring data quality, governance, scalability, and performance.</p><h3 style="font-family:'Segoe UI';margin-top:0px;margin-bottom:0.5rem;color:rgb(0,0,0);">Key Responsibilities</h3><ul style="font-family:'Segoe UI';color:rgb(0,0,0);"><li style="font-family:DarwinSans, 'Noto Sans Variable';">Design, develop, and maintain scalable data pipelines using Azure Databricks.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Develop ETL/ELT workflows using PySpark, Spark SQL, and Azure Data Factory.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Build and manage Lakehouse architectures using Delta Lake and Azure Data Lake Storage (ADLS Gen2).</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Implement data ingestion frameworks for structured and unstructured data sources.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Optimize Spark jobs, cluster performance, and query execution.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Work with business and technical teams to translate requirements into data solutions.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Design and maintain data models, schemas, and data integration processes.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Ensure data quality, governance, security, and compliance standards.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Troubleshoot production issues and optimize data processing performance.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Create and maintain technical documentation and operational procedures.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Participate in Agile methodologies including Scrum, Kanban, or SAFe.</li></ul><h3 style="font-family:'Segoe UI';margin-top:0px;margin-bottom:0.5rem;color:rgb(0,0,0);">Mandatory Skill Sets</h3><h4 style="font-family:'Segoe UI';color:rgb(0,0,0);">Azure Databricks & Big Data</h4><ul style="font-family:'Segoe UI';color:rgb(0,0,0);"><li style="font-family:DarwinSans, 'Noto Sans Variable';">Strong experience with <span style="font-weight:bolder;">Azure Databricks</span>.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Hands-on expertise in <span style="font-weight:bolder;">PySpark</span>, <span style="font-weight:bolder;">Spark SQL</span>, and <span style="font-weight:bolder;">Delta Lake</span>.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Experience with Lakehouse architecture and large-scale data processing.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Understanding of Spark performance tuning and optimization techniques.</li></ul><h4 style="font-family:'Segoe UI';color:rgb(0,0,0);">Azure Data Platform</h4><ul style="font-family:'Segoe UI';color:rgb(0,0,0);"><li style="font-family:DarwinSans, 'Noto Sans Variable';">Experience with:<ul><li>Azure Data Factory (ADF)</li><li>Azure Data Lake Storage Gen2 (ADLS)</li><li>Azure SQL Database</li><li>Azure Key Vault</li></ul></li></ul><h4 style="font-family:'Segoe UI';color:rgb(0,0,0);">Data Engineering & Databases</h4><ul style="font-family:'Segoe UI';color:rgb(0,0,0);"><li style="font-family:DarwinSans, 'Noto Sans Variable';">Strong experience in ETL/ELT development.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Advanced SQL and T-SQL proficiency.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Experience with <span style="font-weight:bolder;">PostgreSQL</span> databases.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Knowledge of data warehousing concepts and dimensional modeling.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Experience in data architecture and data modeling.</li></ul><h4 style="font-family:'Segoe UI';color:rgb(0,0,0);">Programming</h4><ul style="font-family:'Segoe UI';color:rgb(0,0,0);"><li style="font-family:DarwinSans, 'Noto Sans Variable';">Strong proficiency in <span style="font-weight:bolder;">Python</span> and <span style="font-weight:bolder;">PySpark</span>.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Experience building reusable data engineering frameworks.</li></ul><h4 style="font-family:'Segoe UI';color:rgb(0,0,0);">Development Methodology</h4><ul style="font-family:'Segoe UI';color:rgb(0,0,0);"><li style="font-family:DarwinSans, 'Noto Sans Variable';">Experience working in Agile environments (Scrum, Kanban, or SAFe).</li></ul><h3 style="font-family:'Segoe UI';margin-top:0px;margin-bottom:0.5rem;color:rgb(0,0,0);">Good to Have Skill Sets</h3><ul style="font-family:'Segoe UI';color:rgb(0,0,0);"><li style="font-family:DarwinSans, 'Noto Sans Variable';">Unity Catalog</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Microsoft Purview</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Azure Fabric</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Snowflake</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Power BI</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Azure DevOps</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Structured Streaming</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">CI/CD implementation for Databricks deployments</li></ul><h3 style="font-family:'Segoe UI';margin-top:0px;margin-bottom:0.5rem;color:rgb(0,0,0);">Preferred Skill Sets</h3><ul style="font-family:'Segoe UI';color:rgb(0,0,0);"><li style="font-family:DarwinSans, 'Noto Sans Variable';">Strong hands-on experience with Azure Databricks, PySpark, Python, and SQL.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Experience building enterprise-scale Azure data platforms.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Expertise in ETL, ELT, Data Warehousing, and Lakehouse architectures.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Experience with data governance, metadata management, and lineage.</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Knowledge of performance optimization and cost management within Azure Databricks.<br /><br /></li></ul><h3 style="margin-top:0px;margin-bottom:0.5rem;color:rgb(19,19,19);"><font color="#000000">Years of Experience Required</font></h3><h3 style="margin-top:0px;margin-bottom:0.5rem;color:rgb(19,19,19);"><font color="#000000">4-7 years of exp.</font></h3><h3 style="font-family:'Segoe UI';margin-top:0px;margin-bottom:0.5rem;color:rgb(0,0,0);">Certifications / Credentials (Any One Mandatory)</h3><ul style="font-family:'Segoe UI';color:rgb(0,0,0);"><li style="font-family:DarwinSans, 'Noto Sans Variable';">Databricks Certified Data Engineer Associate</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Databricks Certified Data Engineer Professional</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Microsoft Certified: Azure Data Engineer Associate (DP-203)</li><li style="font-family:DarwinSans, 'Noto Sans Variable';">Microsoft Certified: Azure Solutions Architect Expert</li></ul><h3 style="font-family:'Segoe UI';margin-top:0px;margin-bottom:0.5rem;color:rgb(0,0,0);">Educational Qualification</h3><p style="font-family:'Segoe UI';margin-bottom:1rem;color:rgb(0,0,0);">Bachelor's Degree in <span style="font-family:DarwinSans, 'Noto Sans Variable';font-weight:bolder;">Computer Science, Information Technology, Data Engineering, or a related field</span>.</p>