Data Engineer (Microsoft Azure)

Project description

The project focuses on designing and delivering scalable cloud-based data engineering solutions on Microsoft Azure. The role will support large-scale data integration, analytics, machine learning, AI solutions, and business intelligence through modern data platforms and automated ETL/ELT pipelines.

Responsibilities

• Design, develop, and optimize ETL/ELT solutions using Azure Databricks, LakeBase, and Spark. Build and deploy large-scale data pipelines supporting analytics, machine learning, predictive modeling, reporting, and data products. Integrate structured and unstructured data from databases, APIs, FTPs, cloud storage, and various file formats. Develop and optimize complex SQL queries and data models. Build AI-based solutions for business needs, process automation, and operational efficiency. Improve data reliability, integrity, performance, and overall platform quality. Contribute to data architecture, engineering standards, frameworks, and best practices. Provide technical guidance and mentorship to other data engineers.

SKILLS

Must have

• 7+ years of experience in data engineering, data integration, data modeling, data architecture, and ETL/ELT processes. 5+ years of hands-on Python experience and 2+ years of experience with SQL, complex data schemas, and query performance optimization. At least 2 years of hands-on experience with Redis-backed state management and event-driven processing, as well as building cloud solutions using Azure Databricks with Unity Catalog, Snowflake, Azure Functions, and Service Bus. Practical experience with Terraform-based DevOps automation and CI/CD tools including GitHub Actions, Git, Artifactory, and Sonar is required. Mandatory Skills (only names): Azure Databricks Apache Spark Python SQL ETL/ELT Data Engineering Data Modeling Redis Snowflake Azure Functions Azure Service Bus Terraform GitHub Actions Git Artifactory Sonar

Nice to have

Excellent communication skills Bachelor’s degree in Computer Science, Engineering, Mathematics, or a related discipline. Strong understanding of Data Lake, Delta Lake, and Data Warehousing architectures, as well as advanced data modeling techniques including slowly changing dimensions, aggregation, partitioning, and indexing. Exposure to LLM/AI solution delivery is preferred. Ability to independently troubleshoot and performance-tune large-scale enterprise systems, collaborate effectively across technical and business teams, mentor engineers, and communicate technical concepts to both technical and non-technical audiences.