Azure Data Engineer
Summary
Build and maintain scalable Azure-based data pipelines using Data Factory, Databricks, PySpark, and Azure SQL to ingest, transform, and deliver data for analytics and reporting.
This role is for one of Weekday’s clients
Salary range: Rs 400000 - Rs 2600000 (ie INR 4 - 26 LPA)
Min Experience: 6+ years
Location: Bengaluru, Hyderabad, Chennai, Tamil Nadu, India
JobType: full-time
We are looking for an experienced Azure Data Engineer with 6–15 years of hands-on experience in designing, developing, and maintaining scalable data engineering solutions on Microsoft Azure. The ideal candidate will have strong expertise in Azure Data Factory, Azure Databricks, PySpark/Scala, Azure SQL, ADLS Gen2, and CI/CD pipelines.
You will be responsible for building robust data pipelines, developing data transformation workflows, integrating cloud-based data services, and ensuring data platforms are secure, scalable, reliable, and optimized for performance.
Requirements
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory.
- Build and optimize large-scale data processing solutions using Azure Databricks and PySpark/Scala.
- Develop data ingestion and transformation workflows across structured and unstructured data sources.
- Work extensively with Azure Data Lake Storage Gen2 (ADLS Gen2) for data storage, organization, and processing.
- Develop and optimize solutions using Azure SQL, T-SQL, and Stored Procedures.
- Design and implement integrations and event-driven workflows using Azure Logic Apps.
- Implement secure access and credential management using Azure Key Vault.
- Create and maintain CI/CD pipelines for data engineering applications, notebooks, workflows, and infrastructure.
- Monitor data pipelines and troubleshoot performance, reliability, and production issues.
- Collaborate with data architects, software engineers, analysts, and business stakeholders to translate requirements into scalable technical solutions.
- Follow best practices for data security, governance, performance optimization, and cloud architecture.
- Participate in code reviews, technical documentation, deployment planning, and production support activities.
Must-Have Skills
- 6–15 years of professional experience in data engineering, with significant experience working on Microsoft Azure.
- Strong hands-on expertise with Azure Data Factory (ADF).
- Strong experience with Azure Databricks.
- Proficiency in PySpark and/or Scala for large-scale data processing.
- Strong knowledge of Azure Data Lake Storage Gen2 (ADLS Gen2).
- Hands-on experience with Azure SQL and T-SQL.
- Strong experience developing and optimizing Stored Procedures.
- Experience designing and implementing CI/CD pipelines for data engineering workloads.
- Practical experience with Azure Logic Apps and cloud-based integrations.
- Experience using Azure Key Vault for secrets, credentials, and secure configuration management.
- Strong understanding of data engineering principles, ETL/ELT architecture, data transformation, and distributed processing.
Good-to-Have Skills
- Experience with Azure Synapse Analytics.
- Hands-on experience with Azure DevOps.
- Familiarity with Azure Log Analytics and cloud monitoring solutions.
- Experience with infrastructure creation and automation using Bicep, Terraform, or Ansible.
- Understanding of Infrastructure as Code (IaC) practices.
- Experience working in Agile/Scrum development environments.
- Knowledge of data security, governance, and Azure cloud best practices.