Data Engineer

Dear Candidates,

We have an urgent opening for Azure Data Engineer role for one of our reputed clients.

Job Title: Senior Data Engineer Azure Databricks & SQL

Experience: 5+ Years
Primary Skills: Azure Databricks, ADLS, Apache Spark, Scala, SQL, Data Streaming
Preferred Domain: Healthcare

Job Summary

We are looking for an experienced Senior Data Engineer with strong hands-on expertise in Azure Databricks, Apache Spark, Scala, and SQL to design, develop, and optimize enterprise-grade data engineering solutions on Azure.

The ideal candidate should have experience building scalable batch and streaming data pipelines, implementing ETL/ELT solutions, optimizing Spark workloads, and working with Azure Data Lake Storage (ADLS). The candidate will also provide technical leadership, mentor team members, and support QA/UAT and production implementation activities.

Key Responsibilities

• Design, develop, and implement scalable data engineering solutions using Azure Databricks and ADLS.
• Build and maintain enterprise-grade ETL/ELT pipelines for batch and streaming data processing.
• Develop data transformation workflows using Apache Spark, Scala, and SQL.
• Implement streaming solutions using Spark Structured Streaming and relevant streaming design patterns.
• Optimize Spark applications and SQL queries to improve performance, scalability, and reliability.
• Design reusable and standardized components for data ingestion, transformation, and integration.
• Develop and maintain data pipelines across different data zones, including Raw, Curated, and Consumption layers.
• Implement data quality frameworks, validation rules, and data governance practices.
• Identify opportunities to improve existing solutions through automation, performance tuning, and operational efficiency.
• Lead and mentor data engineering teams by providing technical guidance, conducting code reviews, and supporting solution design.
• Collaborate with architects, business analysts, and cross-functional teams in Agile delivery environments.
• Support QA, UAT, deployment, production implementation, and troubleshooting activities.
• Ensure adherence to engineering standards, documentation, logging, monitoring, and CI/CD practices.

Mandatory Technical Skills

• Strong hands-on experience with Azure Databricks.
• Experience with Azure Data Lake Storage (ADLS).
• Strong knowledge of Apache Spark, including core concepts and performance tuning.
• Proficiency in Scala programming.
• Strong experience in SQL, including query development and performance optimization.
• Hands-on experience with Data Streaming, particularly Spark Structured Streaming and streaming design patterns.
• Experience designing and developing enterprise-grade ETL/ELT data pipelines.

Good-to-Have Skills

• Experience with Python and PySpark.
• General knowledge of the Azure cloud platform.
• Experience working with MongoDB.
• Healthcare domain experience, preferably involving:
  
• Healthcare Claims
• Clinical Data
• Eligibility
• Provider Data
• Member Data
• HIPAA-aware data handling

Required Experience & Competencies

• Minimum 5+ years of hands-on experience in Data Engineering / Data Warehousing development and operations.
• Proven experience designing and implementing scalable enterprise data pipelines.
• Strong understanding of data integration, transformation, and processing frameworks.
• Experience working in Agile development environments.
• Ability to lead technical discussions and provide guidance to engineering teams.
• Strong analytical, problem-solving, and troubleshooting skills.
• Excellent communication and collaboration skills.