Senior Data Engineer (Databricks + Pyspark + Adf + Unity Catalog)

Overview

We are seeking a highly skilled Senior Data Engineer with hands-on experience in Databricks, Unity Catalog, and Azure Data Factory (ADF) to support a large-scale enterprise data modernization initiative. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines, enabling secure data governance, and supporting cloud-based analytics solutions in Azure.

This role will collaborate closely with architects, business stakeholders, and cross-functional engineering teams to build reliable, high-performance data platforms and accelerate data-driven decision-making.

Responsibilities

Your Impact

Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
Build and maintain scalable data ingestion frameworks for structured and unstructured data sources.
Develop data transformation processes using PySpark, Spark SQL, and Databricks notebooks.
Implement and manage Unity Catalog for enterprise data governance, security, and access controls.
Create and optimize Delta Lake architectures for analytics and reporting workloads.
Monitor pipeline performance and troubleshoot production issues.
Collaborate with data architects to implement enterprise data models and best practices.
Ensure data quality, integrity, security, and compliance standards are met.
Support CI/CD deployment processes and DevOps practices for data engineering solutions.
Work closely with business and analytics teams to understand requirements and deliver scalable solutions.

Requirements

Your Skills & Experience

4-7+ years of experience in Data Engineering or Big Data development.
Strong hands-on experience with Azure Databricks.
Proficiency in PySpark, Spark SQL, and distributed data processing.
Experience implementing and managing Unity Catalog.
Strong experience with Azure Data Factory (ADF).
Experience working with Delta Lake and modern Lakehouse architectures.
Solid understanding of data modeling, ETL/ELT, and data warehousing concepts.
Experience with Azure cloud services and ecosystem.
Proficiency with Git, version control, and CI/CD processes.
Strong analytical and problem-solving skills.
Excellent communication and stakeholder management abilities.

Preferred Qualifications

Experience with Azure Synapse Analytics.
Familiarity with Collibra or other data governance platforms.
Experience with Infrastructure as Code (Terraform, ARM Templates).
Databricks Certification (Data Engineer Associate/Professional).
Microsoft Azure Data Engineer Associate Certification.

Nice to Have

Experience supporting enterprise-scale data migration projects.
Knowledge of healthcare, retail, or consumer products data environments.
Exposure to real-time data processing and streaming technologies.

Show