Senior Data Engineer (Databricks & Azure)
Summary
Build and optimize scalable data pipelines on Databricks and Azure, using PySpark, Python, and SQL to power analytics and reporting for the organization.
Job Summary
We are looking for an experienced Senior Data Engineer with strong expertise in Databricks and Azure to design, develop, and optimize scalable data solutions. In this role, you will build reliable data pipelines, develop ETL processes, improve data quality, and support the organization's modern data platform.
You will work closely with cross-functional teams to deliver high-quality data solutions that enable analytics, reporting, and business decision-making.
Key Responsibilities
Design, develop, and maintain scalable data pipelines using Databricks.
Build and optimize ETL/ELT processes for efficient data ingestion and transformation.
Develop data solutions using Python, PySpark, and SQL.
Implement data ingestion using Auto Loader and Delta Live Tables.
Manage and maintain Unity Catalog for secure data governance.
Integrate data from multiple sources using Azure Data Factory.
Monitor data quality and troubleshoot pipeline issues.
Optimize data processing performance and ensure reliability.
Collaborate with developers, analysts, architects, and business stakeholders.
Document data pipelines, workflows, and technical solutions.
Provide technical guidance and contribute to architectural decisions and best practices.
Required Qualifications
At least 5 years of experience in Data Engineering.
Strong hands-on experience with Databricks Unified Data Analytics Platform.
Proficiency in Python and PySpark.
Strong SQL development and query optimization skills.
Experience with Azure Data Factory (ADF).
Hands-on experience with:
Unity Catalog
Delta Live Tables (DLT)
Auto Loader
Strong understanding of ETL/ELT concepts and best practices.
Experience with data modeling and data warehousing.
Familiarity with Azure cloud services and modern data architecture.
Excellent analytical, problem-solving, and communication skills.
Nice to Have
Experience with Azure cloud ecosystem.
Knowledge of CI/CD for data engineering projects.
Experience working in Agile development environments.
Exposure to data governance and data security best practices.