Senior Data Engineer Databricks
Job Summary
We are looking for an experienced
Senior Data Engineer – Databricks
to design, develop, and maintain scalable data pipelines on the Databricks platform. The role requires strong expertise in
PySpark, Databricks, and modern data engineering practices , along with experience modernizing legacy data pipelines and building high-performance data solutions in cloud environments. Key Responsibilities Design, build, and maintain
scalable data pipelines
using Databricks and PySpark. Develop
end-to-end data workflows
including data ingestion, transformation, and consumption. Optimize
Spark jobs, cluster configurations, and pipeline performance . Manage and orchestrate workflows using
Databricks Jobs and notebooks . Refactor
legacy ETL pipelines into modern PySpark-based ELT frameworks . Implement
data quality checks, monitoring, and error handling mechanisms . Design and maintain
Delta Lake tables
with proper optimization strategies. Collaborate with
data architects, analysts, and infrastructure teams
to deliver reliable data solutions. Troubleshoot and resolve
production data pipeline issues . Mandatory Technical Skills Strong
Data Engineering fundamentals
including ETL/ELT pipeline design. Hands-on experience with
PySpark (DataFrames API, Spark SQL, performance tuning) . Experience with
Databricks platform
including workspace, clusters, notebooks, and job orchestration. Knowledge of
Delta Lake features
such as ACID transactions and schema evolution. Strong
Python programming
for data processing and automation. Experience with
data modelling
(Dimensional modeling, Data Vault, or Lakehouse architecture). Strong
SQL skills
for data transformation and analysis. Experience with
cloud platforms (Azure, AWS, or GCP) . Experience with
Git version control and CI/CD practices . Professional Experience Minimum 8 years
of experience in
data engineering or related roles . 2–3 years of hands-on experience with Databricks . Experience building and maintaining
production-grade data pipelines at scale . Proven experience
modernizing legacy data pipelines . Experience working in
Agile development environments . Certifications (Mandatory) Databricks Certified Data Engineer Associate OR Databricks Certified Data Engineer Professional Preferred Skills Knowledge of
Spark Structured Streaming . Understanding of
data governance and data quality frameworks . Cloud certifications such as
Azure Data Engineer, AWS Data Analytics, or GCP Data Engineer .
Senior Data Engineer – Databricks
to design, develop, and maintain scalable data pipelines on the Databricks platform. The role requires strong expertise in
PySpark, Databricks, and modern data engineering practices , along with experience modernizing legacy data pipelines and building high-performance data solutions in cloud environments. Key Responsibilities Design, build, and maintain
scalable data pipelines
using Databricks and PySpark. Develop
end-to-end data workflows
including data ingestion, transformation, and consumption. Optimize
Spark jobs, cluster configurations, and pipeline performance . Manage and orchestrate workflows using
Databricks Jobs and notebooks . Refactor
legacy ETL pipelines into modern PySpark-based ELT frameworks . Implement
data quality checks, monitoring, and error handling mechanisms . Design and maintain
Delta Lake tables
with proper optimization strategies. Collaborate with
data architects, analysts, and infrastructure teams
to deliver reliable data solutions. Troubleshoot and resolve
production data pipeline issues . Mandatory Technical Skills Strong
Data Engineering fundamentals
including ETL/ELT pipeline design. Hands-on experience with
PySpark (DataFrames API, Spark SQL, performance tuning) . Experience with
Databricks platform
including workspace, clusters, notebooks, and job orchestration. Knowledge of
Delta Lake features
such as ACID transactions and schema evolution. Strong
Python programming
for data processing and automation. Experience with
data modelling
(Dimensional modeling, Data Vault, or Lakehouse architecture). Strong
SQL skills
for data transformation and analysis. Experience with
cloud platforms (Azure, AWS, or GCP) . Experience with
Git version control and CI/CD practices . Professional Experience Minimum 8 years
of experience in
data engineering or related roles . 2–3 years of hands-on experience with Databricks . Experience building and maintaining
production-grade data pipelines at scale . Proven experience
modernizing legacy data pipelines . Experience working in
Agile development environments . Certifications (Mandatory) Databricks Certified Data Engineer Associate OR Databricks Certified Data Engineer Professional Preferred Skills Knowledge of
Spark Structured Streaming . Understanding of
data governance and data quality frameworks . Cloud certifications such as
Azure Data Engineer, AWS Data Analytics, or GCP Data Engineer .