Senior Data Engineer Databricks

Job Summary

We are looking for an experienced Senior Data Engineer – Databricks to design, develop, and maintain scalable data pipelines on the Databricks platform. The role requires strong expertise in PySpark, Databricks, and modern data engineering practices , along with experience modernizing legacy data pipelines and building high-performance data solutions in cloud environments.

Key Responsibilities

• Design, build, and maintain scalable data pipelines using Databricks and PySpark.
• Develop end-to-end data workflows including data ingestion, transformation, and consumption.
• Optimize Spark jobs, cluster configurations, and pipeline performance .
• Manage and orchestrate workflows using Databricks Jobs and notebooks .
• Refactor legacy ETL pipelines into modern PySpark-based ELT frameworks .
• Implement data quality checks, monitoring, and error handling mechanisms .
• Design and maintain Delta Lake tables with proper optimization strategies.
• Collaborate with data architects, analysts, and infrastructure teams to deliver reliable data solutions.
• Troubleshoot and resolve production data pipeline issues .

Mandatory Technical Skills

• Strong Data Engineering fundamentals including ETL/ELT pipeline design.
• Hands-on experience with PySpark (DataFrames API, Spark SQL, performance tuning) .
• Experience with Databricks platform including workspace, clusters, notebooks, and job orchestration.
• Knowledge of Delta Lake features such as ACID transactions and schema evolution.
• Strong Python programming for data processing and automation.
• Experience with data modelling (Dimensional modeling, Data Vault, or Lakehouse architecture).
• Strong SQL skills for data transformation and analysis.
• Experience with cloud platforms (Azure, AWS, or GCP) .
• Experience with Git version control and CI/CD practices .

Professional Experience

• Minimum 8 years of experience in data engineering or related roles .
• 2–3 years of hands-on experience with Databricks .
• Experience building and maintaining production-grade data pipelines at scale .
• Proven experience modernizing legacy data pipelines .
• Experience working in Agile development environments .

Certifications (Mandatory)

• Databricks Certified Data Engineer Associate
• OR
• Databricks Certified Data Engineer Professional

Preferred Skills

• Knowledge of Spark Structured Streaming .
• Understanding of data governance and data quality frameworks .
• Cloud certifications such as Azure Data Engineer, AWS Data Analytics, or GCP Data Engineer .