Senior Data Engineer - Databricks

Job Summary

We are looking for an experienced Senior Data Engineer with strong expertise in Databricks, Apache Spark, Python, and cloud-based data engineering solutions. The ideal candidate will have hands-on experience designing, developing, and optimizing scalable data pipelines, implementing data lakehouse architectures, and processing large volumes of structured and unstructured data.The candidate should have strong technical skills in data engineering, ETL/ELT development, data modeling, and cloud platforms, along with the ability to work independently and collaborate with cross-functional teams.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Build and optimize ETL/ELT workflows for large-scale data processing.
• Develop data transformation solutions using Python, PySpark, and SQL.
• Implement and manage Lakehouse architecture using Delta Lake.
• Work with Databricks Workflows, Unity Catalog, and Delta Live Tables (Lakeflow Declarative Pipelines).
• Integrate data from multiple sources into centralized data platforms.
• Optimize Spark jobs, cluster configurations, and SQL queries for performance and cost efficiency.
• Implement data quality, governance, security, and monitoring practices.
• Collaborate with architects, analysts, and business stakeholders to understand data requirements.
• Troubleshoot data pipeline issues and ensure reliability and scalability.

Required Technical Skills

• 15+ years of experience in Data Engineering.
• Strong hands-on experience with Databricks and Apache Spark.
• Proficiency in Python, PySpark, and SQL.
• Experience with Delta Lake and Lakehouse architecture.
• Strong experience developing complex ETL/ELT pipelines.
• Experience with cloud platforms such as Azure (Azure Databricks), AWS, or GCP.
• Knowledge of data warehousing, data modeling, and distributed computing.
• Experience with Databricks Workflows and job orchestration.
• Understanding of data governance, security, and performance optimization.

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional.
• Experience with Azure Data Factory, AWS Glue, or similar orchestration tools.
• Experience with CI/CD, Git, and DevOps practices.
• Knowledge of Unity Catalog and Delta Live Tables.
• Experience working with enterprise-scale data migration and modernization projects.