Senior Data Engineer
Summary
A senior data engineer who designs, builds, and optimizes scalable batch and streaming ETL pipelines on the Databricks Lakehouse Platform. Core technologies include PySpark, Spark SQL, Delta Lake, Delta Live Tables, Unity Catalog, cloud platforms, and Git-based CI/CD, with mentoring and ML productionization support.
We are looking for a Senior Data Engineer to join our team. In this role, you will design, build, and optimize scalable data pipelines on the Databricks Lakehouse Platform. You will partner with data science, analytics, and business teams to deliver reliable, high-performance data solutions in a collaborative environment.
Your responsibilities will include building and maintaining batch or streaming ETL pipelines using PySpark, Delta Lake, and Delta Live Tables. You will manage Databricks clusters and implement Unity Catalog for governance. You will also design dimensional models, optimize Delta tables, and troubleshoot performance issues. A key part of the role involves building deployment pipelines using CI/CD tools and supporting ML productionization while mentoring fellow team members.
Required Skills:
- 5+ years of experience in data engineering with significant hands-on work in Databricks
- Strong proficiency in PySpark, Spark SQL, Delta Lake, and Delta Live Tables
- Advanced skills in Python and SQL
- Experience with Unity Catalog, cluster administration, and at least one major cloud platform (Azure, AWS, or GCP)
- Familiarity with orchestration tools such as Databricks Workflows or Airflow and Git-based CI/CD
- Solid understanding of data warehousing and dimensional modeling
Preferred Skills:
- Databricks Certified Data Engineer (Associate or Professional)
- Cloud data engineering certification