Senior Databricks Data Engineer
Posted
Summary
Senior data engineer role centered on designing, building, and optimizing data pipelines on the Databricks Lakehouse platform. Day to day involves ETL/ELT development, workflow orchestration, Unity Catalog and streaming management, and performance/cost tuning using Python, PySpark, Spark SQL, and Git.
Mandatory Skills: Databricks, Data Engineering, Lakeflow, Unity Catalogue, Python, PySpark, ETL.
About the Role:
We are looking for an experienced Senior Databricks Data Engineer with strong expertise in Databricks, SQL, Lakeflow Jobs, PySpark notebooks, Python, and Unity Catalogue. This role requires hands-on experience in designing scalable data pipelines and delivering reliable data solutions on the Databricks Lakehouse platform.
Key Responsibilities:
- Design, develop, and optimize data pipelines using Databricks, Python, PySpark, and Spark SQL.
- Orchestrate and schedule workflow jobs that invoke notebooks, Python wheel packages, and Python notebooks.
- Work with Unity Catalog, manage tables, and streaming jobs.
- Implement and manage data ingestion, transformation, and loading processes from various sources into the Databricks Lakehouse.
- Build and maintain efficient and reliable ETL/ELT processes to ensure data quality and availability.
- Optimise Databricks workloads for performance, cost efficiency, and scalability.
- Troubleshoot and resolve data-related issues while ensuring data integrity and accuracy.
- Stay current with the latest Databricks features, best practices, and industry trends.
Required skills & Qualifications:
- At least 4 years of proven experience as a Data Engineer with a strong focus on Databricks (mandatory).
- Strong knowledge of Databricks architecture, Lakeflow Jobs, and data modelling.
- At least 10 years of experience in the data domain, particularly in data warehousing and ETL (mandatory).
- Expertise in PySpark, Spark SQL, and Python programming (mandatory).
- Hands-on experience with Git, including code check-in and check-out processes (mandatory).
- Strong understanding of data warehousing, data lakes, and Lakehouse architecture.
- Databricks Certified Data Engineer Associate or Professional certification is an advantage.
Soft Skills:
- Strong communication and stakeholder collaboration skills.
- Proactive, self-driven, and accountable mindset.
- Excellent analytical and problem-solving abilities.
- Ability to work effectively in Agile and cross-functional teams.
- Customer-focused with a commitment to quality delivery.
- Ability to converse seamlessly in English and one of either Dutch or French or both