Data Engineer (Databricks)
Description
Databricks
Consultant – Build and Optimise Production Lakehouse Pipelines
We’re
looking for a hands-on Databricks Consultant who designs, builds and optimises
lakehouse data pipelines and solutions on the Databricks platform.
This
is a role for someone who likes getting into the platform and making data
pipelines reliable, fast and cost-efficient. You’ll work across ingestion,
transformation and delivery of curated data products using PySpark, Spark SQL
and Delta Lake, together with job orchestration, performance and cost tuning,
data quality and platform governance.
What you’ll do:
· Design,
build and maintain production data pipelines for ingestion, transformation and
delivery of curated data.
· Develop
and optimise PySpark and Spark SQL transformations, including incremental and
batch processing patterns.
· Design
and operate Delta Lake tables, including table layout, optimisation and
file-size management.
· Contribute
to platform performance and cost efficiency through job, compute and query
tuning.
· Implement
data quality checks, validation, testing and clear technical documentation.
· Build
and maintain workflow orchestration, job scheduling, retries and alerting for
production pipelines.
· Apply
Unity Catalog for data cataloguing, permissions and lineage where in use.
· Follow
version control and CI/CD practices for notebooks, packages, jobs and pipeline
configuration.
· Tune
Spark jobs for scale and cost, including partitioning, shuffles, caching and
cluster configuration.
· Ingest
and transform data from enterprise and SaaS sources into well-structured
medallion (bronze, silver, gold) or equivalent layers.
· Troubleshoot
failed jobs, slow queries and data defects, and implement lasting technical
corrections.
· Package
reusable pipeline components, parameters and configuration for consistent
delivery.
· Document
pipeline design, table contracts, refresh behaviour and operational steps.
Requirements
What we’re looking for:
· Minimum
two years of hands-on, day-to-day Databricks experience.
· Strong
proficiency in Python, PySpark and Spark SQL.
· Practical
experience building and operationalising data pipelines on Databricks.
· Experience
working with Delta Lake, including table design and routine optimisation.
· Solid
grasp of data modelling and the ability to write efficient, well-structured SQL
and Spark transformations.
· Experience
with incremental processing, orchestration and production operationalisation of
pipelines.
· Experience
on at least one major cloud platform (Azure, AWS or GCP).
· Familiarity
with Databricks compute, jobs, workflows and SQL warehouses, including
performance and cost awareness.
· Experience
with Git-based version control and CI/CD for data workloads.
· Working
knowledge of data quality checks, testing, technical documentation and access
control.
Skills
Sign in to read and join the discussion.