Data Engineer - Spark & Scala
Posted 32
views
Summary
A hybrid (2-3 days onsite) data engineering role in Glasgow focused on processing, transforming and validating large-scale datasets with Apache Spark and Scala, orchestrating pipelines with Apache Airflow, and providing production support for data pipelines using Python, SQL and PostgreSQL.
We are hiring: Data Engineer – Spark & Scala
Location: Glasgow, UK – Hybrid (2–3 days onsite)
- Strong hands-on experience with Apache Spark and Scala for processing, transforming and validating large-scale datasets.
- Experience developing and maintaining data pipelines and orchestration workflows using Apache Airflow.
- Proficiency in Python, SQL and PostgreSQL for data processing, validation, reconciliation and troubleshooting.
- Experience with production support, investigating failed workflows, data mismatches, missing files and pipeline failures.
- Familiarity with GitLab, CI/CD, Docker and controlled software release practices.
- Understanding of data quality, monitoring, logging, auditability and operational resilience.
- Experience with Starburst/Trino, AWS S3, Kubernetes or OpenShift would be an advantage.