data engineer data platform transformation
Posted Updated
Summary
Freelance data engineer (initial 6-month contract, Netherlands) who designs and builds scalable data pipelines in Databricks using PySpark, Python and SQL, migrates existing workloads to a Delta Lake/Lakehouse platform, and optimises Spark performance as part of a major data platform transformation.
Описание
The company is a European technology company undergoing a major data platform transformation. Its data platform is being built and modernised with Databricks at the centre of the architecture.
Задачи
- Design and build scalable data pipelines within Databricks
- Develop production-grade workloads using PySpark, Python and SQL
- Work extensively with Delta Lake and Lakehouse architecture
- Support the migration of existing workloads into Databricks
- Optimise Spark workloads and improve pipeline performance
- Build reliable ETL/ELT processes across large datasets
- Work with Data Scientists, Analytics Engineers and Cloud Engineers
- Implement data quality, testing and monitoring
- Contribute to the wider architecture and technical direction of the platform
Требования
- 5+ Years of experience in Data Engineering
- Strong hands-on Databricks experience
- Excellent PySpark, Python and SQL skills
- Strong understanding of Delta Lake and Lakehouse architecture
- Experience building and optimising production data pipelines
- Experience working with Azure, AWS or GCP
- Experience working within large-scale data environments
- Ability to operate independently and hit the ground running
Условия
Initial 6-month freelance contract.