Data Engineer - Databricks (f/m/d)
Summary
Data engineer building and operating Databricks data pipelines (Spark, Delta Lake) for an international insurance group's data platform. Day-to-day: data modelling, lakehouse structures, integrating source systems, ensuring governance in a regulated environment, and optimizing Spark performance/cost using Python/PySpark and SQL.
In this project you will work in the data platform environment of an international insurance group
Must haves
- Databricks
- Python/PySpark
- SQL
Nice to have
Your Tasks
- Design, build and operate data pipelines on Databricks
- Data modelling and building lakehouse structures (Delta Lake)
- Connecting and integrating source systems
- Ensuring data quality and governance in a regulated environment
- Performance and cost optimisation of Spark workloads
- Providing data for analytics and AI use cases
- Collaboration with data scientists, analysts and the platform team
Requirements
- At least 5 years of professional experience in data engineering
- Solid hands-on experience with Databricks (Apache Spark, Delta Lake)
- Strong skills in Python/PySpark and SQL
- Experience building and operating data pipelines (batch and/or streaming)
- Experience with data modelling and lakehouse architectures is a plus
- Cloud experience (preferably Microsoft Azure) is a plus
- Excellent English skills (project language)
The content of this advertisement relates to an order from our customer company in the context of personnel placement. As a recruiter, we support you throughout the entire application process.
Skills
Sign in to read and join the discussion.