Senior Databricks Data Engineer: Spark, Delta Lake, Python
Posted Updated
Description
softwareXperts GmbH is seeking a senior data engineer to shape our data platform for an international insurance group. You will design, build and operate data pipelines on Databricks and contribute to lakehouse architectures.
You will collaborate with data scientists, analysts and the platform team, ensure data quality in regulated environments, and optimize Spark workloads for performance and cost while leveraging Azure cloud where applicable. English is the project language.
Design, build and operate data pipelines on Databricks. Data modelling and building lakehouse structures (Delta Lake). Connecting and integrating source systems. Ensuring data quality and governance in a regulated environment. Performance and cost optimisation of Spark workloads. Providing data for analytics and AI use cases. Collaboration with data scientists, analysts and the platform team. At least 5 years of professional experience in data engineering. Solid hands-on experience with Databricks (Apache Spark, Delta Lake) Strong skills in Python/PySpark and SQL Experience building and operating data pipelines (batch and/or streaming) Experience with data modelling and lakehouse architectures is a plus Cloud experience (preferably Microsoft Azure) is a plus Excellent English skills (project language)Skills
Sign in to read and join the discussion.