Data Engineer - Databricks
Summary
Hands-on data engineer implementing a Databricks data and analytics platform for an insurance enterprise in a regulated environment, building pipelines, Unity Catalog configs, orchestration and governance practices. Core stack: Databricks/Spark, PySpark, SQL, Git, CI/CD. Must be an Australian citizen living in Sydney.
We are implementing a modern data and analytics platform to strengthen data quality, governance, automation, and insight generation across our insurance and broader enterprise reporting functions. We operate in a regulated, high-sensitivity environment and are building sustainable internal capability to operate and extend our data solutions over time.
This is a hands-on technical role contributing directly to our Databricks platform implementation. You will work alongside engineering, architecture, and data governance colleagues, collaborating across actuarial, IT, security, and implementation partners.
The Responsibilities
- Contribute to day-to-day technical delivery across actuarial, business, IT, and vendor stakeholders.
- Build and maintain Databricks pipelines, notebooks, Unity Catalog configurations, orchestration workflows, and performance optimisations.
- Support migration of priority data assets, processing logic, and analytics code into Databricks.
- Develop reusable patterns that support maintainability, governance, auditability, and performance.
- Apply governance practices: version control, code review, CI/CD, access control, data quality checks, and documentation standards.
- Adhere to and contribute to standards for open source, libraries, and third-party components including security and licensing compliance.
- Participate in technical reviews of key artefacts; support management of technical debt and BAU readiness.
- Contribute to knowledge sharing and capability building to reduce reliance on external vendors.
The Requirements
- Solid Databricks / Spark delivery in production: pipeline build, orchestration, performance tuning, and operationalisation.
- Exposure to data governance practices: RBAC, audit logging, lineage, data quality gates, and change control.
- Experience working in a regulated or high-sensitivity data environment with strong security discipline.
- Ability to collaborate effectively in co-delivery models with vendors and internal teams.
- Strong SQL and Python (including PySpark); solid software engineering discipline including Git, PR/code review, and CI/CD.
To be considered for this position, you must have real world working experience in Databricks - only those who have the same will be considered.
NB: **This position requires you to be an Australian Citizen & living in Sydney**