Data Engineer
Posted Updated 12
views
Summary
An 8-week contract (strong extension chance) in Sydney: a Databricks Data Engineer completes a Proof of Technology and stands up a secured simulated MVP environment (Unity Catalog, streaming pipelines, simulated asset telemetry) for a Global Optimisation and bidding platform supporting Quant Research.
Contract position (8 weeks) Strong chance of extension
Our client is seeking an experienced Databricks Data Engineer for an 8-week contract to bring the Databricks Proof of Technology (PoT) to completion and stand up a simulated MVP environment. This role supports the Global Optimisation Platform Project, which is creating an innovative optimisation and bidding solution.
About the role:
- Be part of a core Data Engineering team, this engagement adds the specialist Databricks capacity to close that gap across the team.
- Contributing direct Databricks engineering effort within the existing team, with the goal of clearing the PoT gate on schedule and handing Quant Research a workable simulated MVP.
- Stand up the core Databricks environment for the simulated MVP, with security built in through Unity Catalog, access controls, and compute policies.
- Create simulated asset telemetry and align the market data, giving the Quant Research team a complete, end-to-end modelling pipeline
- Use real effort from reference pipelines as a baseline to refine the delivery-phase estimate for the data ingestion and integration workstream.
Required Experience:
- Over five years of data engineering experience, including a minimum of two years building and running Azure Databricks platforms in production.
- Hands‑on experience designing and governing Unity Catalog, covering catalog structure, access permissions, external locations, storage credentials and system tables.
- Deep knowledge of Azure Databricks security, including VNet injection, Private Link, Entra ID with SCIM provisioning, service principals and Key Vault-backed secret scopes.
- Proven experience building real‑time pipelines with Structured Streaming on Event Hubs or Kafka, along with Delta Lake and medallion architecture.
- Practical experience deploying with Databricks Asset Bundles and running CI/CD pipelines in Azure DevOps or GitHub, plus working with Git folders.
- Advanced Python skills, including building wheel packages and managing library dependencies in privately networked environments.