Data Engineer

This is a specialist government agency that provides data, research and analysis on key economic and social themes.

You’ll be working on a significant national policy program, building a standardised, evidence-based framework used to classify and connect information across interconnected sectors.

The Opportunity

This is your chance to build the complex data pipelines behind a national classification challenge.

• Real influence over how the technical build takes shape, not just maintaining someone else’s pipeline
• Led by a people-focused Band 1 with a robust academic background and a clear direction for the project
• Take existing pipelines and AI models into production as a fully operationalised CI/CD capability
• Core stack is Databricks and Python
• Long-term contract, October 2026 to 30 June 2027, with a 12-month extension option

The Role

The key technical priorities for you to deliver in the first 3-6 months are:

• Refactor and expand the data ingestion stage to support new use cases, including turning unstructured data into structured data
• Automate data ingestion across multiple data sources
• Design and implement a data quality and validation layer for large-scale datasets

Additionally, you will be:

• Building and maintaining model-serving, deployment and API integration capabilities
• Applying data governance, metadata, lineage and versioning practices alongside DevOps/MLOps standards – CI/CD, testing and documentation
• Working with data scientists, policy and information management stakeholders to translate agreed definitions and business rules into the pipeline, including integrating outputs with downstream systems

The Ideal Person

You’re a senior data engineer with advanced Python skills, comfortable managing large-scale datasets, automating workflows and building pipelines across cloud or distributed platforms.

You’ve taken machine learning, NLP or generative AI solutions into production, with the data quality, lineage and MLOps practices to back it up.

You will bring the following skills and experience:

• Advanced Python and modern data engineering skills, at scale
• Hands-on experience with Databricks or an equivalent cloud/distributed platform
• A track record taking ML, NLP or generative AI into production – APIs, model serving and MLOps included

Due to the nature of the work, Australian Citizenship and an existing Baseline security clearance is mandatory.

Salary

• This role is being offered as an EL1 equivalent full-time labour hire contract from October 2026 to 30 June 2027, with a 12-month extension option
• There is no set rate for this RFQ. However the value for you coming through myself for this opportunity is that I have extensive experience recruiting into this market, I work with each candidate individually to land on a rate that’s competitive
• Hybrid working, multiple locations across the country

HorizonOne is committed to building a diverse and inclusive workforce and we encourage applications from people of all cultures, capabilities and backgrounds.