Data Engineer
Posted Updated 8
views
Summary
Hands-on data engineer building scalable Databricks pipelines (PySpark, Spark SQL, Delta Lake) and migrating SAS processes for a Federal Government schools data team. Hybrid role in Canberra requiring Baseline clearance, with Azure DevOps/Git CI/CD, testing and data governance work.
Salary: $105 - $120 per hour
Data Engineer – Databricks / PySparkCanberra | Hybrid | Baseline Clearance Required
$105–$120 per hour incl. super
We are seeking an experienced Data Engineer to support the design, migration and operation of data solutions within a Federal Government environment.
This is a hands-on engineering role focused on Databricks, PySpark, Spark SQL, Delta Lake and SAS-to-Databricks migration.
You will work within the Schools Data Production Team to build scalable data pipelines, migrate existing SAS processes and establish secure, maintainable and well-documented data engineering practices.
Key responsibilities
- Design, build and maintain scalable Databricks data pipelines
- Migrate SAS code, workflows and scheduled processes to Databricks
- Develop solutions using PySpark, Spark SQL and advanced SQL
- Build ETL/ELT pipelines and medallion-layer data models
- Implement Delta Lake solutions
- Establish source control, deployment and testing practices
- Work with Azure DevOps, Git and CI/CD
- Implement automated testing, monitoring and performance optimisation
- Apply data security, privacy, metadata and governance controls
- Produce clear technical documentation and support knowledge transfer
- Strong Databricks, PySpark, Spark SQL and SQL capability
- Demonstrated experience designing scalable Databricks solutions
- Experience building maintainable ETL/ELT pipelines
- Experience with Delta Lake and medallion architecture
- Strong experience with Azure DevOps, Git and CI/CD
- Experience with automated testing, monitoring and performance optimisation
- Ability to produce secure, maintainable and well-documented data solutions
- Demonstrated experience migrating SAS code, workflows and reporting processes to Databricks
- Be able to work from the Canberra office under hybrid arrangements
- Hold a current Baseline security clearance or higher
Respond with your tailored CV. Suitable candidates will be asked to complete and application pack, including a One-page pitch for the role.
Candidates not meeting requirements abve, Citizenship and Security clearance requirements, will not be contacted.