Senior Data Engineer

Summary

Build and optimize scalable data pipelines using Python, PySpark, and Databricks for batch and streaming workloads, managing Delta Lake layers and AWS S3 storage.

This range is provided by Humanised Group. Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.

Base pay range

A$100.00/hr - A$125.00/hr

Direct message the job poster from Humanised Group

Recruitment Consultant | Data Analytics & AI | Data Mornings - Helping Data Professionals Network & Grow

3 Month Contract | Hybrid Environment | Must be Sydney Based

If you are someone who enjoys designing scalable data pipelines and building modern, high performance data platforms, this is a great opportunity!

We are partnering with a global tech consultancy to find a Data Engineer for an initial 3 month contract.

Responsibilities:

Build/ improve data pipelines using Python and PySpark for both batch and streaming workloads
Design and manage Delta Lake layers (Bronze, Silver, Gold) -Databricks
Set up and maintain Databricks workflows, jobs, notebooks, and DLT pipelines
Optimise performance using Databricks tools and follow best practices for clean, modular code
Work with AWS S3 to manage raw and processed data securely
Use Terraform and Databricks Asset Bundles
Automate AWS workflows using Boto3 (S3, Glue, Lambda)
Work with data scientists and engineers to keep data flowing across teams
Apply place checks for data quality, validation, and monitoring

Required Skills & Experience:

Strong coding in Python and PySpark
Experience with Databricks and Delta Lake
experience in AWS services, especially S3, Glue, Lambda, and Redshift
Experience using Terraform or CloudFormation
Comfortable automating AWS tasks with Boto3
prior experience building efficient, secure, and scalable pipelines
Works well in cross functional and remote teams
Databricks certification
Experience working in consultancies or fast paced delivery environments

Seniority level

Seniority level
Mid-Senior level

Employment type

Employment type
Contract