Senior Data Engineer (Databricks / Spark)

Salary: Up to $180,000

Our client will not hire a data engineer who isn't already using AI in their daily engineering practice.

Not "open to it." Not "keen to learn." You're already doing it, and already opinionated about where it helps and where it produces confident nonsense lol. Most other things on this brief are negotiable.

About the opportunity
Our client is a fast-growing Australian business that treats technology as the engine of the organisation rather than a support function. They're not a consultancy and not an agency; you'd be building their own platform for their own product, and you'd see what your work does to the business within weeks.

They are unusually far along on AI. It's embedded in how the engineering team works day-to-day, and there is a real product roadmap behind it rather than a pilot and a press release. For the right engineer, that's the most interesting aspect of this role, and it's what sets it apart from most senior data roles currently advertised in Brisbane.

The role
You'll join a small, senior data team with two connected mandates.
The first is present-tense delivery. The business runs several disconnected source systems — a finance platform, a CRM, and its own technology generating operational data. The goal is one trusted data warehouse that replaces the siloed picture they have today. Part of that is re-architecting pipelines that already exist but aren't performing; part of it is greenfield ingestion to onboard everything else.

The second is forward-looking: how this environment should be built over the next three years, across scalability, governance, compute strategy and cost. Hands-on architecture, not a whiteboard role.

Both sit in the same team, with the same interview panel. You don't need to choose between them when you apply; the process is designed to determine where you're strongest, and both positions are open.

What you'll be doing

• Designing, building and maintaining ingestion pipelines and API integrations across the core systems of the business
• Re-architecting existing pipelines for performance, reliability and observability
• Building dimensional models that support reporting, analytics and AI workloads
• Working with REST APIs at scale - OAuth2, pagination, rate limiting, webhooks, error handling
• Contributing to platform decisions across governance, compute strategy and cost
• Setting engineering standards around testing, CI/CD and documentation, and lifting the capability of the team around you

What you'll need

• 7+ years in data engineering, building and running production-grade pipelines
• Databricks experience — or very strong Spark / PySpark and the appetite to pick Databricks up quickly
• Strong Python and SQL
• Genuine depth in dimensional data modelling and pipeline reliability
• Experience integrating REST APIs at scale
• Comfort with CI/CD and modern engineering practice
• An established AI-assisted engineering practice. This is the client's hard non-negotiable. You'll be asked to talk specifically about how it has changed the way you build, and where you don't trust it.
• Self-awareness. The panel will ask you about an architecture decision you got wrong. They're not testing for a flawless record - they're testing whether you can look at your own work honestly. Come prepared with a real one.

Highly regarded:
Delta Lake, Unity Catalog, streaming, CDC, event-driven architecture, Terraform or equivalent IaC. Azure or AWS both fine — AWS is not a requirement.


**For more information or to make an outstanding impression, email: usama@blackroc.co or jason@blackroc.co

At BLACKROC Recruitment, we are committed to fostering diversity, equity, and inclusion.

We welcome applications from individuals of diverse backgrounds, including those with lived experiences.

If you require any adjustments during the recruitment process, please don’t hesitate to reach out to the above-mentioned BLACKROC Consultant.