Data Engineer

Blackbook.ai is one of Australia's most successful and progressive AI companies. Our use of Data and AI to deliver on our client's pursuit of bottom-line benefits has resulted in a company that can claim a 100% success rate in solution delivery. We have an agile approach to tackling problems. We understand that there isn't just one solution to every problem! Our unique approach to problem-solving will see you working directly with our customers, capturing additional requirements and developing intuitive solutions while drawing on the vast knowledge of existing team members.

The Data Engineer role will see you working with multiple clients, projects and stakeholders alongside fellow experts in the DataOps, Digital and AI teams. You will be hands-on building modern cloud data platforms, primarily on Microsoft Fabric and Databricks, covering pipeline engineering, data modelling and reporting, with the opportunity to expand into AWS as our client base grows.

Your responsibilities as a Data Engineer are as follows:

• Work directly with clients as a Blackbook.ai consultant

• Gather requirements, design, build and test data and analytics solutions

• Build and orchestrate data pipelines using Microsoft Fabric Pipelines and Azure Data Factory

• Develop scalable data transformations in Python, PySpark and SQL across Fabric and Databricks

• Design and implement cloud data warehouses and lakehouses using medallion (bronze, silver, gold) architecture

• Build star schema dimensional models that support scalable, cost-efficient reporting and analytics

• Deliver reporting and semantic models that business users can trust

• Design and build metadata-driven data ingestion into data lakes, lakehouses and databases

• Use AI tools appropriately to accelerate delivery and help clients get value from AI

• Apply engineering best practice: Git, code review, automated testing and CI/CD

• Take ownership of the end-to-end process from requirements gathering to implementation

• Keep up with emerging technologies and share knowledge with your peers

To be considered for the position you must have a bachelor's degree or higher, and be comfortable demonstrating your skills in the following areas:

• Python and SQL for data engineering (mandatory)

• Hands-on experience with Microsoft Fabric (Lakehouse, Warehouse, Notebooks and Pipelines)

• Building and orchestrating pipelines in Azure Data Factory or Fabric Pipelines

• Data warehousing in the cloud using medallion architecture

• Designing and building star schema dimensional models

• Commercial experience with Databricks and PySpark / Apache Spark

• Reporting and data visualisation, ideally with Power BI

• Using AI tools and techniques appropriately in data engineering work

• Professional presentation and communication skills: written and verbal, formal documentation, diagramming and modelling

• Critical thinking and problem solving skills

Highly regarded although not mandatory experience includes:

• AWS data engineering (e.g. S3, Glue, Redshift, Lambda), or a willingness to upskill as our AWS client base grows

• R for data transformation and analysis

• Microsoft Fabric, Azure or Databricks certification in Data Engineering

• Delta Lake, Unity Catalog or MLflow

• Spark performance tuning: partitioning, joins, caching and cluster sizing

• Terraform, PowerShell or Azure/AWS CLI for managing cloud infrastructure

In return, we can offer you:

• Competitive Remuneration

• Flexible Working Conditions

• Gym and PT

• Free Spotify Account

• Extra leave

Blackbook.ai can offer a fast-paced and collaborative delivery environment with an emphasis on culture and teamwork, ensuring all projects are delivered with our high standards. We have a large team of dedicated consultants able to work alongside and guide you through your journey, and offer on-the-job training and development with access to multiple certifications.