Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

CareCone Group

NewBe an early applicant

Lead Data Engineer (Databricks)

Posted

Summary

Lead a team of data engineers building and running enterprise-scale data platforms on Databricks and AWS in Brisbane (on-site): design lakehouse architecture, set standards and governance, tune performance/cost, and manage BAU/DataOps support. Core stack: Databricks Lakehouse, PySpark, Delta Lake, Unity Catalog, dbt, Kafka, and AWS services.

Job Title: Lead Data Engineer (Databricks)

Experience: 10+ Years

Location: Brisbane (On-site)


Job Summary

We are seeking an experienced Databricks Lead to drive the design, development, modernization, and operational excellence of enterprise-scale data platforms on Databricks and AWS. The ideal candidate will combine deep technical expertise with team leadership, stakeholder management, and platform governance capabilities.

As a Databricks Lead, you will be responsible for leading a team of data engineers, defining best practices, overseeing delivery, ensuring platform stability, and managing BAU operations while partnering with business and technology stakeholders to deliver scalable, high-performing data solutions.

The role requires strong experience across Databricks Lakehouse Architecture, PySpark, Data Engineering, AWS, DataOps, and ITIL-based support operations.


Technical Competencies

Databricks & AWS Platform Leadership

  • Extensive hands-on experience with Databricks Lakehouse Platform and enterprise-scale implementations.
  • Strong expertise in:
  • PySpark
  • Delta Lake
  • Unity Catalog
  • Delta Live Tables (DLT) / Lakeflow Pipelines
  • Serverless Compute
  • SQL Warehouses
  • Databricks Dashboards & Alerting
  • Lead data architecture discussions and establish platform standards and best practices.
  • Strong experience managing Databricks environments, security models, and governance frameworks.

Performance Optimization

  • Expertise in:
  • Liquid Clustering
  • Z-Ordering
  • Spark Performance Tuning
  • Query Optimization
  • Cost Optimization and Workload Management
  • Experience optimizing large-scale batch and real-time processing workloads.

AWS Cloud

  • Strong experience with AWS services including:
  • Amazon S3
  • IAM
  • Lambda
  • CloudWatch
  • SNS/SQS
  • Secrets Manager
  • Experience designing secure, scalable, and highly available cloud-based data platforms.

Data Engineering & Integration

  • Advanced experience with:
  • Databricks Asset Bundles (DABs)
  • dbt Core / dbt Cloud
  • Kafka Streaming
  • Structured Streaming
  • Data Modeling
  • ETL/ELT Frameworks
  • Experience handling large-scale structured and semi-structured datasets.


Interested Candidates can share their updated resumes to nidhi.goyal@carecone.com.au or can reach me at +61 291 575 539

Skills

What Lead Data Engineering jobs ask for — and how much of it you have →
Apply

See also

Data Engineering jobs by country — openings, pay and top skills →