Lead Data Engineer (Databricks)
Summary
Lead a team of data engineers building and running enterprise-scale data platforms on Databricks and AWS in Brisbane (on-site): design lakehouse architecture, set standards and governance, tune performance/cost, and manage BAU/DataOps support. Core stack: Databricks Lakehouse, PySpark, Delta Lake, Unity Catalog, dbt, Kafka, and AWS services.
Job Title: Lead Data Engineer (Databricks)
Experience: 10+ Years
Location: Brisbane (On-site)
Job Summary
We are seeking an experienced Databricks Lead to drive the design, development, modernization, and operational excellence of enterprise-scale data platforms on Databricks and AWS. The ideal candidate will combine deep technical expertise with team leadership, stakeholder management, and platform governance capabilities.
As a Databricks Lead, you will be responsible for leading a team of data engineers, defining best practices, overseeing delivery, ensuring platform stability, and managing BAU operations while partnering with business and technology stakeholders to deliver scalable, high-performing data solutions.
The role requires strong experience across Databricks Lakehouse Architecture, PySpark, Data Engineering, AWS, DataOps, and ITIL-based support operations.
Technical Competencies
Databricks & AWS Platform Leadership
- Extensive hands-on experience with Databricks Lakehouse Platform and enterprise-scale implementations.
- Strong expertise in:
- PySpark
- Delta Lake
- Unity Catalog
- Delta Live Tables (DLT) / Lakeflow Pipelines
- Serverless Compute
- SQL Warehouses
- Databricks Dashboards & Alerting
- Lead data architecture discussions and establish platform standards and best practices.
- Strong experience managing Databricks environments, security models, and governance frameworks.
Performance Optimization
- Expertise in:
- Liquid Clustering
- Z-Ordering
- Spark Performance Tuning
- Query Optimization
- Cost Optimization and Workload Management
- Experience optimizing large-scale batch and real-time processing workloads.
AWS Cloud
- Strong experience with AWS services including:
- Amazon S3
- IAM
- Lambda
- CloudWatch
- SNS/SQS
- Secrets Manager
- Experience designing secure, scalable, and highly available cloud-based data platforms.
Data Engineering & Integration
- Advanced experience with:
- Databricks Asset Bundles (DABs)
- dbt Core / dbt Cloud
- Kafka Streaming
- Structured Streaming
- Data Modeling
- ETL/ELT Frameworks
- Experience handling large-scale structured and semi-structured datasets.
Interested Candidates can share their updated resumes to nidhi.goyal@carecone.com.au or can reach me at +61 291 575 539
Skills
Sign in to read and join the discussion.