Azure Databricks Data Engineer: Real-Time Pipelines DevOps
Summary
A Sydney-based data engineer role focused on designing, building, and optimizing high-throughput real-time data pipelines on Azure Databricks using Spark (Scala) and SQL. Day-to-day work covers ETL/ELT with medallion architecture and Delta Lake, plus CI/CD, secret management, and observability.
CareCone Group in Sydney is seeking a Data Engineer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL, with a focus on scalable, robust data platforms.
You will develop ETL/ELT processes following medallion architecture (Bronze/Silver/Gold) and Delta Lake, deliver real-time and near real-time integrations, and implement CI/CD, secret management, and observability to support analytics across the business.