Data Engineer (Azure Databricks)
Posted 3
views
Summary
Design, build, and optimize high-throughput ETL/ELT data pipelines in Azure Databricks using Spark (Scala), SQL, Delta Lake medallion architecture, and real-time streaming (Spark Streaming, Kafka), then operationalize them with CI/CD, secrets management, and observability in a Sydney-based role.
- We are looking for a Data Engineer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL
- Design & develop scalable ETL/ELT pipelines in Azure Databricks using Spark (Scala) and SQL, following medallion architecture (Bronze/Silver/Gold) and Delta Lake best practices.
- Proven expertise in designing real time and near real time integrations
- Operationalize workloads: jobs & workflows scheduling, CI/CD with Git (e.g., Azure DevOps), secrets management, and observability.
Expertise in cloud platforms and related technologies, specifically:
- Data Platforms (e.g. Databricks) and big data technologies.
- Real time data processing frameworks Spark Streaming, Apache/Confluent Kafka
- CICD tools GitHub, Confluence, Jira, Jenkins
- Proficiency in SQL, Python, and data integration tools.
- Familiarity with Master Data Management solutions
- Excellent communication and stakeholder management skills.