Azure Data Engineer - Databricks
Summary
Builds and optimizes scalable data pipelines on Azure using Databricks, PySpark, Delta Lake, and related services for ETL/ELT and streaming workloads.
Role : Azure Data Engineer - Databricks
Type of role - Permanent Position
Location : Sydney, Australia
An Azure Data Engineer specializing in Databricks is responsible for designing, building, and optimizing scalable data pipelines on Microsoft Azure, leveraging Databricks, PySpark, Delta Lake, and related services. The role blends ETL/ELT development, data governance, CI/CD automation, and collaboration with analytics teams.
Data Pipeline Development
- Design, build, and optimize batch and streaming pipelines using Azure Databricks, PySpark, Delta Lake, and SQL.
- Implement ETL/ELT frameworks for ingestion from APIs, databases, flat files, and cloud storage.
- Transform data across Bronze, Silver, and Gold layers in a Lakehouse architecture.
Work with Azure Data Factory (ADF), Azure Data Lake Storage Gen2 (ADLS), and Azure Synapse Analytics.
Configure and maintain Databricks clusters, jobs, workflows, and notebooks.
Tune Spark jobs for performance and cost efficiency.