Data Engineer- Azure Databricks
Summary
A permanent Data Engineer role (6–12 years) in Sydney, hired via IT recruitment agency Renaissance Infosystems, to design, build, and optimize high-throughput data pipelines on Azure Databricks using Spark/Scala, SQL, and Python, with Delta Lake medallion architecture, real-time streaming (Kafka), and CI/CD. No visa sponsorship offered.
About Us
Renaissance Info Systems is a technology and digital recruitment agency, connecting contract and permanent professionals with clients across Asia-Pacific.
We aim to differentiate ourselves through our level of responsiveness, and our understanding that comes from being an IT recruitment agency from the IT Industry. Our recruiters balance sophisticated and simple inter-personal techniques to assure a strong candidate network.
Hiring for Data Engineer- Azure Databricks
Role/Title: Data Engineer- Azure Databricks - 6-12 Years
Mode of Hiring: Permanent
Location: Sydney
Mandatory skills
- Azure Databricks
- Scala Spark
- SQL
- Python
Good to have skills
- Databricks, SQL, Python, Data warehousing concepts
- Detailed Job description (Role & Responsibility, Skill and Experience range)
- We are looking for a hands on Developer Programmer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL
- Design & develop scalable ETL/ELT pipelines in Azure Databricks using Spark (Scala) and SQL, following medallion architecture (Bronze/Silver/Gold) and Delta Lake best practices.
- Proven expertise in designing real time and near real time integrations
- Operationalize workloads: jobs & workflows scheduling, CI/CD with Git (e.g., Azure DevOps), secrets management, and observability.
- Expertise in cloud platforms and related technologies, specifically:
- Hyperscalars Azure, AWS
- Data Platforms '¿ (e.g. Databricks) and big data technologies.
- Big Data Hive, Impala, Sqoop, Spark (Scala/ Python)
- Real time data processing frameworks Spark Streaming, Apache/Confluent Kafka
- CICD tools '¿ GitHub, Confluence, Jira, Jenkins
- Proficiency in SQL, Python, and data integration tools.
- Familiarity with Master Data Management solutions
- Excellent communication and stakeholder management skills.