Data Engineer - Azure Databricks
Summary
Data engineer at AC3 in Sydney building and maintaining Azure data pipelines and platforms, transitioning workloads into Databricks Medallion architecture. Day to day: data modelling, ETL/ELT scheduling and monitoring, and visualisations using Azure (ADF, Functions), SQL and Python.
- Hybrid work with 2 days/week in Sydney CBD office
- Great team, collaborative and supportive environment
- ASAP start - $1,100-$1,200/day
- Hybrid work with 2 days/week in Sydney CBD office
- Great team, collaborative and supportive environment
- ASAP start - $1,100-$1,200/day
About the company
At AC3, our purpose is to make technology real, and as the leader in secure multi-cloud solutions, we get to do this for our customers every day. We are an Australian owned IT Services Company, with offices in Sydney, Melbourne, Brisbane and Auckland, servicing over 50% of the NSW State Government and over 1,200 enterprise customers in Australia & NZ.
Our expertise:
- Cloud and infrastructure
- Cyber security
- Software and data integration
- ServiceNow
The position
As aData Engineer, you'llbring your expertise to help transition some elements into the Databricks Medallion architecture.
- Create and maintain a data pipeline and infrastructure that utilizes diverse storage technologies.
- Develop a scalable data platforms using APIs and quality frameworks for increasing data volume and complexity.
- Assist in the establishment of processes and standards for data mining, data modeling, and data protection.
- Create data models into Databricks and Medallion architecture
- Create clear and effective data visualisations to communicate findings to stakeholders.
- Collaborate with the broader data and analytics team to enhance data models.
- Assist with building and optimising data pipelines and infrastructure
Requirements
- 5+ years of experience as a Data Engineer
- Experience developing processes that support data transformation, data structures, and the manipulation, processing, and extraction of value from datasets.
- Experience in machine learning, data mining, mathematics and statistical modelling and analysis.
- In depth knowledge of data platform concepts, including data lakes, data warehousing, big data processing, real-time processing architecture (ADF, Databricks, Logic Apps, Functions, Data Lake Gen2)
- Scheduling, and monitoring of ETL/ELT workflows.
- Experience with SQL, Python, SQL Server, NoSQL and Oracle databases.
- Experience with data pipeline and workflow management tools, and Cloud data platforms.
- Bachelor’s Degree in Data Science or a similar related field.
Additional information
- Hybrid work with 2 days/week in Sydney CBD office
- Great team, collaborative and supportive environment
- ASAP start - $1,100-$1,200/day