Azure Data Engineer
Posted 1
view
Summary
A Sydney-based Azure Data Engineer role focused on building complex data pipelines and optimizing SQL/NoSQL data stores day to day, using Python, SQL, PySpark, Azure Synapse, ADF, Azure Functions, batch-processing clusters, and DevOps tooling like Azure DevOps and Git, with MLOps knowledge expected.
- Minimum 8+ years in development/software engineering and has extensively knowledge of working in Agile organisations.
- Strong Python and SQL skill and good to have Java experience.
- Have hands-on experience building complex data pipelines,
- Have extensive experience with Apache PySpark, Python, Azure Synapse, ADF, Azure Functions, data models and ETL pipelines.
- Have experience in setting up and optimizing both SQL and NoSQL data stores (MS-SQL, ADLS Gen2), as well as familiarity with object storage services (e.g., ADLS Gen2, Big Data),
- Experience in managing and further developing distributed systems and clusters for batch processing,
- Knowledge of MLOps architecture and practices,
- Experience with modern data modelling concepts and large-scale data warehousing architecture.
- Basic understanding of modern DevOps tool set, e.g. Azure DevOps, Git.
- Knowledge of ETL and data warehousing methods, good to have Informatica experience.