Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Showtime Consulting Pty Ltd

Senior Data Engineer: Secure Cloud Data Pipelines

Posted Updated

Description

Showtime Consulting Pty Ltd is seeking an experienced Data Engineer to join a large-scale enterprise data environment in Sydney. You will design, build and optimise high-throughput data pipelines using Azure Databricks, Spark (Scala), SQL and Python, delivering scalable ETL/ELT solutions while applying Medallion Architecture and Delta Lake best practices.

Collaborate with stakeholders across data, engineering and delivery to enable real-time insights.

Design, build and optimise high-throughput data pipelines using Azure Databricks. Develop scalable ETL/ELT pipelines using Spark (Scala), SQL and Python. Implement data solutions following Medallion Architecture across Bronze, Silver and Gold layers. Apply Delta Lake principles and best practices across data workloads. Design and develop real-time and near-real-time data integrations. Develop data processing solutions using Spark Streaming and Apache/Confluent Kafka. Build and manage Databricks jobs, workflows and scheduling processes. Support CI/CD pipelines using Git and tools such as Azure DevOps. Implement appropriate secrets management and observability practices. Work across cloud platforms including Azure and AWS. Support large-scale data processing using technologies including Spark, Hive, Impala and Sqoop. Collaborate with technical, business and delivery stakeholders to deliver effective data solutions. Hands-on experience with Azure Databricks. Commercial experience with Spark (Scala). Strong SQL skills. Strong Python skills. Experience designing scalable ETL/ELT data pipelines. Strong understanding of modern data warehousing concepts. Experience with Medallion Architecture and Delta Lake. Proven experience designing real-time and near-real-time integrations. Experience with cloud platforms, particularly Azure and/or AWS. Experience with big data technologies such as Hive, Impala, Sqoop and Spark. Experience with real-time data processing frameworks such as Spark Streaming and Apache/Confluent Kafka. Understanding of CI/CD practices and associated tools. Excellent communication and stakeholder management skills.

Skills

What Senior Data Engineering jobs ask for — and how much of it you have →
Apply

See also

Data Engineering jobs by country — openings, pay and top skills →