Data Engineer-Azure Databricks

Greetings from SolarSys Innovation.

We have full time permanent role with our client in Sydney, Australia.

Job Description:

Mandatory Skills :

Primary or Mandatory skills

1. Azure Databricks

2. Scala Spark

3. SQL

4. Python

Good to have skills

Databricks, SQL, Python, Data warehousing concepts

Detailed Job description (Role & Responsibility, Skill and Experience range)

We âre looking for a hands on Developer Programmer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL

Design & develop scalable ETL/ELT pipelines in Azure Databricks using Spark (Scala) and SQL, following medallion architecture (Bronze/Silver/Gold) and Delta Lake best practices.

Proven expertise in designing real time and near real time integrations

Operationalize workloads: jobs & workflows scheduling, CI/CD with Git (e.g., Azure DevOps), secrets management, and observability.

Expertise in cloud platforms and related technologies, specifically:

o Hyperscalars Azure, AWS

o Data Platforms (e.g. Databricks) and big data technologies.

o Big Data Hive, Impala, Sqoop, Spark (Scala/ Python)

o Real time data processing frameworks Spark Streaming, Apache/Confluent Kafka

o CICD tools GitHub, Confluence, Jira, Jenkins

Proficiency in SQL, Python, and data integration tools.

Familiarity with Master Data Management solutions

Excellent communication and stakeholder management skills.