Data Engineer-Azure Databricks/ AWS Data Engineer

Greetings from SolarSys Innovation.

We have full time permanent role with our client in Sydney, Australia.

Job Description for Azure Data Engineer

Mandatory Skills :

Primary or Mandatory skills

1. Azure Databricks

2. Scala Spark

3. SQL

4. Python

Good to have skills

Databricks, SQL, Python, Data warehousing concepts

Detailed Job description (Role & Responsibility, Skill and Experience range)

We âre looking for a hands on Developer Programmer to design, build, and optimize high throughput data pipelines on Azure Databricks using Spark (Scala) and SQL

Design & develop scalable ETL/ELT pipelines in Azure Databricks using Spark (Scala) and SQL, following medallion architecture (Bronze/Silver/Gold) and Delta Lake best practices.

Proven expertise in designing real time and near real time integrations

Operationalize workloads: jobs & workflows scheduling, CI/CD with Git (e.g., Azure DevOps), secrets management, and observability.

Expertise in cloud platforms and related technologies, specifically:

o Hyperscalars Azure, AWS

o Data Platforms (e.g. Databricks) and big data technologies.

o Big Data Hive, Impala, Sqoop, Spark (Scala/ Python)

o Real time data processing frameworks Spark Streaming, Apache/Confluent Kafka

o CICD tools GitHub, Confluence, Jira, Jenkins

Proficiency in SQL, Python, and data integration tools.

Familiarity with Master Data Management solutions

Excellent communication and stakeholder management skills.

Job Description for AWS Data Engineer

The Senior AWS Data Engineer is responsible to design, build, and support scalable data pipelines and curated datasets on AWS. He/She will work with cross-functional teams to ingest, transform, and serve data for reporting, analytics, and downstream applications. The ideal candidate is hands-on, strong in SQL/Python, and experienced with AWS-native data services and modern data engineering practices.

Key Responsibilities

• Design, develop, and maintain end-to-end data pipelines (batch and near real-time) on AWS Data Platform

• Build and manage ETL/ELT workflows using AWS services (e.g., AWS Glue, S3, Redshift, Athena, EMR), dbt and orchestration tools such as Airflow

• Implement data ingestion patterns from diverse sources (databases, APIs, files, event streams) into lake/warehouse layers such as raw, cleansed, and curated data layers

• Develop transformation logic using SQL and Python/PySpark for cleansing, enrichment, and standardisation

• Implement robust data quality checks, reconciliation controls, and monitoring/alerting for failures and anomalies

• Collaborate with data analysts/data scientists to model datasets for analytics and machine learning consumption.

• Contribute to DataOps/DevOps practices: version control, CI/CD, automated testing, release management, and operational support.

• Produce and maintain technical documentation (data flows, mappings, job schedules, runbooks, and operational procedures)

• Optimise Data Pipeline performance and Support workflow orchestration and scheduling

• Support production deployments and operations

Required Skills & Experience

• 8–10 years’ experience as a Data Engineer

• Advanced SQL skills

• Hands-on experience working with Teradata and Siebel CRM data sets

• Experience delivering data pipelines in a large-scale enterprise data platform environment

• Strong hands-on AWS experience with common data services such as : Amazon S3, AWS Glue, Amazon Redshift, Amazon Athena, Amazon EMR and dbt

• Strong programming capability in Python and strong data transformation experience using PySpark (preferred) and/or Spark.

• Advanced SQL skills (query optimisation, complex joins, window functions, performance tuning)

• Experience with workflow orchestration tools such as Airflow

• Solid understanding of data warehousing concepts (dimensional modelling, partitioning, incremental loads, CDC concepts).

• Experience implementing monitoring, logging, alerting, and operational support processes.

• Strong communication skills and ability to work with stakeholders to translate requirements into data deliverables

• Telco Industry Experience is highly desirable