Lead Data Engineer
Summary
Leads data engineering initiatives, building and maintaining cloud-based data pipelines and warehouses using Python, Scala, SQL, Azure, Databricks, and Big Data tools.
Overview
Job Description for Lead Data Engineer in Sydney
Responsibilities
- Leads data engineering initiatives with a focus on data warehousing, data analysis, visualization and data modeling.
- Designs and implements data solutions and pipelines using Python, Scala and SQL.
- Works with cloud platforms (Azure and Databricks) to build and maintain data infrastructure.
- Leverages Big Data technologies (Cloudera: HDFS, Spark, Hive, Kafka) and Apache Airflow for data workflows.
- Applies DevOps practices with Azure DevOps (Git, Repos, Pipelines), Git, Terraform and Docker.
- Supports reporting needs using SQL Server Reporting Services and Power BI.
Qualifications
- 10+ years of experience in software engineering, cloud computing, Databricks and Big Data.
- Data Engineering, Data Warehousing, Solution Design, Data Analysis, Visualization and Data Modeling.
- Programming: Python, Scala, SQL.
- Cloud: Azure and Databricks.
- Big Data: Cloudera (HDFS, Spark, Hive, Kafka), Apache Airflow.
- DevOps: Azure DevOps (Git, Repos, Pipelines), Git, Terraform, Docker.
- Reporting: SQL Server Reporting Services, Power BI.
- Certified DevOps Engineer Expert, Azure Cloud, Azure Data Lake.