Data Engineer- NV1 Security Clearance
Posted 19
views
Summary
Data engineer building and debugging large-scale ETL pipelines on the Cloudera Data Platform (Spark, Airflow, NiFi, Hive, Impala, Iceberg) for agency/government data-warehouse projects in Melbourne. Candidates must already hold NV1 security clearance; Defence experience is a plus.
- Security clearance: Candidates must hold NV1 clearance
- ETL pipeline development in Spark / Cloudera Data Engineering / Airflow (critical requirement)
- ETL pipeline development using NiFi within Cloudera Data Platform (strong requirement)
- ETL development using DDLs and Erwin (Strong requirement)
- ETL development experience using Ozone & Iceberg, Hive, Hue and Impala (critical requirement)
- Configuration and management of Cloudera Data Platform environments and tenancies, including complex ranger and atlas security & access controls, linked to enterprise identity management services, such as EntraID
- Complex ETL pipeline debugging and design of various methods of ETL, including truncate & reload etc (critical requirement)
- Developer / Engineering skills in Cloudera Data Visualisation (nice to have)
- Developer / Engineering / Integration skills in MSSQL & PowerBI (nice to have)
- Excellent problem solving, communication and task management skills (strong requirement)
- Previous experience working on large agency data warehouses and data integration projects (strong requirement)
- Pervious experience working in Defence (desirable)