Senior Data Engineer- GCP & Cloud
Posted Updated
Summary
Contract role for a senior data engineer who designs and hands-on builds large-scale transaction data platforms on GCP for enterprise clients — including batch and real-time ingestion pipelines (Confluent Kafka, GCS to Spanner/BigQuery), legacy code transpilation to BigQuery SQL, CI/CD with Cloud Build/Terraform, Cloud Composer orchestration, and data governance with Dataplex. Core stack: Python,
Job Type- Contract
Location- Sydney/Melbourne
JD-
Must have
- Must have: programming knowledge and willingness to be hands-on - Python, Java
In-depth knowledge and experience of GCP data and analytics technologies
- Designing Large scale Transaction Data Platform architecture for (a) migrating open source or public cloud data platforms to GCP cloud native services (b) designing greenfield data platforms on GCP
- Define solution architecture and detailed design, and perform hands-on implementation for –
data pipelines for batch and event driven ingestion and processing of data from a variety of sources such as on-prem files, on-prem databases, APIs, etc.
real-time data ingestion and event-driven processing pipelines from Confluent Kafka and GCS to Spanner and BigQuery
- Automatic transpilation of legacy code (HIVE, Teradata, python logic etc.) to BQ SQL
- CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform
- Orchestration setup and Cloud Composer DAG development for data pipeline workflows
- Data governance solutioning using GCP governance tooling (Dataplex, Data Catalog)
- Experience applying Generative AI technologies and integrations in enterprise environments.