Data Engineer
Posted Updated
Summary
Data engineer in Singapore migrating and building data ingestion pipelines and ETL/ELT workflows, with automated data quality checks, real-time streaming via Kafka, and orchestration using Airflow. Core stack includes Linux, Python, Docker, Kubernetes, Apache Iceberg, Trino, Minio/ADLS, Datahub, and Ranger.
Position : Data Engineer
Migratedatapipelines from existingdataacquisition framework to the new GDPdataacquisition framework
Configure,developand deliverdataingestion scripts for loadingdatainto T1datalayer
Develop and manage ETL/ELT workflows, ensuringdataquality, integrity, and reliability.
Integrate and automatedataquality checks and validation processes withindatapipelines.
Deploy and manage containerized applications using Docker and orchestrate workloads on Kubernetes.
Work with moderndatalake and warehouse technologies such as Iceberg
Implement real-timedatastreaming solutions using Kafka.
Orchestrate complex workflows using Airflow.
Integrate withdatacatalogand governance tools such asDatahub and Ranger
Collaborate with cross-functional teams to understand business requirements and deliverdatasolutions.
Ensure security, compliance, and best practices indatamanagement and governance.
Strongproficiencyin Linux, Python, and Shell scripting.
Hands-on experience with Docker, Kubernetes, and container orchestration.
Hands-on experience with Minio and AzureDataLake Storage (ADLS) using S3 protocols.
Experience with ApacheIceberg, Kafka, Airflow,Datahub, Trino, and Ranger .
Work Location : Singapore.
Migratedatapipelines from existingdataacquisition framework to the new GDPdataacquisition framework
Configure,developand deliverdataingestion scripts for loadingdatainto T1datalayer
Develop and manage ETL/ELT workflows, ensuringdataquality, integrity, and reliability.
Integrate and automatedataquality checks and validation processes withindatapipelines.
Deploy and manage containerized applications using Docker and orchestrate workloads on Kubernetes.
Work with moderndatalake and warehouse technologies such as Iceberg
Implement real-timedatastreaming solutions using Kafka.
Orchestrate complex workflows using Airflow.
Integrate withdatacatalogand governance tools such asDatahub and Ranger
Collaborate with cross-functional teams to understand business requirements and deliverdatasolutions.
Ensure security, compliance, and best practices indatamanagement and governance.
Strongproficiencyin Linux, Python, and Shell scripting.
Hands-on experience with Docker, Kubernetes, and container orchestration.
Hands-on experience with Minio and AzureDataLake Storage (ADLS) using S3 protocols.
Experience with ApacheIceberg, Kafka, Airflow,Datahub, Trino, and Ranger .
Work Location : Singapore.