Data Engineer
Posted 5
views
Summary
A data engineer in Singapore who migrates and builds data ingestion pipelines and ETL/ELT workflows into a T1 data layer, with data quality automation. Core stack: Linux, Python, shell scripting, Docker/Kubernetes, Kafka, Airflow, Apache Iceberg, Trino, Datahub, Ranger, Minio and ADLS over S3 protocols.
Position : Data Engineer
- Migratedatapipelines from existingdataacquisition framework to the new GDPdataacquisition framework
- Configure,developand deliverdataingestion scripts for loadingdatainto T1datalayer
- Develop and manage ETL/ELT workflows, ensuringdataquality, integrity, and reliability.
- Integrate and automatedataquality checks and validation processes withindatapipelines.
- Deploy and manage containerized applications using Docker and orchestrate workloads on Kubernetes.
- Work with moderndatalake and warehouse technologies such as Iceberg
- Implement real-timedatastreaming solutions using Kafka.
- Orchestrate complex workflows using Airflow.
- Integrate withdatacatalogand governance tools such asDatahub and Ranger.
- Collaborate with cross-functional teams to understand business requirements and deliverdatasolutions.
- Ensure security, compliance, and best practices indatamanagement and governance.
- Strongproficiencyin Linux, Python, and Shell scripting.
- Hands-on experience with Docker, Kubernetes, and container orchestration.
- Hands-on experience with Minio and AzureDataLake Storage (ADLS) using S3 protocols.
- Experience withApacheIceberg, Kafka, Airflow,Datahub, Trino, and Ranger. Work Location : Singapore.