Data Engineer AMK
Posted 1
view
Summary
Data engineer in Singapore who migrates workflow integrations to new platforms, builds orchestration and ETL pipelines for file-based ingestion into relational databases, and supports Lakehouse batch/streaming environments. Core stack includes SQL/SQL Server, Spark, Python, REST APIs and cloud data services.
Job Responsibilities
Analyse and migrate existing workflow integrations to new platforms. Develop workflow orchestration, including triggers, routing, retries and exception handling. Build automation for batching, pagination, looping and throttling. Design and maintain ETL pipelines for file-based data ingestion into relational databases. Perform schema validation, data quality checks and error reconciliation. Develop data pipelines using cloud-based big data platforms and distributed processing frameworks such as Spark. Support Lakehouse-based data environments for batch and streaming workloads. Implement logging, monitoring and alerting for data pipelines and workflows. Troubleshoot issues and participate in root-cause analysis and continuous improvement. Work with application teams to understand data requirements and upstream/downstream dependencies. Ensure data pipelines meet performance, reliability, security and governance requirements. Job Requirements
3-5 years
of relevant experience in Data Engineering, Integration Engineering or a related role. Experience with
workflow orchestration or integration platforms . Hands-on experience with
ETL
and file-based data ingestion. Proficient in
SQL
and relational databases such as
SQL Server . Experience with
Spark
or other distributed data processing frameworks. Experience with
Python
or other programming/scripting languages. Understanding of
batch and streaming data processing . Experience with
cloud platforms and managed data services . Experience working with
REST APIs, JSON and CSV . Understanding of
CI/CD
for data and integration workflows. Good understanding of data quality, error handling and pipeline reliability. Good to Have
Experience with
Lakehouse table formats , incremental processing and time travel. Experience with
Flink or Spark Streaming . Familiarity with analytical query engines. Experience in banking, financial services, manufacturing or other regulated environments. Knowledge of data observability and data quality tools.
Analyse and migrate existing workflow integrations to new platforms. Develop workflow orchestration, including triggers, routing, retries and exception handling. Build automation for batching, pagination, looping and throttling. Design and maintain ETL pipelines for file-based data ingestion into relational databases. Perform schema validation, data quality checks and error reconciliation. Develop data pipelines using cloud-based big data platforms and distributed processing frameworks such as Spark. Support Lakehouse-based data environments for batch and streaming workloads. Implement logging, monitoring and alerting for data pipelines and workflows. Troubleshoot issues and participate in root-cause analysis and continuous improvement. Work with application teams to understand data requirements and upstream/downstream dependencies. Ensure data pipelines meet performance, reliability, security and governance requirements. Job Requirements
3-5 years
of relevant experience in Data Engineering, Integration Engineering or a related role. Experience with
workflow orchestration or integration platforms . Hands-on experience with
ETL
and file-based data ingestion. Proficient in
SQL
and relational databases such as
SQL Server . Experience with
Spark
or other distributed data processing frameworks. Experience with
Python
or other programming/scripting languages. Understanding of
batch and streaming data processing . Experience with
cloud platforms and managed data services . Experience working with
REST APIs, JSON and CSV . Understanding of
CI/CD
for data and integration workflows. Good understanding of data quality, error handling and pipeline reliability. Good to Have
Experience with
Lakehouse table formats , incremental processing and time travel. Experience with
Flink or Spark Streaming . Familiarity with analytical query engines. Experience in banking, financial services, manufacturing or other regulated environments. Knowledge of data observability and data quality tools.