Data Platform Integration Engineer
Description
Salary: RM 6,000 – RM 8,000 per month
About the role
Join the data platform operations team to maintain and optimise core integration pipelines that support reliable data ingestion, real-time processing and data delivery. You will work across NiFi, Kafka, Snowflake, Azure Event Hub, Flink and PostgreSQL, collaborating with upstream and downstream teams to keep production data flows stable, observable and performant.
Key responsibilities
Perform daily maintenance, monitoring, troubleshooting and performance optimisation of core data platform integration pipelines.
Maintain NiFi, Kafka and Snowflake data ingestion and data-landing pipelines, ensuring reliable data flows from multiple source systems into Snowflake.
Maintain Event Hub, Flink and PostgreSQL real-time data processing pipelines, ensuring stable and reliable computation and writes.
Manage daily operations, capacity assessment and incident handling for Kafka, Event Hub, Flink, NiFi, Snowflake and PostgreSQL.
Work with upstream and downstream teams on data onboarding, interface integration, data quality checks and production issue diagnosis.
Support production environment changes, releases, on-call response and emergency incident handling.
Use APM and log-monitoring tools to track data pipeline health, detect anomalies early and drive issues through to resolution.
Prepare English technical documents, operation manuals, incident post-mortems and handover materials to support smooth project transitions.
Participate in a fully English working environment and communicate with overseas teams.
Support production on-call duties, change activities and emergency issue response, including short-term travel or cross-time-zone collaboration when required by the project.
What we're looking for
Bachelor's degree or above in Computer Science, Software Engineering, Information Management or a related discipline.
Around three years of experience in data platforms, data integration, big data operations or real-time data development.
Hands-on experience with Apache NiFi, including data ingestion, flow configuration, troubleshooting and performance optimisation.
Experience with Apache Kafka or Azure Event Hub and an understanding of messaging concepts, consumer lag handling and common operational issues.
Experience maintaining Apache Flink real-time stream processing jobs, including state management, checkpoints or troubleshooting.
Experience with Snowflake or another cloud data warehouse, including data loading, table design, basic query optimisation and access management.
Experience with PostgreSQL, including database connectivity, write performance, indexes, locks and common troubleshooting.
Proficiency with Linux, including using logs, monitoring and command-line tools to locate issues.
Ability to use monitoring tools such as Splunk and Grafana.
Ability to use orchestration tools such as Airflow and Azure Functions.
Strong ownership, problem-analysis and cross-team collaboration skills, with the ability to work effectively under pressure.
Experience with real-time data platforms, data pipelines, ETL/ELT or unified batch-and-stream projects is advantageous.
Experience with Azure, AWS or other cloud platform data services is advantageous.
Experience with APM, log monitoring, alerting framework build-out or production incident troubleshooting is advantageous.
Experience with international projects, overseas team collaboration or 24/7 production support is advantageous.