Senior Data Engineer - Data Lakehouse

Salary: RM 8,000 – RM 12,000 per month

About the role
This is a hands-on, senior individual-contributor role with a strong passion on building a possible vendor neutral lakehouse. You will own data pipelines end to end and build out the governance and data-quality layer that a fast-growing platform now needs.

Our stack (so you know what you'd be working with)

• Ingestion / streaming: Debezium CDC → Apache Kafka

• Lakehouse: Apache Iceberg tables on S3, Apache Polaris (Iceberg REST catalog); Kafka Connect and Apache Flink write into it.

• Processing: Apache Flink (real-time upserts / streaming) and Apache Spark (batch silver/gold), both reading and writing Iceberg. Serving / query: ClickHouse

• Orchestration: Dagster.

• Platform / IaC: AWS, Amazon EKS (Kubernetes), Terraform, Helm; observability via Prometheus, Grafana, and Alertmanager.

We value experience with equivalent tools too — if you've done Delta/Hudi instead of Iceberg, Kinesis instead of Kafka, or Airflow instead of Dagster, that transfers.

Key responsibilities

• Own, operate and harden the streaming platform for reliability, correctness, and cost at scale.

• Build and tune the processing layers: batch transformations in Spark (silver/gold models) and real-time streaming jobs in Flink, including partitioning, compaction, and job/state tuning.

• Develop and optimize the serving layer (ClickHouse over Iceberg) so analysts and product teams get fast, reliable query access.

• Design data models and physical table layout (partitioning, sort order, compaction, retention) to balance write throughput, query performance, and storage cost across the lakehouse.

• Establish and enforce data governance: catalog organization, access control, data lineage, retention/compliance, and automated data-quality checks.

• Collaborate with data analysts, product teams, and backend engineers to ensure discoverable, well-documented, and usable data access.

• Stay current with the lakehouse ecosystem and propose targeted improvements.
  

Requirements

• 6+ years in data engineering or backend data-intensive roles.

• Deep, hands-on experience on data engineering practices.

• Production experience with an open table format / lakehouse.

• Strong distributed compute: Apache Spark and/or Apache Flink for batch and/or stream processing, including performance tuning is a plus.

• Expert SQL, dimensional/data modeling, and query performance tuning — including experience with an analytical/columnar store (e.g., ClickHouse, Redshift, Snowflake, BigQuery).

• Solid AWS fundamentals: S3, IAM, VPC/networking, and RDS.

• A production-reliability mindset: monitoring, alerting, on-call, and incident root-cause analysis.

• Proficiency in at least one of Python, Java, or Scala for pipeline and streaming code.

• Excellent communication and stakeholder-management skills.
  

Nice to have

• Real-time OLAP experience (ClickHouse, Apache Druid, Apache Pinot).

• Workflow orchestration (Dagster, Airflow, or equivalent).

• Multi-region and/or cross-account data platform experience.

• Data governance, lineage, and quality tooling (OpenLineage/Marquez, Great Expectations, dbt tests, catalog-based access control).

• Security and compliance frameworks (GDPR, HIPAA, SOC 2, etc.).

• Observability stacks (Prometheus, Grafana, Alertmanager).

• Cost optimization for cloud data/streaming platforms.