Data Engineer
Posted 1
view
Summary
A data engineer in Kuala Lumpur who designs, builds, and scales an AWS data platform, developing real-time and batch ingestion pipelines with CDC, PySpark, and SQLMesh/DBT, managing S3/RDS components, orchestration via Airflow or Dagster, and data lake modeling while ensuring data quality and reliability.
Role Summary
Design, build, and scale a modern AWS data platform and robust data ingestion pipelines.
Roles & Responsibilities
- Pipeline Architecture & Development:Design and implement scalable real-time and batch data ingestion pipelines using CDC approaches, PySpark, and SQLMesh or DBT.
- AWS Platform Management:Build and manage AWS data platform components (S3, RDS) and implement orchestration and scheduling tools like Airflow or Dagster.
- Data Modeling & Warehousing:Develop structured data lake layers (raw/staged/curated) and optimize storage formats and query performance.
- Quality & Reliability:Implement robust data validation processes, monitor pipeline health, and systematically troubleshoot production issues.
- Continuous Improvement:Evaluate new technologies through POCs to enhance system architecture, scalability, and cost efficiency.
Requirements
- Education:Bachelor’s or Master’s Degree in Computer Science, Engineering, or a related field.
- Experience:3–5+ years of proven experience in Data Engineering or related roles.
- Technical Skills:Strong proficiency in SQL, Python, PySpark, transformation tools (SQLMesh, DBT), AWS data pipelines, CDC concepts (Debezium/Kafka), and AWS RDS.
- Core Knowledge:Deep understanding of data architecture, data modeling, warehousing concepts, and distributed systems fundamentals.
- Preferred Skills:Familiarity with the Kafka ecosystem, orchestration tools (Airflow, Dagster), open table formats (Iceberg, Delta Lake), Docker & Kubernetes, S3/MinIO, and ClickHouse.
- Candidate Profile:Ownership-driven with a strong engineering mindset, highly independent, and capable of balancing fast delivery with scalable system design.
Benefits
- 13-Month Salary:Guaranteed additional month of base salary.
- Yearly Performance Bonus:Rewarded annually based on individual and company achievements.
- Monthly Transport Allowance:Financial support to cover daily commuting expenses.
- Monthly Rental Subsidies:Monthly allowance provided to support your housing and accommodation needs.
Please note: Only shortlisted candidates will be notified.