Data Engineer
Posted 11
views
Summary
Data engineer who designs data lakehouse architectures and builds ETL/ELT pipelines (batch and real-time), working onsite at a client's office in Petaling Jaya on a healthcare-industry project for a Malaysian ICT systems integration firm. Core stack: Python/SQL/Scala, Spark, Delta Lake/Iceberg, Databricks, Snowflake, Airflow, dbt.
- A Malaysian ICT professional services and systems integration company established on 22 March 1999.
- 25+ years of experience delivering enterprise ICT and systems integration solutions across Malaysia.
- Exposure to enterprise-scale digital transformation projects, including cloud, cybersecurity, networking, infrastructure, and managed services.
Project/Goal:
- Working in client's office, serving current project in the healthcare industry
Key Responsibilities:
- Design and implement scalable data lakehouse architectures using modern technologies.
- Build and maintain ETL/ELT pipelines for batch and real-time data processing
- Develop data ingestion frameworks from various sources (APIs, databases, files, streams)
- Optimize data storage, partitioning, and query performance in lakehouse environments
- Support machine learning workflows and AI model deployment pipelines
- Design disaster recovery and backup strategies for critical data assets
Skills Requirements:
- Programming: Python, Scala, Java, SQL, Shell scripting
- Big Data: Apache Spark, Hadoop ecosystem, Delta Lake, Apache Iceberg
- Data Platforms: Databricks, Snowflake, Apache Airflow, dbt
Good to Have:
- Experience with real-time streaming platforms (Kafka, Kinesis, Pub/Sub)
- Experience with MLOps and ML pipeline orchestration
- Understanding of data privacy regulations (GDPR, Malaysian PDPA)
- Previous experience in financial services, e-commerce, or technology sectors
Work Location:
- Petaling Jaya, onsite
- Mid to Senior level with 4-6 years of experience.