Data Engineer
Posted 5
views
Summary
A data engineer in Singapore who designs and builds scalable ETL/ELT pipelines and lakehouse data platforms, deploying them with Kubernetes, Docker and CI/CD in on-premise environments, and provides L3 support and performance tuning. Core stack: Python, SQL Server, MSBI, PySpark/Apache Spark, Airflow, Iceberg.
Role & Responsibilities
Responsibilities
- Design and develop scalable ETL/ELT data pipelines using Python, PySpark, MSBI and Apache Spark.
- Build and manage lake house/data platforms, including data modelling, partitioning, metadata and Iceberg tables.
- Deploy and manage data pipelines using Kubernetes, Docker, CI/CD and DevOps practices in on-premise environments.
- Provide L3 support, optimise data platform performance, ensure security/governance and collaborate with technical and business stakeholders.
Requirements
- Demonstrated experience in data engineering/pipeline development, with strong Python, SQL Server and MSBI expertise.
- Hands-on experience with Kubernetes, Docker, Apache Spark/PySpark and Apache Airflow for data processing and orchestration.
- Strong data modelling skills across relational and NoSQL databases, including MongoDB or Cassandra.
- Experience with lakehouse/data lake architecture, Iceberg, partitioning, metadata management, CI/CD, Git and IaC.
- Strong understanding of data security, governance, RBAC and data masking.
NOTE: - Only shortlisted candidates will be contacted back.