Senior Data Engineer
Posted
Summary
Senior Data Engineer in Singapore designing and operating enterprise Lakehouse/data platform solutions. Day to day involves building batch, streaming and API ingestion pipelines, data products and marketplace capabilities using Spark/PySpark, Databricks, Snowflake, Kafka, Airflow, Iceberg/Delta Lake and Kubernetes-based CI/CD tooling.
Experience : 10+ Years
Role : Senior Data Engineer
Key Skills:
8-12 years of experience in Data Engineering, Big Data, Data Lake, or Lakehouse implementations.
Hands-on experience with Databricks, Snowflake, Cloudera, Azure, AWS, GCP, Huawei, or Alibaba data platforms.
Hands-on experience in developing Data products and Market place
Strong expertise in Spark, PySpark, SQL, Python and Scala.
Strong programing skills (Java, Scala, Python, SQL)
Experience with Iceberg, Hudi, Delta Lake and object storage platforms.
Experience implementing data ingestion, transformation, reconciliation and data quality frameworks.
Experience with Trino, Dremio, Hive, Impala, Kafka, Flink, Spark Streaming and Airflow.
Strong hands on experience with Kubernetes, OpenShift, Docker, CI/CD, MLflow and observability tools.
Ability to design data architectures supporting NLP and AI‑driven analytics, including ingestion, curation, and governance of unstructured data within Data Lake, Data warehouse platforms.
Experience working with ML platforms such as CML, Spark MLlib, and Python ML libraries (scikit‑learn, XGBoost), including model deployment.
Develop full‑stack applications and internal engineering tools using Python, shell scripting, and modern web frameworks (e.g., Flask, React).
Knowledge of data modelling, metadata management, lineage and governance.
Experience exposing data through APIs, event streams, dashboards and BI platforms.
Knowledge of Teradata, Netezza, Greenplum or MPP migration programs is advantageous.
Experience with Kubernetes, OpenShift, Terraform, Jenkins, Git and CI/CD pipelines.
Responsibilities:
Implement and operationalize enterprise Lakehouse platforms, data products, and data marketplace capabilities.
Develop scalable batch, streaming, CDC, and API-based data ingestion pipelines.
Develop scalable multimodal data ingestion pipelines including content extraction from various file formats, regex for specific field extraction, content extraction from embedded images, frame extraction from video files, transcript extraction from audio files, etc
Build, test, and maintain foundation and business data products with agreed data contracts, SLAs, and data quality controls.
Implement open table formats such as Iceberg, Hudi, and Delta Lake.
Support RAG, vector search, GenAI and agentic data pipelines.
Perform performance tuning, optimization, production support, and root cause analysis.
Create technical documentation, deployment guides, and operational runbooks.
Ensure compliance with engineering standards, DevSecOps controls, and software delivery practices.
Requirements: EDUCATION
Bachelor’s degree in Computer Science, Engineering or related discipline
PREFERRED CERTIFICATIONS
Databricks Certified Data Engineer
Azure Data Engineer Associate
AWS Data Analytics Specialty
Google Professional Data Engineer
SnowPro Certification
DAMA CDMP
Strong engineering and automation mindset.
Excellent troubleshooting and performance optimization skills.
Ability to work across distributed teams and multiple projects.
Strong communication and stakeholder management skills.
Experience in Agile delivery and enterprise-scale platforms.
Commitment to quality, operational excellence and continuous improvement.
Key Skills:
8-12 years of experience in Data Engineering, Big Data, Data Lake, or Lakehouse implementations.
Hands-on experience with Databricks, Snowflake, Cloudera, Azure, AWS, GCP, Huawei, or Alibaba data platforms.
Hands-on experience in developing Data products and Market place
Strong expertise in Spark, PySpark, SQL, Python and Scala.
Strong programing skills (Java, Scala, Python, SQL)
Experience with Iceberg, Hudi, Delta Lake and object storage platforms.
Experience implementing data ingestion, transformation, reconciliation and data quality frameworks.
Experience with Trino, Dremio, Hive, Impala, Kafka, Flink, Spark Streaming and Airflow.
Strong hands on experience with Kubernetes, OpenShift, Docker, CI/CD, MLflow and observability tools.
Ability to design data architectures supporting NLP and AI‑driven analytics, including ingestion, curation, and governance of unstructured data within Data Lake, Data warehouse platforms.
Experience working with ML platforms such as CML, Spark MLlib, and Python ML libraries (scikit‑learn, XGBoost), including model deployment.
Develop full‑stack applications and internal engineering tools using Python, shell scripting, and modern web frameworks (e.g., Flask, React).
Knowledge of data modelling, metadata management, lineage and governance.
Experience exposing data through APIs, event streams, dashboards and BI platforms.
Knowledge of Teradata, Netezza, Greenplum or MPP migration programs is advantageous.
Experience with Kubernetes, OpenShift, Terraform, Jenkins, Git and CI/CD pipelines.
Responsibilities:
Implement and operationalize enterprise Lakehouse platforms, data products, and data marketplace capabilities.
Develop scalable batch, streaming, CDC, and API-based data ingestion pipelines.
Develop scalable multimodal data ingestion pipelines including content extraction from various file formats, regex for specific field extraction, content extraction from embedded images, frame extraction from video files, transcript extraction from audio files, etc
Build, test, and maintain foundation and business data products with agreed data contracts, SLAs, and data quality controls.
Implement open table formats such as Iceberg, Hudi, and Delta Lake.
Support RAG, vector search, GenAI and agentic data pipelines.
Perform performance tuning, optimization, production support, and root cause analysis.
Create technical documentation, deployment guides, and operational runbooks.
Ensure compliance with engineering standards, DevSecOps controls, and software delivery practices.
Requirements: EDUCATION
Bachelor’s degree in Computer Science, Engineering or related discipline
PREFERRED CERTIFICATIONS
Databricks Certified Data Engineer
Azure Data Engineer Associate
AWS Data Analytics Specialty
Google Professional Data Engineer
SnowPro Certification
DAMA CDMP
Strong engineering and automation mindset.
Excellent troubleshooting and performance optimization skills.
Ability to work across distributed teams and multiple projects.
Strong communication and stakeholder management skills.
Experience in Agile delivery and enterprise-scale platforms.
Commitment to quality, operational excellence and continuous improvement.
Skills
- Agentic AI
- Agile
- AI
- Airflow
- Analytics
- API
- Automation
- AWS
- Azure
- Bash
- CI/CD
- Data Analytics
- Data Engineering
- Data Ingestion
- Data Lake
- Data Modeling
- Data Pipelines
- Data Quality
- Data Warehousing
- Databricks
- Delta Lake
- DevSecOps
- Docker
- Flask
- Flink
- GCP
- Generative AI
- Git
- Hive
- Java
- Jenkins
- Kafka
- Kubernetes
- Lakehouse
- Machine Learning
- Metadata Management
- MLflow
- Model Deployment
- NLP
- Observability
- OpenShift
- PySpark
- Python
- React
- Scala
- scikit-learn
- Snowflake
- Spark
- SQL
- Stakeholder Management
- Teradata
- Terraform
- Trino
- Vector Search
- XGBoost