Senior Data Engineer
Posted
Summary
Senior Data Engineer in Singapore who designs, builds and maintains batch and streaming data pipelines and a Delta Lake lakehouse on AWS and Databricks, with Python, PySpark and SQL at the core, plus Terraform/Git CI/CD, data quality, security and pipeline monitoring.
Job Role:
Senior Data Engineer Experience:
610 Years Location:
Singapore Role Overview We are looking for a Senior Data Engineer with strong experience in AWS, Databricks, Python, PySpark and SQL to design, develop and maintain scalable data pipelines and data platforms. Key Responsibilities Design and develop batch and streaming data pipelines using AWS and Databricks. Build and maintain data lakehouse solutions using Delta Lake. Develop data ingestion, transformation and data modelling processes. Implement data quality checks, validation and data governance. Automate workflows using Databricks and AWS services. Monitor and troubleshoot data pipelines and production issues. Optimise data processing performance and cloud costs. Implement data security, access controls and encryption. Review code, follow engineering standards and mentor team members. Maintain technical documentation and operational procedures. Mandatory Skills Strong experience in Python, PySpark and SQL. Hands-on experience with Databricks, Apache Spark and Delta Lake. AWS data services such as S3, Glue, IAM, Lambda and Redshift. Experience in data lake/lakehouse architecture and data modelling. Knowledge of incremental data processing and Change Data Capture (CDC). Experience with data quality, workflow orchestration and monitoring. Knowledge of Git, CI/CD and Terraform or equivalent IaC tools. Experience in performance tuning, troubleshooting and cloud security. Technical leadership, code reviews or mentoring experience. Preferred Skills Databricks Unity Catalog, Lakeflow and Structured Streaming. Apache Airflow, dbt, Kafka or AWS Kinesis. Experience with ML/AI data pipelines. Relevant AWS or Databricks certifications. Education Bachelors degree in Computer Science, Information Technology, Data Engineering or a related field.
Senior Data Engineer Experience:
610 Years Location:
Singapore Role Overview We are looking for a Senior Data Engineer with strong experience in AWS, Databricks, Python, PySpark and SQL to design, develop and maintain scalable data pipelines and data platforms. Key Responsibilities Design and develop batch and streaming data pipelines using AWS and Databricks. Build and maintain data lakehouse solutions using Delta Lake. Develop data ingestion, transformation and data modelling processes. Implement data quality checks, validation and data governance. Automate workflows using Databricks and AWS services. Monitor and troubleshoot data pipelines and production issues. Optimise data processing performance and cloud costs. Implement data security, access controls and encryption. Review code, follow engineering standards and mentor team members. Maintain technical documentation and operational procedures. Mandatory Skills Strong experience in Python, PySpark and SQL. Hands-on experience with Databricks, Apache Spark and Delta Lake. AWS data services such as S3, Glue, IAM, Lambda and Redshift. Experience in data lake/lakehouse architecture and data modelling. Knowledge of incremental data processing and Change Data Capture (CDC). Experience with data quality, workflow orchestration and monitoring. Knowledge of Git, CI/CD and Terraform or equivalent IaC tools. Experience in performance tuning, troubleshooting and cloud security. Technical leadership, code reviews or mentoring experience. Preferred Skills Databricks Unity Catalog, Lakeflow and Structured Streaming. Apache Airflow, dbt, Kafka or AWS Kinesis. Experience with ML/AI data pipelines. Relevant AWS or Databricks certifications. Education Bachelors degree in Computer Science, Information Technology, Data Engineering or a related field.
Skills
- AI
- Airflow
- AWS
- CI/CD
- Cloud
- Cloud Security
- Data Engineering
- Data Governance
- Data Ingestion
- Data Lake
- Data Modeling
- Data Pipelines
- Data Quality
- Databricks
- dbt
- Delta Lake
- Git
- IAM
- Infrastructure as Code
- Kafka
- Kinesis
- Lakehouse
- Lambda
- Machine Learning
- PySpark
- Python
- Redshift
- S3
- Spark
- SQL
- Terraform
- Unity
- Unity Catalog
- Workflow Orchestration