Data Engineer
Posted Updated
Summary
Builds and maintains large-scale cloud data pipelines and distributed data processing in Singapore using AWS, SQL, PySpark, Apache Iceberg, dbt and Apache Airflow. Also designs thorough tests, coaches teammates on coding practices, and manages stakeholders across multiple workstreams.
Must-have technical skills
AWS, SQL, PySpark, Apache Iceberg, dbt and Apache Airflow.
Key Responsibilities
. Build and maintain data pipelines using cloud technologies and distributed systems.
. Work with data transformation frameworks and pipeline orchestration tools.
. Manage large volumes of data and optimise data processing.
. Design and implement effective tests, including unit, component, integration, end-to-end and performance tests.
. Coach team members on coding practices, design principles and implementation patterns that produce high-quality, maintainable solutions.
. Manage stakeholders and prioritise work effectively across multiple workstreams.
Requirements
. Formal training or certification in data engineering concepts, together with applied experience.
. Recent hands-on professional experience as a data engineer.
. Experience coding in a recent version of Python.
. Hands-on experience with AWS, SQL, PySpark, Apache Iceberg, dbt and Apache Airflow.
. Experience with cloud technologies and distributed systems.
. Experience with data transformation frameworks and data pipeline orchestration tools.
. Experience managing large volumes of data and optimising data processing.
. Understanding of event-based architecture, data streaming and messaging frameworks.
. Experience designing and implementing effective software tests.