Data Engineer
Posted Updated
Summary
Data engineer in Singapore who designs, builds and maintains large-scale batch and real-time data pipelines, deploys data lake/warehouse tables and develops backend APIs. Core stack: Spark with Scala/PySpark, Hadoop, and Azure big-data tooling (Data Factory, Synapse), plus a technical-lead role over data engineers.
Design, build and maintain efficient and reliable large-scale batch and real-time data pipelines
Utilize data processing/engineering frameworks for data ingestion, cleaning, transformation to ensure the data pipelines are efficient, reliable and scalable
Design, develop and deploy data tables in data lake / data warehouses.
Design and develop backend APIs and database integrations to support applications/use cases.
Prerequisites
- Degree or Diploma in Computer Science, Computer Engineering, Information Technology, Business Analytics or related disciplines.
- Minimum 2 - 8 years' experience in designing, developing and/or maintaining data platforms.
- Played the role of a technical lead to a team of data engineers.
- Spark, Scala/Pyspark
- Big data frameworks/tools (e.g. Hadoop, Azure Data Factory, Azure Synapse)
- Data platform and data structure/model design and management (e.g. Data Mart, DataLake, Data Virtualization and Data Warehousing)
- Experience with building production-grade data pipelines, ETL/ELT data integration in a big data environment using Cloud platforms such as Azure.
- Experience working with Agile teams with strong collaboration skills.
- Knowledge/experience on data governance policies, access control and security best practices.
- Stakeholders' engagement/management and requirements analysis skills.