Data Engineer
Posted Updated
Summary
Design, build and maintain large-scale batch and real-time data pipelines, data lake/warehouse tables and backend APIs for applications. Core stack is Spark with Scala/PySpark plus big-data tools like Hadoop, Azure Data Factory and Azure Synapse on Azure cloud; role includes technical-lead duties for a team of data engineers.
Design, build and maintain efficient and reliable large-scale batch and real-time data pipelines
Utilize data processing/engineering frameworks for data ingestion, cleaning, transformation to ensure the data pipelines are efficient, reliable and scalable
Design, develop and deploy data tables in data lake / data warehouses.
Design and develop backend APIs and database integrations to support applications/use cases.
Prerequisites:
Degree or Diploma in Computer Science, Computer Engineering, Information Technology, Business Analytics or related disciplines.
Minimum 2 - 8 years' experience in designing, developing and/or maintaining data platforms.
Played the role of a technical lead to a team of data engineers.
Spark, Scala/Pyspark
Big data frameworks/tools (e.g. Hadoop, Azure Data Factory, Azure Synapse)
Data platform and data structure/model design and management (e.g. Data Mart, DataLake, Data Virtualization and Data Warehousing)
Experience with building production-grade data pipelines, ETL/ELT data integration in a big data environment using Cloud platforms such as Azure.
Experience working with Agile teams with strong collaboration skills.
Knowledge/experience on data governance policies, access control and security best practices.
Stakeholders' engagement/management and requirements analysis skills.