Data Engineer
Posted
Description
Job Summary:
As the Group is working towards massive technology transformation, the Data Engineer role expected to able to plan, create and maintain the foundation of database architecture. He/she required to apply relevant database techniques to create robust data architecture for future use of data mining, design, build, and maintain scalable, secure, and well-governed data pipelines on our on-premise data lakehouse and support future data analytical requirement.
Job Description
Collecting, managing and transforming raw data into Information that can be interpreted.
- Design or participate on the data model design.
- Understand business requirement and translate to design document from data ingestions to data modelling.
- Identify hidden patters from stored data.
- Transforming legacy data to robust and informative data.
- Demonstrating a strong work ethic, a “can do” attitude and being highly productive.
- Learning new technologies and developing tools to remain abreast of current technology, while leveraging your skills in legacy technologies to support existing platform development activities.
Technical Problems / Issues Mangement
- Must target to work towards resolving technical problems/ issues.
- Develop agile way of documentations and improve current processes.
Job Requirements
- Bachelor’s Degree Computer Science related field or equivalent experienced required.
- At least 3 years' experience in Data Engineering (e.g. Data modelling, data pipeline, data architecture, ETL)
- Programming & Ingestion: Strong proficiency in Python and SQL, with proven experience ingesting data from relational databases (SQL Server, PostgreSQL), REST APIs, and file-based sources (CSV, JSON, Parquet, etc.).
- Distributed Processing & Resource Management: Production experience with Apache Spark (PySpark, SparkSQL) and Apache Hive running on the Apache Tez execution engine (not MapReduce), with Apache Hadoop YARN as the resource manager (job submission, queue/capacity management, container & memory tuning).
- Storage & Table Formats: Practical knowledge of Hadoop (HDFS) and Delta Lake (ACID transactions, time travel, schema enforcement/evolution, file compaction).
- Orchestration & Metastore: Hands-on experience building and operating workflows in Apache Airflow and managing schemas/partitions in Hive Metastore.
- Engineering Practices: Experience with Git, CI/CD, unit/integration testing for data pipelines, and Linux/shell scripting.
Skills
Sign in to read and join the discussion.