Data Engineer (Hybrid, BGC)

Overall Duties and Responsibilities

Does (The tasks / responsibilities that the role performs to address requirements in Key Result Areas).

1. Data Warehousing Design

• Design data warehouse and operational data marts following the Client standards.

• Prepare data model, data dictionary, and mapping documents

2. Data Pipeline design, development, testing, and support

• Design and develop data pipelines that will ingest the data from the respective source systems into the data warehouse

• Support and optimize existing data pipelines to ensure that the downstream SLAs are met

• Ensures the data engineering standards and practices are followed like code version history, readability, functionality, extendibility, test runs, deployments, and post deployment support

• Manage data storage, computing unit requirements, and data security within the data warehouse

3. Comply with Client's agile ways of working

• Responsible in attending agile ceremonies as part of the Agile Tribe set-up and delivering the different demands of the Tribe related to data ingestions to the data warehouse platform

• Gather data requirements coming from Tribe demands

4. Maintain high data quality in the data warehouse

• Ensures data quality within the data warehouse tables by implementing data validation rules and data quality checks within the data pipeline

5. Incident Management Support

• Supports L2 data operations team during production incidents by troubleshooting data pipelines and understanding data behaviors

• Supports stakeholders of the data warehouse platform by solving data related issues and conducting data investigations

• Displays (The knowledge, skills and behaviors indicating how tasks / responsibilities will be performed).

• Good understanding of data warehousing concepts and data modeling

• Proficient in SQL and optimizing queries

• Good communication and problem solving skills

• Delivers (The specific outputs / tangible results produced by the role; resources responsible for)

• Data Delivery Documentations – Data Dictionary, Data Mapping Documents, Data Models

• Optimized data pipelines

• Root cause analysis for production related issues and incidents

Qualifications (Minimum requirements in terms of experience and background, skills and competencies for sourcing purposes)

• Graduate of any IT related courses like BS Computer Science, Information Technology or the likes

• 4-6 years of experience in IT project delivery or data development delivery particularly in Data Warehousing or Business Intelligence

• Proficient in ETL development related to Data Warehousing, Business Intelligence, or other Data Delivery projects.

• Proficient in optimizing data pipelines in order to meet SLA requirements and save cost

• Proficient in SQL, Python, R, Java, Scala

• Proficient in Agile delivery methodologies like Scrum or Kanban

• Proficient in data modeling, data integrations, data quality, and data architecture is a plus

• Competent in Big Data, Hadoop, EMR, Hive, and cloud environments is a plus

• Competent in using Alibaba Cloud products particularly Dataworks, Maxcompute, OSS, Hologres Competent in Kafka, Confluent, Real-time processing, Data Streaming is a plus

• Proficient in using AWS Cloud products particularly S3, Glue, Athena, Data Sync, Lake Formation, Airflow, Secrets Manager, IAM, KMS, Cloudwatch

• Competent in using Terraform, Gitlab