Data Engineer
Summary
Builds and maintains scalable data warehouses and pipelines for Shopee’s ecommerce analytics, using SQL, Spark, and Python/Scala to turn raw data into trusted business insights.
- Design and develop data marts and data warehouses to support business intelligence and data analytics initiatives within the project lifecycle.
- Handle end-to-end data pipelines from multiple internal and external sources, ensuring high performance, scalability, and establishing a Single Source of Truth for project deliverables.
- Collaborate closely with Data PMs, Business Intelligence Analysts, and engineering team members to transform complex data into actionable business insights within designated project timelines.
- Implement and maintain data management best practices across all project pipelines, focusing on data quality, security, and privacy.
- Execute software development best practices, including test-driven development and code documentation, to ensure robust and maintainable data infrastructure.
- Design and develop data marts and data warehouses to support business intelligence and data analytics initiatives within the project lifecycle.
- Handle end-to-end data pipelines from multiple internal and external sources, ensuring high performance, scalability, and establishing a Single Source of Truth for project deliverables.
- Collaborate closely with Data PMs, Business Intelligence Analysts, and engineering team members to transform complex data into actionable business insights within designated project timelines.
- Implement and maintain data management best practices across all project pipelines, focusing on data quality, security, and privacy.
- Execute software development best practices, including test-driven development and code documentation, to ensure robust and maintainable data infrastructure.
- Hold a Master\'s or Bachelor\'s degree in Computer Science, Information Systems, or other quantitative-based majors (Science or Engineering)
- Possess 2 to 4 years of hands-on working experience in data engineering, data pipeline development, or data architecture.
- Demonstrated expert-level proficiency in SQL, Spark, HDFS, and big data processing frameworks.
- Expert knowledge of Object-Oriented and Functional Programming, with a strong preference for Python or Scala.
- Solid experience working in UNIX environments, with deep understanding of batch processing, lambda architecture, and Kimball data modeling concepts.
- Proficient in Software Development Life Cycle (SDLC) practices, including version control, test-driven development, and technical documentation.