Lead Data Engineer
Summary
Lead a scalable data platform using Airflow, Spark, Kafka, and Kubernetes on AWS/GCP to ingest, process, and serve real-time and batch data for a global fashion ecommerce group.
Mô tả công việc: JOB DESCRIPTION
Quyền lợi ứng viên: Thưởng Tháng 13, Tiệc liên hoan, Văn Phòng Tiện Nghi
Yêu cầu hồ sơ:
Từ khóa việc làm: CNTT khác, IT Support
Kỹ năng cần có: Data Analyst, IT phần mềm, Quy Trình Kỹ Thuật
Kinh nghiệm làm việc: Không yêu cầu kinh nghiệm
Giới tính: Không yêu cầu
- Build and manage the data asset using some of the most scalable and resilient open source big data technologies like Airflow, Spark, Kafka, etc
- Build and manage a highly scalable, efficient Data and ML Infrastructure by adopting microservices driven design and architecture with proper DevOps principles and practices
- Design and deliver the next-gen data lifecycle management suite of tools/frameworks, including ingestion and consumption on the top of the data lake to support real-time as well as batch use cases
- Help the team in integrating various data sources across GFGs group vertical
- Build and expose metadata catalog for the Data Lake for easy exploration, profiling as well as lineage requirements
- At least 3+ years of relevant experience in developing scalable, secured, fault tolerant, resilient mission-critical Big Data platform.
- Able to maintain and monitor the ecosystem with high availability.
- Must have sound understanding for all Big Data components Administration Fundamentals. Hands-on in building a complete data platform using various open source technologies.
- Must have good fundamental hands-on knowledge of Linux and building big data stack on top of AWS/GCP using Kubernetes.
- Strong understanding of big data and related technologies like HDFS, Spark, Presto, Airflow, Kafka, Apache Atlas etc.
- Good knowledge of Complex Event Processing systems like Spark Streaming, Kafka, Apache Flink, Beam etc.
- Able to drive devops best practices like CI/CD, containerization, blue-green deployments, secrets management etc in the Data ecosystem.
- Able to develop an agile platform with auto scale capability up down as well vertically and horizontally.
- Able to develop an observability and monitoring ecosystem for all the components in use in the data ecosystem.
- Proficiency in at least one of the programming languages Java, Scala, Python or Go.
- Annual Leave: 15 days
- Sick leave/Mental Health Days: 30 days.
- Occasion leave: 1 day.
- 13th month salary.
- Annual bonus: up to 3 months based salary (depends on company performance).
- Home Workspace support: up to 7mil/person.
- Learning budget up to EUR 500/person.
- Medical benefits: Social insurance, medical insurance AON insurance.
- Macbook is provided.
- Support for English class.
- Support Gym membership.
- Additionally, we offer a flat hierarchy, a fun and flexible work environment.
Quyền lợi ứng viên: Thưởng Tháng 13, Tiệc liên hoan, Văn Phòng Tiện Nghi
Yêu cầu hồ sơ:
Từ khóa việc làm: CNTT khác, IT Support
Kỹ năng cần có: Data Analyst, IT phần mềm, Quy Trình Kỹ Thuật
Kinh nghiệm làm việc: Không yêu cầu kinh nghiệm
Giới tính: Không yêu cầu