AI Data Infrastructure Engineer
Posted Updated
Summary
Builds and operates large-model data-processing infrastructure at MiniMax using Spark and Ray — developing data lakes and catalog services (Paimon, Iceberg, Lance), managing clusters and task scheduling, and improving monitoring, performance, and cost across Kubernetes-based production data workflows.
You will build and operate large-model data-processing infrastructure using Spark and Ray. You will develop data lakes and catalog services, and improve monitoring, logging, tracing, alerting, resource utilization, and costs across production data workflows.
Responsibilities
- Build large-model data-processing infrastructure using Spark and Ray
- Manage clusters, schedule tasks, isolate resources, and analyze performance
- Develop data lakes and catalog services using Paimon, Iceberg, and Lance
- Build monitoring, logging, tracing, and alerting for compute and data-lake workflows
- Investigate performance, stability, and capacity issues with Kubernetes and SRE
- Optimize resource utilization and costs
Requirements
- Proficiency in at least one of Python, Java, or Scala
- Knowledge of Linux, containers, and distributed systems
- Experience with large-scale data processing or cloud-native platforms
- Ability to develop, deploy, debug, and maintain long-running production tasks
- Ability to analyze performance, stability, and capacity issues using metrics, logs, traces, and runtime evidence
- Experience in at least one of Spark or Ray platform development, task scheduling, performance analysis, data-lake engines, or large-scale data organization and metadata services
- Experience using AI to improve delivery efficiency
