Machine Learning Engineer - Distributed Systems

【About the Role】
As a Machine Learning Engineer specializing in Distributed Systems, you will design and build the infrastructure that allows our machine learning workloads to scale from single process, single-GPU, and single-node to multi-process, multi-GPU, and multi-node environments.
You will develop reusable distributed systems and execution frameworks that support multiple products and workloads. Your work will help teams scale data processing, model training, inference, and other compute-intensive or data-intensive workflows while maintaining correctness, reliability, and efficient resource utilization.
You will work closely with machine learning engineers, optimization engineers, data scientists, and infrastructure teams to understand workload requirements and build practical distributed solutions for production environments.

【Responsibilities】

• Design, build, deploy, and maintain reusable distributed systems that scale workloads across multiple processes, GPUs, nodes, and accelerators, including the communication, synchronization, scheduling, and resource-management mechanisms required to support them.
• Investigate and resolve issues involving distributed execution, race conditions, resource contention, scalability, and system reliability.
• Collaborate with optimization engineers and other cross-functional teams to integrate component-level performance improvements into scalable end-to-end systems.
• Validate distributed systems through testing, benchmarking, failure simulation, and clear technical documentation while promoting clean and maintainable engineering practices.

【Key Qualifications】

• Strong programming and software engineering fundamentals, with proficiency in Python, C++, or other relevant programming languages.
• Experience designing and developing concurrent or distributed systems across multiple processes, accelerators, or machines, with an understanding of workload orchestration, resource allocation, and distributed job management.
• Experience developing distributed machine learning or high-performance computing workloads with Git-based development workflows and CI/CD practices.
• Ability to design reusable systems and abstractions that can support multiple products, jobs, and workload patterns.
• Strong analytical, problem-solving, communication, and collaboration skills, with the ability to diagnose complex behavior across multiple system components.

【Preferred, but Not Required】

• Experience scaling model inference, training, data processing, or other compute-intensive and data-intensive workloads across multiple GPUs and nodes.
• Experience measuring and analyzing distributed-system performance, including communication overhead, synchronization delays, resource utilization, throughput, and scaling efficiency.
• Experience designing fault-tolerant systems with recovery, health monitoring, and graceful handling of partial failures.
• Familiarity with GPU profiling, accelerator optimization, and performance-analysis tools, with an understanding of how component-level optimizations affect distributed execution.
• Understanding of the machine learning lifecycle and how distributed execution supports model training, inference, prediction services, and production operation.
• Familiarity with Kubernetes, cluster schedulers, and container technologies such as Docker.

You do not need to match every preferred qualification. We value strong systems fundamentals, thoughtful engineering, and the ability to transform individual workloads into reliable and reusable distributed platforms. We are hiring for both mid-level and senior-level engineering positions, and the scope and expectations of the role will be considered based on your experience and seniority during the interview process.