Software Engineer, Training Infrastructure
Posted Updated
Description
You will build the distributed training stack that runs supervised and reinforcement learning jobs reliably across large GPU clusters. You'll work on scheduling, orchestration, checkpointing and recovery, optimize throughput and cost, bring up new model architectures including mixture-of-experts, build reliability and observability, support multi-tenant LoRA fine-tuning, and collaborate with API and product teams.
Responsibilities
- Design, build, and operate distributed training infrastructure spanning scheduling, orchestration, checkpointing, and recovery
- Improve training throughput and cost efficiency through profiling, parallelism, memory optimization, and kernel- and communication-level tuning
- Bring up and optimize training support for new model architectures including mixture-of-experts
- Build reliability: failure detection, job recovery, multi-tenant isolation, and observability
- Support multi-tenant LoRA fine-tuning and coupled sampling and training for reinforcement learning
- Collaborate with Tinker API, frameworks, and product teams
Requirements
- 4+ years of software engineering including building and operating production distributed systems
- Strong Python plus experience in C++, Rust, or Go
- Hands-on experience with large-scale model training or inference infrastructure using frameworks such as PyTorch
- Track record of debugging complex performance and reliability issues in distributed environments
Benefits
- Health, dental, and vision benefits
- Unlimited PTO
- Paid parental leave
- Relocation support
- Visa sponsorship
Skills
Sign in to read and join the discussion.