Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Thinking Machines Lab

NewBe an early applicant

Software Engineer, Training Infrastructure

Posted Updated

Description

You will build the distributed training stack that runs supervised and reinforcement learning jobs reliably across large GPU clusters. You'll work on scheduling, orchestration, checkpointing and recovery, optimize throughput and cost, bring up new model architectures including mixture-of-experts, build reliability and observability, support multi-tenant LoRA fine-tuning, and collaborate with API and product teams.

Responsibilities

  • Design, build, and operate distributed training infrastructure spanning scheduling, orchestration, checkpointing, and recovery
  • Improve training throughput and cost efficiency through profiling, parallelism, memory optimization, and kernel- and communication-level tuning
  • Bring up and optimize training support for new model architectures including mixture-of-experts
  • Build reliability: failure detection, job recovery, multi-tenant isolation, and observability
  • Support multi-tenant LoRA fine-tuning and coupled sampling and training for reinforcement learning
  • Collaborate with Tinker API, frameworks, and product teams

Requirements

  • 4+ years of software engineering including building and operating production distributed systems
  • Strong Python plus experience in C++, Rust, or Go
  • Hands-on experience with large-scale model training or inference infrastructure using frameworks such as PyTorch
  • Track record of debugging complex performance and reliability issues in distributed environments

Benefits

  • Health, dental, and vision benefits
  • Unlimited PTO
  • Paid parental leave
  • Relocation support
  • Visa sponsorship

Skills

Apply

See also

Software Engineering jobs by country — openings, pay and top skills →