Senior Staff Software Engineer, ML Training Infrastructure
Posted Updated
Description
You will drive the design and development of a high-performance multi-tenant AI training platform, balancing hands-on coding with long-term technical direction. You will work across ML Platform, Infrastructure, Autonomy and Safety Evaluation teams, deliver end-to-end ML pipelines, resolve performance bottlenecks, optimize distributed training throughput, and build tooling that helps ML engineers iterate quickly.
Responsibilities
- Design and evolve training platform components including orchestration, scheduling, training abstractions, control plane, observability and performance tuning
- Deliver end-to-end ML model pipelines across logs processing, feature extraction, dataset schema design and storage, model configuration management, training, and profiling and acceleration
- Instrument and analyze training infrastructure performance to resolve bottlenecks
- Optimize training throughput across distributed nodes using topology-aware frameworks and hardware such as InfiniBand, NVLink and NCCL
- Evangelize system abstractions and tooling that enable MLEs to iterate rapidly
- Incorporate OSS tools so ML engineers can profile and optimize their workflows
- Promote engineering excellence within the team
Requirements
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field
- 8+ years of experience with ML platforms and building ML-based applications
- Proven track record optimizing training with distributed training paradigms
- Experience productionizing large-scale data pipelines and data loaders
- Strong Python skills and experience with Lance, PyTorch, Ray, Spark or equivalent
- Proven track record building scalable, reliable infrastructure working with MLEs
- Deep understanding of design tradeoffs and ability to build alignment across cross-functional teams
- Strong analytical and problem-solving skills
- Excellent verbal and written communication skills
Skills
Sign in to read and join the discussion.