Member of Technical Staff - Pre-Training Evals Infra
Summary
Builds evaluation infrastructure for Reflection's pre-training research: designing eval APIs and abstractions, running distributed checkpoint evaluations, optimizing inference and GPU throughput, and shipping researcher-facing dashboards for inspecting and debugging model results. Core stack: Python, C++, or Rust plus distributed systems.
You will build evaluation infrastructure for researchers to understand model capabilities, guide training decisions, and assess release readiness. You will design evaluation abstractions, develop distributed execution systems, optimize inference performance, ensure correctness and reproducibility, and create tools for inspecting and debugging results. You will monitor evaluation workloads, investigate failures, and turn research needs into reusable self-service systems.
Responsibilities
- Design and evolve evaluation abstractions, APIs, and data models.
- Build scalable and reliable distributed infrastructure for evaluating model checkpoints.
- Optimize inference and distributed-execution performance, throughput, GPU utilization, and time to results.
- Establish evaluation correctness and reproducibility through validation, regression testing, versioning, and provenance.
- Build researcher-facing interfaces and dashboards for model-output inspection, experiment comparison, and debugging.
- Monitor evaluation workloads and results, triage failures, and develop alerts and diagnostics.
- Partner with researchers to turn bespoke workflows into reusable self-service systems.
Requirements
- Proficiency in Python, C++, or Rust.
- Ability to reason about concurrency, memory behavior, and performance.
- Experience designing, building, and operating reliable distributed systems.
- Knowledge of model inference and evaluation.
- Ability to investigate complex problems across system boundaries and identify root causes.
- Experience translating technical-user needs into APIs and usable tools.
- Ability to own ambiguous problems, prioritize competing needs, and balance delivery with reliability and maintainability.
- Attention to detail when investigating discrepancies in model outputs, evaluation results, and system behavior.
Benefits
- Stock options.
- Comprehensive medical, dental, vision, and life insurance.
- Annual wellness allowance.
- Daily in-office lunch and dinner.
- 22 weeks of paid parental leave.
- Unlimited paid time off in the U.S.
- 30 days of vacation in the U.K.
- Visa sponsorship support.
- Regular off-sites, happy hours, and team celebrations.