Remote LLM Benchmark Scientist - Evaluations & Validation
Posted Updated
Description
Anyone AI Labs is seeking a Research Scientist to own evaluation design for frontier models. You’ll design benchmarks across reasoning, coding, agents, tool-use and multi‑modal tasks, grounded in expert-verified truth and validated against multiple models.
You’ll QC results to survive buyer-side review and push for durable, informative evals as models improve. You’ll lead a team to recruit experts across coding and STEM, translate lab goals into robust evaluation pipelines, and publish public
Evaluation research. Turn eval targets into original benchmark designs. Own the hard measurement questions: construct validity, item discrimination, headroom, reliability, contamination, and capability elicitation. Push toward evals that stay informative as models improve. Benchmark development. Build evaluation packages with subject‑matter experts, each with expert‑verified ground truth, multi‑model headroom results, and rigorous QC (calibration layers, severity‑weighted rubrics, deterministic verifiers). Experts. Recruit, calibrate, and review a pool across coding, agentic/tool‑use, and STEM/reasoning. Be the final arbiter of correctness and frontier difficulty. Lab relationships. Be a technical point of contact for labs, with CEO support. Understand what they're trying to measure and translate it into an evaluation design. Delivery & dissemination. Turn lab requests into winning sample packages and own pilots end to end. Where the work generalizes, help turn it into public benchmarks and papers: we support publishing at venues like NeurIPS Datasets & Benchmarks, ICLR, and ACL. Research background in ML evaluation or benchmarking (a track record of published or open benchmarks, eval/measurement research, or equivalent hands-on work that labs have relied on). Deep LLM/frontier-model benchmarking expertise, with real strength in code-model and agentic evaluation. Fluency with the measurement problem itself: construct validity, psychometrics, rubrics, pass rates, headroom, contamination, and what makes a task genuinely discriminate a model. Interest in the safety side of evaluation, capability elicitation, robustness, and measuring the things that are hardest to measure honestly. Proven ability to hold a team or expert pool to a rigorous standard. Comfort with the full research loop: framing the question, running the study, and writing it up. Fluent English; Spanish a plus.Skills
Sign in to read and join the discussion.