Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Black Forest Labs

NewBe an early applicant

Research Lead, Evaluations

Posted

Description

You will drive the evaluation roadmap together with research and product teams, define what and how the lab evaluates, and build benchmarks and objective functions using automated, AI-based and human evaluations. You will minimize noise in the eval pipeline, communicate the limitations of each evaluation, monitor critical training runs independently, and manage and grow the evals team including hiring.

Responsibilities

  • Create, maintain and extend the evaluations roadmap with product and research leads
  • Implement the evals roadmap and build new evals, benchmarks and objective functions using automated, AI-based and human evaluations
  • Minimize noise in the eval pipeline and ensure reproducible, trustworthy results with understood error bars
  • Communicate limitations of evaluations and ensure evaluation sets have no blind spots
  • Monitor training runs and visualize important findings to catch regressions and emerging capabilities early
  • Manage the evals team, define the hiring roadmap and support team members' growth

Requirements

  • Experience designing, running and maintaining large evaluation suites for frontier AI systems, with human and automated methods
  • Proven experience designing and implementing evaluations for AI research
  • Strong Python and agentic coding understanding
  • Actively managed a team for at least one year
  • Strong understanding of statistics: variance, confidence intervals, significance testing
  • Excellent communication skills
  • First principles thinking
  • Experience in visual multimodal AI systems is a plus

Benefits

  • Equity

Skills

Apply

See also