Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Ramp

NewBe an early applicant

Research Engineer (Benchmarks & Evaluation)

Posted Updated

Description

  • Every model decision at Ramp Labs runs through a benchmark. Which model to route to
  • Whether a post-training run worked. Whether an agent is ready to post a corrective journal entry to a customer’s ledger without a human in the loop
  • You’ll own Ramp’s benchmarking effort. You’ll turn real finance work (expense policy review, invoice coding, month-end close, spreadsheet Q&A) into benchmarks and RL environments, build the graders, and tell the team what is actually improving
  • Build benchmarks from real workflows: turn agent traces, customer edge cases and expert labels into versioned benchmark suites for long-horizon agents that run for hours across tools and systems, plus tool use, multi-step reasoning and financial judgment
  • Own the benchmarking effort: decide what we measure, run benchmark suites across models and agent versions, and report results the team makes decisions on
  • Design graders: rubrics, programmatic checks and llm-as-judge, calibrated against human agreement, with a clear view of where each one breaks
  • Close the loop with post-training: turn evals into reward functions and RL environments, and catch reward hacking and contamination before they ship
  • Publish external benchmarks and write-ups on how models perform at real finance work

Benefits

  • Comprehensive medical, dental, and vision insurance
  • One Medical Membership
  • 401(k) including an employer match
  • Unlimited PTO
  • Parental leave
  • Monthly wellness stipend
  • Pet Insurance
  • WFH stipend
  • Relocation support to move to NYC
  • Working knowledge of post-training (SFT, RL with verifiable rewards) and how evals feed into it
  • Comfort with SQL and large datasets; you’ll work directly from production traces
  • Strong Python and software engineering; your eval code is reproducible and fast
  • Hands-on experience building LLM benchmarks or evals that people used to make real decisions
  • Taste for what is worth measuring
  • Statistical judgment: variance, confidence intervals, sample size, inter-rater agreement, and knowing when a delta is noise
  • Industry experience on an eval, benchmarking or post-training team
  • Built RL environments or verifiers for agentic tasks
  • Published benchmark or evaluation research (NeurIPS Datasets & Benchmarks, ICML, ACL)
  • Built or contributed to eval frameworks (verifiers, lm-evaluation-harness, HELM or similar)
  • Familiarity with accounting, finance or other expert-domain work
  • Public work we can look at: a benchmark, an eval suite, a failure-analysis write-up

Skills

Apply

See also