AI Agent Evaluation Engineer
Posted Updated
Description
You will be the first owner of quality for Cresta's AI Receptionist agents. You will define evaluation rubrics, annotate real conversations, build failure taxonomies, and build automated evaluation pipelines using LLM-based judges, golden sets and regression suites. You will calibrate judges against human labels, design sampling strategies, cover edge cases, and report quality trends with clarity while partnering with engineering and product teams.
Responsibilities
- Define and maintain the evaluation rubric across adherence, outcome and conversation experience
- Write annotation guidelines and failure taxonomy; run calibration sessions and measure inter-rater agreement
- Classify and prioritize failure modes by severity and frequency
- Review production conversations weekly and cluster failures
- Trace failures to likely causes and route them to the right owners
- Evaluate quality across languages and segments
- Build and operate the evaluation pipeline combining automated checks, LLM judges and human review
- Calibrate automated judges against human labels
- Design stratified sampling strategies
- Build and maintain golden set and regression suite
- Maintain the failure-mode coverage map and design evaluations for edge cases
- Define quality and latency thresholds
- Report quality trends to stakeholders and customers
- Partner with forward-deployed engineers, backend engineers, product managers and voice/language engineers
- Document methodology
Requirements
- 3+ years in software engineering, applied ML or data-intensive research, including 1-2 years with LLM systems in production
- Bachelor's or Master's degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics or related field, or equivalent experience
- Built and operated an end-to-end evaluation system for a production LLM product
- Deep understanding of evaluation methodology
- Fluent in how LLM agents work
- Strong Python skills and experience with LLM APIs
- Able to write rubrics that independent reviewers score consistently
- Able to design tests from open-ended quality questions
- Willing to do non-scalable work weekly such as listening to calls and labeling
- Professional fluency in at least one language other than English
- Strong written communication
Benefits
- Comprehensive medical, dental, and vision coverage
- Flexible PTO
- Paid parental leave
- Retirement savings plan
- Remote work setup budget
- Monthly wellness and communication stipend
- In-office meal program and commuter benefits for onsite employees
- Equity
Skills
Sign in to read and join the discussion.