Hiring | AI Engineer – Agent Evaluation Specialist | C2C/W2 | 14+ Years Experience | Remote
Summary
Contract role (3+ months, possible extension) for a senior AI Engineer focused on evaluating LLM-powered agents rather than building them: LLM-as-a-Judge frameworks, golden dataset development, and agent observability/tracing with Langfuse. Needs 14+ years experience; hybrid in Los Angeles with remote flag; specialization in AWS Bedrock AgentCore and Claude Agent SDK.
Role: AI Engineer – Agent Evaluation Specialist
Location: Los Angeles, CA (Hybrid)
Duration: 3+ Months (Possible Extension)
Primary Focus: Hands-on Agent Evaluation
Specialisation: Agentic AI, AWS Bedrock AgentCore, Claude Agent SDK, Langfuse, AI Observability & Evaluation
We are looking for an AI Engineer with strong, practical experience in evaluating LLM-powered agents, rather than someone focused primarily on building or developing agents.
Key Requirements (Must-Have):
- LLM-as-a-Judge: Hands-on experience designing, implementing, and running LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability.
- Golden Dataset Development: Experience collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets that serve as benchmarks for agent evaluation.
- Langfuse: Practical experience instrumenting AI agents using Langfuse, including tracing agent execution, capturing evaluation scores, monitoring performance, and troubleshooting failures.
- Agent Evaluation: Strong understanding of end-to-end agent evaluation, including testing agent behavior, tool usage, reasoning workflows, and response quality.
Critical Screening Criteria:
Candidates must demonstrate actual implementation experience with agent evaluation frameworks, not just theoretical knowledge or general GenAI/LLM development.
Agent evaluation is the core requirement and will be the primary selection criterion.
All your information will be kept confidential according to EEO guidelines.