Manager QA Evaluation / AI Evaluation Engineer

We are currently hiring for a Manager QA Evaluation / AI Evaluation Engineer for a leading technology organization building advanced AI agents and enterprise-grade AI platforms.

The role offers an opportunity to build the AI evaluation function from the ground up, defining how AI agents are tested, measured, and validated for reliability, safety, and production readiness.

Key highlights of the role:

• Architect evaluation frameworks for LLMs and multi-step AI agents.
• Define acceptance criteria, quality benchmarks, and release gates for AI systems.
• Evaluate agent reasoning, tool orchestration, behavioral consistency, and failure patterns.
• Establish production monitoring, model regression testing, drift detection, and AI safety evaluation.
• Influence technical architecture and AI quality standards across engineering teams.

What were looking for:

• 10+ years of software engineering experience, including 4+ years of senior individual-contributor leadership in AI evaluation, measurement infrastructure, or AI quality systems.
• Strong hands-on experience with LLM evaluation, agentic AI, evaluation frameworks, and production AI systems.
• Deep understanding of distributed systems, statistical evaluation, and production reliability.
• Experience defining technical standards and influencing architecture across teams.

Role level: Principal / Lead Individual Contributor
Opportunity type: Full-time

Location: Bangalore (Hybrid)

Best regards,
Amreesh Yadav