Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

EPAM Systems Inc

Senior Python AI Evaluation Engineer — Production Quality

Posted Updated

Description

EPAM Systems is seeking a Python AI Evaluation Engineer to join the Enterprise Agent Development Platform. You will own the capabilities that measure and validate AI solutions, designing evaluation approaches, developing custom evaluators, and embedding quality checks into the software delivery lifecycle.

This role requires 5+ years in ML/AI engineering, strong Python skills, and hands-on experience with LLM/GenAI evaluation.

Design and implement evaluation frameworks for LLM-based applications and AI agents Develop LLM-as-a-Judge and deterministic, code-based evaluators Build custom Python evaluators for quality and behavioral checks Define evaluation criteria, metrics, thresholds, and acceptance rules Evaluate agent behavior across individual responses, tool calls, and complete workflows Work with OpenTelemetry traces and spans as evaluation data Integrate evaluations into CI/CD pipelines and automated deployment gates Enable continuous quality monitoring of solutions in production Establish reusable evaluation patterns and engineering standards Work closely with AI Engineers, Architects, and Platform Engineers to embed quality into the development process 5+ years of experience in ML/AI engineering or AI Platform Engineering. Strong Python development experience. Hands-on experience with LLM/GenAI evaluation. Experience designing and implementing evaluation frameworks. Experience developing custom or deterministic evaluators. Experience integrating AI/ML quality checks into CI/CD. Good understanding of LLM and AI agent architectures. Continuous upskilling Career development Private health insurance English courses Mentoring programs Learning platforms

Skills

Apply

See also