Senior Python AI Evaluation Engineer — Production Quality
Posted Updated
Description
EPAM Systems is seeking a Python AI Evaluation Engineer to join the Enterprise Agent Development Platform. You will own the capabilities that measure and validate AI solutions, designing evaluation approaches, developing custom evaluators, and embedding quality checks into the software delivery lifecycle.
This role requires 5+ years in ML/AI engineering, strong Python skills, and hands-on experience with LLM/GenAI evaluation.
Design and implement evaluation frameworks for LLM-based applications and AI agents Develop LLM-as-a-Judge and deterministic, code-based evaluators Build custom Python evaluators for quality and behavioral checks Define evaluation criteria, metrics, thresholds, and acceptance rules Evaluate agent behavior across individual responses, tool calls, and complete workflows Work with OpenTelemetry traces and spans as evaluation data Integrate evaluations into CI/CD pipelines and automated deployment gates Enable continuous quality monitoring of solutions in production Establish reusable evaluation patterns and engineering standards Work closely with AI Engineers, Architects, and Platform Engineers to embed quality into the development process 5+ years of experience in ML/AI engineering or AI Platform Engineering. Strong Python development experience. Hands-on experience with LLM/GenAI evaluation. Experience designing and implementing evaluation frameworks. Experience developing custom or deterministic evaluators. Experience integrating AI/ML quality checks into CI/CD. Good understanding of LLM and AI agent architectures. Continuous upskilling Career development Private health insurance English courses Mentoring programs Learning platformsSkills
Sign in to read and join the discussion.