Senior ML / Evaluation Engineer
Posted Updated
Description
EPAM Systems, Inc. is seeking a Python AI Evaluation Engineer to design evaluation frameworks for LLM-based apps and agents. You will build deterministic evaluators and Python-based tools to measure quality across responses, tool calls, and complete workflows.
You will integrate these checks into CI/CD pipelines, work with OpenTelemetry traces, and collaborate with AI Engineers and Platform Engineers to embed quality into development processes. AWS Bedrock Guardrails experience is a plus.
Design and implement evaluation frameworks for LLM-based applications and AI agents. Develop LLM-as-a-Judge and deterministic, code-based evaluators. Build custom Python evaluators for quality and behavioral checks. Define evaluation criteria, metrics, thresholds, and acceptance rules. Evaluate agent behavior across responses, tool calls, and complete workflows. Work with OpenTelemetry traces and spans as evaluation data. Integrate evaluations into CI/CD pipelines and automated deployment gates. Enable continuous quality monitoring of solutions in production. 5+ years of experience in ML Engineering, AI Engineering, or AI Platform Engineering. Strong Python development experience. Hands-on experience with LLM/GenAI evaluation. Experience designing and implementing evaluation frameworks. Experience developing custom or deterministic evaluators. Experience integrating AI/ML quality checks into CI/CD. Continuous upskilling Mentoring programs Certification support Learning platforms access Extra leave days Referral bonuses Private health insurance Well-being events Recreation areas Snacks and beverages IT equipment (HUP)Skills
Sign in to read and join the discussion.