Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Reply

AI Testing & Evaluation Engineer (Python)

Posted Updated

Description

Reply is seeking an AI Test Engineer to design and develop Python-based testing frameworks for evaluating AI agents and LLM-based systems. You will help shape evaluation pipelines, observability, and data-driven testing, collaborating closely with AI and data teams.

The role requires 2-5+ years in software/AI engineering, strong Python skills, and experience with PyTest/Playwright, CI/CD, and NLP metrics. This position is based in the United Kingdom with potential for growth and impact.

Design and develop Python-based frameworks for testing and evaluating AI agents and LLM-based systems Contribute to architecture design for evaluation pipelines, observability, and data-driven testing systems Build and maintain tools for test data generation (including synthetic and adversarial datasets) Define and implement evaluation strategies and quality KPIs for AI behavior (accuracy, robustness, bias, consistency, hallucination rate, etc.) Integrate LLM evaluation tools, CI/CD pipelines, and external AI platforms Support continuous testing, benchmarking, and production monitoring of AI systems Collaborate with AI engineers, data engineers, and QA teams to improve system reliability and scalability 2-5+ years of experience in Software Engineering, Test Automation, AI Engineering, or Data Engineering Strong proficiency in Python and data/AI libraries (e.g., pandas, numpy, PyTorch or similar) Solid understanding of LLMs, NLP concepts, or generative AI systems Experience with test automation frameworks (e.g., PyTest, Playwright) and CI/CD pipelines Familiarity with data pipelines, dataset design, or feature engineering is a strong plus Experience with evaluation metrics for NLP/AI systems (BLEU, ROUGE, embedding-based metrics, or custom scoring approaches) Ability to design scalable systems and work autonomously on complex technical problems Strong analytical mindset and interest in AI system quality, reliability, and governance

Skills

Apply

See also