AI Operations & QA Engineer - Hybrid
About the role
The Automation QA Engineer defines, implements, and continuously improves quality assurance practices for AI-enabled and agent-based solutions. The role ensures systems meet agreed standards for accuracy, reliability, performance, safety, and compliance, and supports operational readiness through automated evaluation and monitoring.
Key responsibilities
Define quality criteria and testing strategies for agent workflows, covering accuracy, latency, safety, compliance, and operational risk
Build automated evaluation harnesses to assess agent performance, including hallucination rates, tool misuse, policy violations, and task success
Implement continuous production monitoring to detect anomalies, quality degradation, and emerging safety concerns
Develop and maintain automated test suites using Playwright for UI testing and custom scripts for API and workflow validation
Apply LLM evaluation frameworks to assess output quality, regression, and system drift over time
Produce and maintain dashboards and reports that communicate quality metrics and trends to engineering and stakeholders
Develop and maintain runbooks for common failure modes and contribute to incident response activities
Collaborate closely with developers to improve prompts, tool definitions, and workflow designs based on test results
Ensure testing, logging, and monitoring practices align with data privacy, audit, and regulatory requirements
About you
Minimum 3 years' experience in QA, test automation, or DevOps roles (or 2 years with direct experience testing AI or ML-enabled systems)
Strong Python skills for test automation, evaluation harnesses, and basic data analysis
High attention to detail, with a focus on issues that materially impact reliability and user trust
Comfort working with evolving tools, frameworks, and testing practices
Collaborative mindset, using evidence-based insights to influence product and engineering decisions
Programming: Python (test automation, evaluation harnesses, data analysis)
UI Automation: Playwright (end-to-end workflow testing)
AI Evaluation: Deepeval, RAGAS, Evidently.AI (LLM quality, drift, and regression analysis)
Workflow Testing: API and agent workflow validation using custom scripts
Monitoring: Production quality monitoring and anomaly detection
Preferred skills/experience
Pytest or equivalent testing frameworks
SQL for querying logs, metrics, or evaluation datasets
Prometheus, Grafana, or similar monitoring tools
Familiarity with hallucination detection and AI safety patterns
CI/CD pipelines and Git-based workflows