Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

V2soft

AI QA Engineer

Posted 77 views

Description

Work Description :


- Own the acceptance test plan: define how 95% detection accuracy and - 5% area variance are measured, and enforce the split between the calibration set used for tuning and the held-out set used for acceptance


- Build and maintain the coverage matrix across design types and symbol classes, flagging thin or empty cells as known gaps


- Report accuracy per symbol class with confidence intervals, not just an aggregate figure, and enforce the per-class minimum instance count below which a class is reported as provisional


- Run no-regression checks across fine-tune rounds so a new model cannot pass on average while regressing on a rare but critical class


- Build the Knowledge Bot evaluation suite: a curated Q&A set with known-correct answers, scored for groundedness and faithfulness against every prompt and model version


- Perform adversarial and AI security testing before production and after material architectural change prompt injection, jailbreak attempts, RAG-specific attacks, cross-user data isolation and AI data leakage


- Validate correction-quality controls: spot-audit ambiguous classes, track inter-reviewer agreement, and confirm conflicting human corrections are adjudicated before entering the labelled set


- Stand up drift monitoring and produce the periodic model-health report covering drift indicators, correction rates, confidence-score trends, retraining events and acceptance results by version


- Run functional and integration testing of the HE2 workflow alongside the model validation work


What you need :


- Python, and the ability to write your own evaluation harnesses rather than only driving a test tool


- Solid grasp of ML evaluation: precision, recall, per-class metrics, confidence intervals, sampling bias, and why a held-out set must stay held out


- Experience testing an ML or AI system in production - you have caught a model problem, not only an application bug


- Test automation and CI, with pytest or equivalent


- API testing and the discipline to build reproducible, versioned test datasets


- Clear written reporting: your reports go to the customer and support a payment gate


Nice to have :


- LLM red-teaming or prompt injection testing experience


- Familiarity with RAG evaluation frameworks and groundedness scoring


- Statistical background: clustering effects, effective sample size, stratified sampling


- Exposure to SageMaker Model Monitor, MLflow or comparable drift and experiment tracking


- Security testing certification or demonstrable practice


Skills

Apply

See also

QA jobs by country — openings, pay and top skills →