Senior AI Testing- QA Engineer -- (Bangalore, Mumbai )
Summary
Senior AI QA Engineer at QualityKiosk Technologies, a quality-engineering services firm. Day to day: testing GenAI bots, agents and RAG pipelines by designing prompt-based, multi-turn and adversarial test scenarios, then evaluating AI behavior (hallucinations, grounding, drift, safety) using AI evaluation/observability platforms like Arize or Phoenix. On-site roles in Mumbai (Navi Mumbai) and Beng
JOB DESCRIPTION # Senior AI QA Engineer -Testing-
Job location: Bangalore, Mumbai
About QualityKiosk Technologies
Founded in 2000, the company specializes in providing quality engineering, QA automation, performance assurance, intelligent automation (IA) and robotic process automation (RPA), customer experience management, site reliability engineering (SRE), digital testing as a service (DTaaS), cloud, and data analytics solutions and services.
Website: www.qualitykiosk.com
Please find below job details fyr:
kindly revert with your updated CV and below details at email ID - clive.lobo@qualitykiosk.com
JD # Senior AI QA Engineer -Testing
We are looking for an AI QA Engineer with hands-on experience working on GenAI platforms and workflows, particularly AI observability, evaluation, testing, or safety platforms similar in nature to Arize, Cekura, Fiddler, Giskard, WhyLabs, Phoenix, or equivalent systems.
This role requires direct exposure to GenAI bots, RAG pipelines, and AI workflows in real platforms, not theoretical knowledge or generic QA experience. You will evaluate AI behavior, analyze failures, and contribute to improving the quality, reliability, and safety of GenAI systems.
This is not a traditional QA role and not suitable for candidates who have only tested UI, APIs, or deterministic systems.
Key Responsibilities
- Execute hands-on AI evaluation and QA on:
- GenAI bots (chatbots, copilots, agents)
- RAG-based workflows
- AI evaluation and observability platforms
- Design and execute:
- Prompt-based test scenarios
- Multi-turn conversation tests
- Edge-case and adversarial evaluations
- Evaluate AI behavior for:
- Hallucinations
- Grounding failures
- Inconsistency and drift
- Safety and reliability issues
- Work within existing AI platforms (evaluation, observability, monitoring) to:
- Analyze traces, metrics, and outputs
- Interpret model behavior across runs
- Document findings clearly in:
- Evaluation reports
- Platform dashboards
- Structured test summaries
- Collaborate with:
- AI QA Lead on methodology and review
- ML and Product teams to provide actionable feedback
Must-Have Skills & Experience
- 3-5+ years of experience in AI QA, AI evaluation, or GenAI testing
- Hands-on experience working on GenAI platforms, such as:
- AI evaluation platforms
- AI observability or monitoring tools
- GenAI safety or reliability platforms
- Direct experience testing:
- GenAI chatbots or agents
- RAG pipelines and workflows
- Multi-step AI interactions
- Strong understanding of:
- Prompt-based testing
- Non-deterministic AI behavior
- Quality vs correctness in GenAI systems
- Ability to clearly explain why an AI response is good or bad
kindly revert with your updated CV and below details at email ID - clive.lobo@qualitykiosk.com
Full Name:
Total Exp:
Rel exp in AI Chatbot Testing, GenAI testing?
Relevant Hands-on experience testing AI Testing, or GenAI Testing?
Current Company:
CTC:
ECTC:
Notice Period:
Reason for job change:
Current Residential location:
Ready to work from office?
Preferred job location: Mumbai,
Any offers in Hand? (Please share details CTC, LWD, DOJ):
Thanks & Regards,
Clive Lobo
QualityKiosk Technologies
clive.lobo@qualitykiosk.com