Senior QA Engineer (AI-based)
Summary
Senior QA Engineer tests AI-driven conversational agents and data pipelines, designing LLM test scenarios, validating outputs, and ensuring safety/compliance in generative AI systems.
Senior QA Engineer
Summary
We are seeking a detail-oriented Senior QA Engineer to keep the quality assurance efforts across our complex data flows and AI-driven products specifically focusing on conversational agents. This role requires a blend of automation expertise and strong analytical skills to define quality standards validate complex data outputs and ensure the reliable and responsible behavior of our generative AI systems.
Key Responsibilities
- Expertise in quality assurance efforts for data pipelines APIs and conversational agents.
- Design and execute generative AI testing scenarios including exploratory testing for chatbot interactions intent recognition tone and edge-case behavior.
- Experience in LLM test designs including prompt and grounding validation golden sets and non-deterministic assertions.
- Perform crucial safety and quality checks such as testing for hallucination bias toxicity and PII leakage.
- Validate structured and unstructured data outputs ensuring consistency accuracy and compliance.
- Establish and drive real-time testing approaches including streaming data validation and API monitoring.
- Collaborate with ML and NLP teams to define comprehensive evaluation metrics and criteria for agent performance.
- Integrate AI-driven tools (like Copilot) into the QA lifecycle to accelerate test design documentation and defect analysis.
Requirements
- 5 years of experience in QA engineering with proven experience in testing data systems or AI applications.
- Strong analytical skills and attention to detail capable of spotting subtle inconsistencies in data and agent behavior.
- Proficiency in Python and SQL for data validation automation scripting and test preparation.
- Experience with API testing (REST/GraphQL automation and validation).
- Familiarity with chatbot frameworks LLMs or conversational testing.
- (Desirable) Experience or strong interest in using LLM-based assistants (e.g. GitHub Copilot ChatGPT) in test and execution.
- (Desirable) Exposure to QA automation tools (e.g. PyTest Postman dbt tests).
Location:Candidates based in UAE (Dubai or Abu Dhabi)
Required Experience:
Senior IC