Quality Assurance Analyst (Gen AI and RAG)
Posted Updated 5
views
Summary
QA Analyst at Infor testing Gen AI/RAG features of its enterprise AI Studio: designing and executing API, search-relevance, multi-agent pipeline, and tenant-isolation tests with Python, REST tools, and OpenSearch, and defining semantic acceptance criteria for non-deterministic LLM outputs.
As a Quality Assurance Analyst at Infor, you will have a critical role in ensuring the delivery of high-quality enterprise applications that meet the evolving needs of our global customers. You’ll collaborate closely with developers, product managers, and customer success teams to validate functionality, performance, and scalability—helping us deliver smarter, faster, and more intuitive software.
- Design and execute test cases for AI Studio's Orchestrator Agent — verifying that the correct components are selected, the right questions are asked, and assumptions are applied accurately for a given user input.
- Validate gap map outputs — confirming that fields are correctly classified as confirmed, assumed, auto-resolved, or missing based on schema contracts and domain defaults.
- Test schema and knowledge document submissions through the Content API — verifying structure validation, field type enforcement, controlled vocabulary checks, and embedding generation.
- Execute search relevance tests across all six OpenSearch indices — verifying that queries return the expected documents with acceptable relevance scores.
- Test multi-agent task packet assembly — confirming that confirmed values, assumed values, upstream context, and output contracts are correctly populated before specialist agents are invoked.
- Verify execution tier ordering — ensuring components with blocking dependencies are never dispatched before their upstream dependencies have completed.
- Perform tenant isolation testing — confirming that session data, project history, and search results are strictly scoped to the correct tenant across all operations.
- Analyse audit logs and search traces to identify the root cause of failures in the multi-agent pipeline and report findings with clear reproduction steps.
- Bachelor's degree in Computer Science, Information Technology, Engineering, or related field. 3–5 years of experience in software quality assurance.
- Solid understanding of REST API testing and hands-on experience with tools such as Postman, REST Assured, or equivalent.
- Experience writing automated test scripts in Python — ability to build repeatable test cases for API-level and integration-level scenarios.
- Familiarity with search systems — understanding of how relevance scoring works and the ability to design queries that validate expected search behaviour.
- Ability to define acceptability criteria for non-deterministic outputs — understanding that LLM-driven responses require semantic evaluation rather than exact-match assertions.
- Experience with integration testing across multi-service or distributed architectures — ability to trace a failure across service boundaries to its root cause.
- Strong analytical skills and attention to detail — particularly important when validating structured JSON outputs such as task packets, gap maps, and deployment blueprints.
- Good written communication skills — ability to write clear, reproducible bug reports and test documentation.
- Exposure to RAG systems, vector search, or LLM-driven applications — either through professional experience or personal projects.
- Familiarity with OpenSearch, Elasticsearch, or similar search platforms.
- Exposure to LLM evaluation tools such as RAGAS, DeepEval, or equivalent frameworks for assessing AI output quality.
- Experience with multi-tenant SaaS testing — particularly data isolation and session boundary validation.
- Familiarity with CI/CD pipelines and the ability to integrate automated test suites into deployment workflows.
- Knowledge of JSON Schema validation and the ability to write schema-level assertions against complex nested JSON structures.
- Exposure to Infor OS, ERP systems, or enterprise workflow automation products.
- Experience designing synthetic test data for knowledge-driven systems where the quality of test documents directly affects the coverage and reliability of test results.