Team Lead, Quality Assurance - 1
Posted
Summary
Leads the QA team for Infor's AI Studio, an agentic platform for building Infor OS/ERP solutions via natural language. Owns QA strategy covering RAG search relevance, multi-agent orchestration, LLM output evaluation, and tenant isolation, using Python test automation and LLM eval tools like RAGAS/DeepEval.
Infor's AI Studio represents an agentic platform that enables teams to author, configure, and deploy Infor OS and ERP solutions utilizing natural language processing capabilities. We are seeking an Lead Quality Assurance Engineer to join the AI Studio team and lead the QA team to thoroughly validate the agent architecture of AI Studio. The testing will spread across components like ION, Data Fabric, Birst, EPM, RPA/IDP, CSD, and Portal. This initiative constitutes a strategic priority for Infor, designed to accelerate velocity suite sales, enhance customer experience, and facilitate more effective collaboration between Cutomers and Sevices, FDEs, Partners and SCs
We are seeking a Team Lead for Quality Assurance to lead the testing efforts to deliver high quality software solutions across our cloud product portfolio.
We are seeking a Team Lead for Quality Assurance to lead the testing efforts to deliver high quality software solutions across our cloud product portfolio.
A Day in the Life Typically Includes:
- Define and own the overall QA strategy for AI Studio — covering search relevance, multi-agent coordination, schema validation, tenant isolation, and LLM output quality.
- Lead a team of QA engineers, assign test coverage areas, review test plans, and ensure quality gates are met before each release.
- Collaborate with the AI Studio architecture team, component teams, and FDEs to understand new features, contribution model changes, and agent updates — translating these into test requirements.
- Drive the design of evaluation frameworks for non-deterministic Orchestrator outputs — defining what "correct" means for gap maps, dependency graphs, and blueprint quality across varying user inputs.
- Oversee search relevance regression testing — ensuring that as the knowledge corpus grows, existing query-result quality does not degrade.
- Define and enforce tenant isolation testing standards, ensuring no cross-tenant data leakage across sessions, project history, or search results.
- Report quality metrics to engineering leadership — solution generation success rates, clarification round counts, agent failure rates, and corpus coverage gaps.
Required Qualifications:
- Bachelor's degree in Computer Science, Information Technology, Engineering, or related field. 8–10 years of experience in software quality assurance with at least 3 years in a lead or principal QA role.
- Proven experience leading QA for an AI or data-intensive product in production — not just familiarity with concepts but hands-on ownership of quality outcomes.
- Deep understanding of RAG system testing — search relevance validation, embedding quality assessment, hybrid search behaviour, and corpus quality measurement.
- Strong experience with multi-agent or distributed system testing — tracing failures across service boundaries, validating inter-agent contracts, and verifying execution ordering.
- Experience building and maintaining automated test suites in Python, with the ability to define quality frameworks from scratch rather than inheriting them.
- Solid understanding of LLM evaluation methodologies and tools such as RAGAS, DeepEval, or equivalent — with the ability to select and adapt them to non-standard evaluation problems.
- Experience with REST API contract testing and integration testing across multi-service architectures.
- Experience with at least one major cloud platform: AWS (Preferred), Azure, or GCP.
- Strong communication and stakeholder management skills — ability to translate quality findings into clear risk assessments for engineering leadership and product teams.
Preferred Skills:
- Experience leading QA for agentic AI systems or LLM orchestration pipelines where output correctness is semantic rather than deterministic.
- Familiarity with OpenSearch, Elasticsearch, or other vector and hybrid search platforms — ideally with hands-on experience designing relevance test suites.
- Knowledge of prompt adversarial testing — designing inputs that expose reasoning failures, hallucinations, incorrect component selection, or silent gap map errors.
- Experience with MLOps or LLMOps observability tooling — defining monitoring standards, alert thresholds, and trace inspection processes for AI pipelines.
- Familiarity with multi-tenant SaaS security testing, particularly tenant isolation and session boundary enforcement.
- Exposure to Infor OS, ERP systems, or enterprise workflow automation products — understanding the domain context that AI Studio operates in.
- Experience establishing QA contribution processes for knowledge-driven systems where test data quality directly determines test coverage effectiveness.
- Familiarity with CI/CD pipelines and DevOps practices — ability to integrate quality gates into automated deployment workflows.
