Quality Assurance Lead - API Testing
Summary
QA Lead designing, automating, and executing API test suites (functional, contract, performance, resilience) and building LLM evaluation pipelines (golden sets, guardrails, latency KPIs) integrated into Azure DevOps CI/CD. Core stack: Python/TypeScript, Postman/Newman, PyTest/Jest, k6/JMeter, Docker, for a regulated clinical product.
QA Lead
Job Description :
What is expected?
- Design, automate, and execute API test suites (functional, contract, performance, resilience).
- Define LLM evaluation strategies (performance testing, golden sets, guardrails, hallucination/consistency checks, relevance & latency KPIs); build automated eval pipelines tied to releases.
- Create PDLM deliverables : verification plans/protocols, test reports, risk-based coverage, V&V trace matrices to requirements and use cases.
- Integrate test infrastructure with Azure DevOps (CI/CD, test orchestration, quality gates, dashboards); enforce build-time gates for API and LLM quality.
- Partner with Product, Clinical, and R&D to refine requirements and acceptance criteria; ensure summaries render correctly for different user roles and workflows.
What is required from candidate?
- 8+ years in backend/API test automation (REST, JSON, auth), performance testing, and CI/CD.
- Hands-on with Python/TypeScript, Postman/Newman or REST-assured, PyTest/Jest, k6/JMeter, Docker.
- Experience validating ML/LLM systems (eval datasets, metrics, prompt/response testing, safety/guardrails).
- Strong documentation and traceability discipline for regulated product lifecycles (PDLM/QMS).
- Bonus : Azure DevOps pipelines, FHIR/HL7 familiarity, healthcare data privacy/security.
- Bonus : Frontend testing capabilities are preferred, but not part of the PDLM requirements.