AI Assurance & Testing Lead
Description
Job Description – AI Assurance & Testing Lead
Role Details
- Location: Copenhagen, Denmark
- Contract Duration: 6 months initially, with the possibility of extension – B2B contract
- Working Arrangement: 2 days per week (flexible)
- Payrate: €300-330/day
Role Summary
We are looking for an experienced AI Assurance & Testing Lead to drive quality assurance, testing strategy, and governance for three AI assistants. The role will operate alongside the User Acceptance Testing (UAT) phase and will be responsible for ensuring that all AI assistants meet the required quality, performance, security, and responsible AI standards.
The successful candidate will lead the end-to-end assurance process, including test strategy development, AI model evaluation, automated regression testing, responsible AI assessments, and adversarial testing. The role will also oversee quality gates and deliver the final MVP Assurance Report, providing an evidence-based recommendation to support the client's Go/No-Go decision.
Key Responsibilities
1. Business Assurance & Test Strategy
- Define testing strategies, scope, and acceptance criteria for each AI assistant.
- Maintain traceability between business requirements, test scenarios, and test results.
- Ensure testing activities align with business expectations and project objectives.
2. Retrieval & Generation Benchmarking
- Design evaluation datasets, benchmarks, and performance metrics.
- Develop automated regression testing frameworks.
- Monitor AI performance and identify model behaviour changes and performance drift.
3. Responsible AI Testing
- Evaluate fairness, toxicity, PII leakage, transparency, and human oversight.
- Assess AI assistants against client-specific responsible AI policies.
- Identify risks and ensure compliance with AI governance standards.
4. Adversarial & Red-Team Testing
- Conduct adversarial testing to identify AI vulnerabilities.
- Test prompt injection, jailbreaks, data exfiltration, unsafe outputs, and tool misuse.
- Validate remediation measures.
5. Quality Gate & Governance Management
- Define quality gates, testing milestones, and entry/exit criteria.
- Lead quality reviews and coordinate client approvals and sign-offs.
6. UAT Coordination
- Align UAT scenarios with the overall AI assurance strategy.
- Coordinate with business stakeholders and delivery teams.
- Manage defect identification, prioritisation, tracking, and resolution.
7. MVP Assurance Reporting
- Consolidate testing outcomes, performance metrics, and identified risks.
- Prepare the final MVP Assurance Report.
- Provide an evidence-based Go/No-Go recommendation.
Required Qualifications & Technical Skills
- 8+ years of experience in Software QA and Testing.
- 3+ years of hands-on experience testing AI/ML or LLM-based applications.
- Practical experience evaluating Retrieval-Augmented Generation (RAG) systems.
- Experience with LLM evaluation frameworks and tools.
- Strong Python test automation experience, particularly Pytest.
- Experience with CI/CD pipelines.
- Working knowledge of Jira, Azure DevOps, or similar tools.
- Strong analytical, problem-solving, and communication skills.
Ideal Candidate Profile
- Strong understanding of AI assurance, LLM testing methodologies, and Responsible AI principles.
- Hands-on experience with AI-powered application testing.
- Ability to independently manage testing strategies and governance.
- Experience working with business stakeholders, technical teams, and clients.
Skills
Sign in to read and join the discussion.