Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Konecta

NewBe an early applicant

Senior QA Engineer — AI Platform & Agentic Solutions

Posted

Description

About the role


We are looking for a Senior QA Engineer with proven expertise in testing agentic systems

to establish and coordinate quality assurance across the platform and all Agentic

Solutions published on it.

Based in Egypt, you will work with developers, engineering leads, the Proxy Product Owner

and business stakeholders across the entire project. You will combine hands-on testing

and automation with the development of consistent quality standards, evaluation

methods and release criteria.

Your scope will cover both the reliability of the underlying platform and the ability of

individual Agentic Solutions to complete business tasks accurately, securely and within

their authorised boundaries.

Key responsibilities

1. QA strategy and practice

• Define and implement a QA strategy covering functional, integration, end-to-end,

regression, performance and AI evaluation testing.

• Establish consistent testing standards across teams and workstreams.

• Prioritise coverage according to business impact and risk, automating critical journeys

first.

• Work with developers to embed testing throughout delivery and improve the testability

of new features.

• Maintain a consolidated view of quality, coverage gaps and release risks.

2. platform testing

• Design and execute tests for platform APIs, user interfaces, tenant provisioning,

authentication, roles and permissions.

• Validate knowledge-base management, agent and crew orchestration, tracing,

configuration and billing capabilities.

• Test integrations between platform components and external services.

• Verify tenant isolation, access boundaries and correct handling of sensitive data,

working with security specialists.

• Test failure handling and recovery, including timeouts, retries, unavailable

dependencies and interrupted workflows.

• Ensure platform changes do not introduce regressions in existing capabilities or

published Agentic Solutions.

3. Agentic Solution validation

• Define and execute a repeatable validation process for every Agentic Solution before

publication and following significant changes.

• Test complete business journeys against agreed requirements and expected

outcomes.

• Validate tool selection, tool inputs, execution sequences, API interactions, knowledge

retrieval, memory and human handoffs where applicable.

• Inspect execution traces to assess intermediate actions and identify the source of

failures.

• Verify that agents handle ambiguous requests, missing information, unexpected

inputs and dependency failures appropriately.

• Confirm that agents respect authorisation boundaries, approval requirements and

limits on their actions.

• Test safeguards against prompt injection, sensitive-data disclosure, unauthorised

actions and unintended or duplicate execution.

• Use controlled test environments to validate actions without affecting live customers

or production data.

4. AI evaluation and regression

• Build and maintain versioned evaluation datasets covering representative business

scenarios, edge cases and adversarial inputs.

• Define measurable acceptance thresholds with the Proxy Product Owner and relevant

business owners.

• Evaluate task completion, factual accuracy, grounding, tool-use correctness and

compliance with solution-specific requirements.

• Combine automated scoring with human review where appropriate, accounting for

variability in AI outputs.

• Run regression evaluations following changes to models, prompts, tools, knowledge

sources or platform components.

• Maintain traceability between evaluation results and the versions of the solution,

model and configuration tested.

5. Test automation and CI/CD

• Build and maintain automated API, integration, browser-based and end-to-end test

suites.

• Integrate tests and AI evaluations into GitHub-based CI/CD workflows.

• Implement agreed quality gates for pull requests, releases and solution publication.

• Maintain reliable test data, fixtures and environments.

• Investigate unstable tests and keep automated results actionable and trustworthy.

6. Performance and reliability

• Test platform and solution behaviour under expected load and concurrent usage.

• Measure latency, throughput, failure rates and recovery behaviour.

• Assess response times and cost per completed agentic journey against agreed

targets.

• Identify bottlenecks and work with engineers to verify improvements.

7. Defect management and release readiness

• Record defects in GitHub with reproducible steps, expected and actual results, severity,

evidence and affected versions.

• Work with the Proxy Product Owner and engineering leads to assess business impact

and prioritise fixes.

• Track defects through resolution, retesting and closure.

• Provide evidence-based release and publication recommendations.

• Document outstanding defects, untested areas and residual risks so designated

release owners can make informed decisions.

• Translate production incidents and recurring defects into regression tests and process

improvements.

8. Reporting and collaboration

• Provide a formal weekly quality update to the Global Head of Platform, Proxy Product

Owner and engineering leads.

• Report test results, critical defects, coverage gaps, AI evaluation trends and release

readiness.

• Escalate significant quality risks promptly, with their impact and recommended

actions.

• Participate in refinement and planning to challenge unclear requirements and

improve acceptance criteria.

• Present relevant findings and demonstrations during team reviews and Show & Tell

sessions.


Required experience and skills

• At least five years of experience in software QA or test engineering, including

substantial hands-on automation experience.

• Demonstrated experience establishing or improving QA practices across software

delivery teams.

• Proven hands-on expertise testing tool-using AI agents and multi-step agentic

workflows, including intermediate actions, failure recovery and business-outcome

completion.

• Practical experience building AI evaluation datasets, automated evaluation pipelines

and regression suites.

• Strong Python skills for test automation, using pytest or equivalent tooling.

• Strong API testing skills and experience with browser automation frameworks such as

Playwright, Cypress or equivalent.

• Experience integrating automated tests into CI/CD pipelines, preferably GitHub

Actions.

• Experience with performance and load testing of API-based services.

• Working understanding of RAG, agent orchestration, tool calling, memory and

execution tracing.

• Familiarity with security and privacy testing relevant to enterprise AI systems.

• SQL and data-validation skills.

• Excellent written and spoken English, with the ability to explain quality risks clearly to

technical and business stakeholders.

• A proactive approach: you identify gaps, investigate issues independently and bring

practical recommendations.


Nice to have

• Experience with AI evaluation tools such as promptfoo, Ragas, DeepEval or equivalent.

• Experience testing enterprise platforms with multiple tenants.

• Background in conversational AI, contact-centre technology or customer experience.

• Experience testing voice agents, including interruptions, speech recognition, latency

and human transfer.

• Exposure to GCP, containerised environments and cloud-native applications.

• Experience with production monitoring and AI observability.

• ISTQB, ISTQB AI Testing or equivalent certification.

Skills

What Senior QA jobs ask for — and how much of it you have →
Apply

See also

QA jobs by country — openings, pay and top skills →