Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Arabic.AI

NewBe an early applicant

Senior QA Engineer - AI Products (6 Months Contract)

Posted

Who recruits here

Up to three recruiters at Arabic.AI with their work email, so you can write to a person.

View plans

Available on Pro.

Description

We are looking for a Senior QA Engineer to own quality across our AI products. You will test LLM-powered features end to end, build automated test and evaluation suites, and help ensure every release is accurate, safe, and reliable.

Responsibilities

  • Define test strategies and plans for AI features across web, API, and backend
  • Evaluate model outputs for accuracy, consistency, hallucinations, bias, and safety
  • Build evaluation datasets, including edge cases and adversarial / red-team prompts
  • Develop and maintain automated UI, API, and regression tests integrated into CI/CD
  • Run regression testing on model, prompt, and configuration changes
  • Test performance, latency, error handling, and fallback behavior
  • Report defects clearly and provide release-readiness sign-off
  • Test non-deterministic outputs using scoring rubrics, LLM-as-judge, human evaluation, and defined pass/fail thresholds
  • Perform AI security testing, including prompt injection, jailbreaks, and PII / data leakage across chatbots, RAG, and agents
  • Work with ML engineers and product managers to define quality metrics and acceptance criteria
  • Where applicable, monitor production quality and model drift after release

Requirements

7+ years of experience in QA / software testing, with strong test automation background.

  • Proven experience testing AI/ML or LLM-based products (chatbots, RAG, agents, etc.)
  • Hands-on scripting experience in Python and/or TypeScript/JavaScript
  • Hands-on with at least one UI automation framework (Playwright, Cypress, Selenium) and a test framework such as Pytest
  • API testing with Postman, REST Assured, or similar
  • Experience with LLM evaluation tools
  • CI/CD and version control: GitHub Actions, GitLab CI or Jenkins; Git
  • Test management and tracking: Jira, TestRail, or Zephyr
  • Excellent written English and clear bug reporting

Nice to Have

  • Performance/load testing (k6, JMeter, Locust)
  • Understanding of prompt engineering, embeddings, and vector databases
  • Experience with AI safety, red-teaming, or content policy testing
  • Cloud experience (AWS, GCP, or Azure) and Docker

Skills

What Senior QA jobs ask for — and how much of it you have →
Apply

See also

QA jobs by country — openings, pay and top skills →