Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

apreel

NewBe an early applicant

QA Engineer (AI + backend)

Posted Updated

Description

Wynagrodzenie
B2B: 160 – 170 PLN / godz. netto

Engagement: We are partnering with a US-based health-tech company on the takeover of a production AI-powered mobile coaching platform. Real users, real production, real releases — not a greenfield project.

Role summary: Own quality on the two hardest surfaces of a production AI coaching platform:

the LLM agents that answer differently every run, and the API layer everything else depends on.

As AI & Backend QA Engineer you own the quality of what the model says and what the APIs return. On the AI side that means building the evaluation practice: golden sets, quality metrics, and regression detection that catch a degraded answer before a user does. On the backend side it means API test suites and consumer-driven contract testing with Pact — agreed with the client's US engineering team as a standing requirement for this role, not an optional technique.

The defining skill is localization: when an answer is wrong, you can tell whether the problem sits in the model, in the service code, or in the input data — and you bring the evidence, not a guess.

What you'll be testing:

  • AI core — a Python service with LangGraph agents producing streaming, non-deterministic coaching responses; this

  • role's primary surface

  • Backend APIs — a NestJS REST gateway behind Auth0, plus the services behind it

  • Mobile app — React Native, shipping to iOS and Android app stores; AI features surface directly in the app that

  • members use

  • Data layer — MongoDB, Redis, and Postgres behind the services

  • Process — Agile/Scrum, Jira for work tracking and defects, TestRail for test management

  • You join the Polish QA function alongside our Manual QA Engineer and QA Automation Engineer, and work in a shared workflow with the client's US-based engineering and QA teams.

What you'll do:

  • Build and run LLM evaluation. Curate datasets and golden sets, define response quality metrics, and catch quality regressions between prompt and model versions before they reach production

  • Define acceptance criteria for non-deterministic output. Assertions on properties and contracts — structure, constraints, invariants — rather than exact wording

  • Treat prompts as tested artifacts. Change a prompt, measure the effect of that change; probe systematically for hallucinations and prompt injection

  • Own API testing. REST endpoints, JSON payloads, auth token flows, error handling — driven through Postman and Swagger/OpenAPI specs

  • Implement contract testing with Pact. Consumer-driven contracts between the mobile app, the gateway, and the backend services, verified on every change

  • Test with security in mind. Authorisation boundaries, token misuse, malformed input — the API-level abuse cases nobody writes a happy-path test for

  • Localise failures across the stack. Logs, traces, and database queries to pin a defect to the model, the service code, or the input data

  • Help wire evaluations into CI/CD. Evaluation runs in the pipeline and quality gates on pull requests, so a bad prompt change cannot merge quietly

  • Keep the record straight. Results mapped to test cases in TestRail, defects logged and driven to closure in Jira, findings written up so the team can act on them

Must-have skills:

  • 3+ years in technical QA or test automation, including API testing on production systems

  • Python — writing and maintaining test code, pytest, fixtures

  • Strong HTTP and API fundamentals — REST, JSON, status codes, headers, auth tokens

  • Contract testing with Pact — consumer-driven contracts in practice; this is a hard requirement for the role, agreed with the client's engineering team

  • Security-aware API testing — authorisation boundaries, token handling, malformed and hostile input

  • Comfort with non-determinism — defining acceptance criteria when there is no single correct output: assertions on properties and contracts rather than exact wording

  • LLM evaluation — datasets and golden sets, response quality metrics, catching quality regressions between prompt and model versions

  • Prompt engineering — the ability to change a prompt and measure the effect of that change; awareness of hallucinations and prompt injection

  • Backend savvy — HTTP and REST, JSON, tokens, logs, database queries; you can tell whether a problem sits in the model, in the service code, or in the input data

  • Postman and Swagger/OpenAPI as daily tools

  • Git in a team workflow — branches, pull requests, reviews

  • TestRail for test-case management and run tracking

  • Mobile product context (Android/iOS) — comfortable testing features that ship in a mobile app, including apprelease timing and store builds

  • Strong English — you work in a shared workflow with US colleagues

  • Must work hours that overlap US Mountain Time afternoons (~14:00–18:00 PL time)

Nice-to-haves:

  • Langfuse or equivalent (LangSmith, Phoenix) — working with traces and attaching evaluations to them

  • LangGraph — testing an agent as a state graph, both a single node and the full flow

  • CI/CD — running evaluations in a pipeline and quality gates on pull requests

  • Python with Appium — mobile automation exposure alongside the API work

  • Testing other AI or LLM-driven features in production

  • Experience taking over quality ownership of someone else's product

Offer:

  • Location: 100% remote

  • Engagement: B2B contract

  • Duration: Initial 3-month contract, expected to extend

  • Rate: upt to 160 PLN/h

  • Start: asap

Skills

What Middle Backend jobs ask for — and how much of it you have →
Apply

See also

Backend jobs by country — openings, pay and top skills →