Senior Test Automation Engineer
Posted Updated
Description
One of our clients is looking for a Senior Automation Test Engineer who will own the full testing strategy across UI, API, and pipelines - including automation, CI gates, and evaluation of AI responses - so every release has clear, trusted pass/fail signal.
Key responsibilities
- Layout the end-to-end test strategy: unit, API, UI, contract, smoke, and release gates.
- Implement durable automation (Playwright or equivalent + API frameworks) for critical user journeys.
- Design AI response testing: golden sets, prompt/playbook regressions, structured-output checks, grounding/citation asserts.
- Build eval harnesses for LLM / AI workflow outputs - scoring, thresholds, failure triage, and regression gates.
- Cover retrieval-quality checks where the product needs them, without owning the data platform.
- Wire suites into CI/CD with flaky-test discipline and clear dashboards for engineering and release owners.
- Mentor the team on risk-based coverage and maintainable fixtures for tenant or multi-env data.
Must-have experience
- B.tech Degree
- 10+ years QA / SDET; owned whole-product test strategy at senior level.
- Strong coding (TypeScript, C#, or Python) for automation and eval harnesses.
- Proven LLM / AI output evaluation (golden datasets, rubrics, regression gates).
- CI/CD (Azure DevOps / GitHub Actions) and API + UI automation at scale.
- Track record turning flaky suites into trusted release signal.
Nice to have
- Experience testing RAG or hybrid search quality.
- Security smoke (OWASP) and performance smoke in the same pipeline.
- Ephemeral, seeded test environments.