Senior Automation Test Engineer - Playwright/AI
Summary
A senior QA/SDET role owning end-to-end test strategy (unit, API, UI, contract, release gates), building Playwright automation frameworks, and creating LLM/AI evaluation harnesses with golden datasets and regression gates, integrated into CI/CD (Azure DevOps/GitHub Actions). Based in Gurgaon; requires 10+ years in QA and TypeScript, C#, or Python.
About the Role:
We are looking for a Senior Automation Test Engineer to own the end-to-end testing strategy across UI, APIs, and CI/CD pipelines. The role involves building reliable automation frameworks, evaluating AI-generated responses, and establishing trusted pass/fail signals to ensure release quality.
Key Responsibilities:
- Define and implement end-to-end testing strategies covering unit, API, UI, contract, smoke, and release-gate testing.
- Develop and maintain robust automation frameworks using Playwright or equivalent tools, along with API testing frameworks.
- Design AI response testing using golden datasets, prompt/playbook regression tests, structured-output validation, and grounding/citation assertions.
- Build evaluation harnesses for LLM and AI workflows, including scoring, thresholds, failure triage, and regression gates.
- Implement retrieval-quality checks wherever required by the product, without owning the data platform.
- Integrate automated test suites into CI/CD pipelines with a focus on reducing flaky tests and providing clear quality dashboards.
- Mentor team members on risk-based test coverage and maintainable test fixtures for tenant-based or multi-environment data.
Required Skills & Experience:
- B.Tech degree.
- 10+ years of experience in QA / SDET, including ownership of end-to-end product testing strategy at a senior level.
- Strong programming skills in TypeScript, C#, or Python for test automation and evaluation frameworks.
- Hands-on experience evaluating LLM / AI outputs using golden datasets, evaluation rubrics, and regression gates.
- Experience with CI/CD tools such as Azure DevOps or GitHub Actions.
- Strong experience in API and UI automation at scale.
- Proven ability to improve flaky test suites and establish reliable release-quality signals.
Nice to Have:
- Experience testing RAG or hybrid search quality.
- Exposure to OWASP security smoke testing and performance smoke testing within CI/CD pipelines.
- Experience working with ephemeral and seeded test environments.