AI Quality and Test Automation
Posted Updated
Summary
QA Automation Engineer (3-5 yrs) in the Kolkata office who builds and maintains code-based UI and API test automation (Playwright/Cypress/Selenium with TypeScript or Python), wires tests into CI/CD pipelines, and tests and evaluates LLM-powered features using datasets and AI evaluation tools.
We are looking for a skilled QA Automation Engineer with strong experience in code-based test automation, API testing, and CI/CD integration. The ideal candidate should have hands-on experience in automating web applications and evaluating AI-powered features.
Key Responsibilities
- Develop and maintain automated UI and API test scripts across products.
- Integrate automated tests into CI/CD pipelines to ensure release quality.
- Build accuracy and regression tests for AI-powered features.
- Implement dependency and security vulnerability scanning.
- Support requirement analysis, acceptance criteria, and user acceptance testing.
- Leverage AI tools to generate test cases, scripts, synthetic data, and troubleshoot failures.
- Test LLM outputs using datasets and AI-based evaluation techniques.
Must-Have Skills
- 35 years of experience in code-first test automation.
- Hands-on experience with Playwright, Cypress, or Selenium using TypeScript or Python.
- API testing experience with Postman/Newman or automation frameworks.
- Experience with GitHub Actions or similar CI/CD tools.
- Good knowledge of SQL and software testing practices.
- Understanding of testing and evaluating LLM-powered applications.
- Regular experience using AI tools such as Claude Code.
- Willingness to work from the Kolkata office.
Good to Have
- Experience with LLM evaluation tools such as Promptfoo, DeepEval, or Ragas.
- Knowledge of Behaviour-Driven Development (BDD) and security scanning tools