Mid-Level QA Engineer (AI & Automation)

We are looking for a mid-level **QA Engineer** to own quality for a platform that integrates with multiple AI providers and uses them for data enrichment. You will combine manual testing, test automation, and AI-specific validation to make sure enriched outputs are accurate, consistent, and reliable in production. #### **Responsibilities** - Design and execute manual test plans, test cases, and exploratory testing for web and API features. - Build and maintain automated test suites for UI, API, and integration layers, and wire them into CI/CD. - Validate AI-enriched outputs for accuracy, relevance, consistency, and hallucination risk, using golden datasets and evaluation criteria. - Test integrations with multiple AI providers (such as OpenAI, Anthropic, and others), including failure handling, timeouts, rate limits, fallback behavior, and provider response variations. - Compare output quality across providers and model versions, and flag regressions when models or prompts change. - Test prompt changes, edge cases, adversarial inputs, and data quality issues in the enrichment pipeline. - Monitor cost, latency, and token usage behavior in test scenarios and report anomalies. - Log clear, reproducible bugs and work closely with developers, product owners, and DevOps to resolve them. - Contribute to QA processes, documentation, and release sign-off. #### **Required Skills** - 3-5 years of QA experience covering both manual and automation testing. - Hands-on experience with automation tools such as Playwright, Cypress, or Selenium. - Strong API testing skills (Postman, REST Assured, or pytest/requests-based frameworks). - Solid understanding of testing non-deterministic systems, including LLM output evaluation, prompt testing, and handling variability in results. - Experience testing third-party integrations, webhooks, and asynchronous or queue-based workflows. - Familiarity with CI/CD pipelines (GitHub Actions, GitLab CI, or Jenkins) and version control with Git. - Working knowledge of SQL and basic scripting in Python or JavaScript/TypeScript. - Strong written communication and bug reporting skills. #### **Nice to Have** - Experience with AI evaluation tools or frameworks (such as promptfoo, DeepEval, or Ragas). - Understanding of embeddings, RAG pipelines, and vector databases. - Experience with performance and load testing (k6, JMeter, Locust). - Exposure to Rails or Python/FastAPI backends, Redis, and background job systems like Sidekiq. - Familiarity with AWS environments and log/monitoring tools. - Experience with data-heavy or healthcare-adjacent products, including data privacy awareness. #### **What Success Looks Like** Within three months, you will have a stable automated regression suite covering core enrichment flows, a repeatable AI output evaluation process, and clear quality metrics that the team trusts for release decisions. #### **What We Offer** - Work on a real production AI platform with multiple provider integrations. - A collaborative engineering team and room to shape QA practices. - Competitive salary and growth opportunities.