Senior Software Quality Engineer – AI

We are seeking a Senior Software Quality Engineer – AI Focused to drive quality, reliability, and automation for AI-enabled software products. The role will collaborate with Development, DevOps, Product, and QE teams to design robust testing strategies and scalable automation frameworks. The engineer will test AI/ML and Generative AI features, APIs, microservices, integrations, and data pipelines while ensuring reliable and secure product delivery. The position is full-time and onsite in Toronto, ON.

Key Requirements

• 5+ years of experience in Quality Engineering, QA, SDET, or Test Automation.
• 5+ years of programming experience with Java, Python, Groovy, or similar languages.
• Strong experience building and maintaining test automation frameworks.
• Hands-on experience with Selenium, Playwright, Cucumber/Gherkin, or RestAssured.
• Experience with API, database, integration, regression, and microservices testing.
• Experience with JUnit, TestNG, ReadyAPI, qTest, and JMeter.
• Hands-on experience testing AI/ML, Generative AI, conversational AI, or LLM-based applications.
• Knowledge of AI quality concepts such as prompt evaluation, accuracy, relevance, grounding, hallucination detection, toxicity, bias, and response consistency.
• Experience with CI/CD, Agile, BDD, TDD, ATDD, and risk-based testing.
• Strong analytical, problem-solving, communication, and documentation skills.

Key Responsibilities

• Develop and execute comprehensive quality strategies, test plans, test cases, and test data.
• Design, develop, and maintain scalable test automation frameworks.
• Create automated tests for AI/ML and Generative AI features.
• Evaluate AI responses for accuracy, relevance, safety, hallucination, reliability, and latency.
• Test APIs, microservices, databases, integrations, data pipelines, and third-party services.
• Create AI evaluation datasets, expected behaviors, quality thresholds, and regression coverage.
• Integrate automated testing and AI quality checks into CI/CD pipelines.
• Identify, document, prioritize, and track defects through resolution.
• Analyze test coverage, quality metrics, AI evaluation results, and production feedback.
• Collaborate with Development, DevOps, Product, and QE teams to improve product quality and release readiness.