Agentic QA Engineer
Posted Updated
Description
- TribalScale is looking for an Agentic QA Engineer who combines strong software testing and quality engineering expertise with hands-on experience in AI-powered applications and agentic systems
- This role is similar to a Forward Deployed Engineer (FDE) or Agentic Engineer, with a particular focus on software quality, test automation, and validating AI-driven workflows
- You will help ensure that AI agents and LLM-powered applications perform reliably in real-world environments by designing testing strategies, building automated evaluation frameworks, and identifying edge cases and failure modes
- Design and implement testing strategies for AI agents, LLM-powered applications, and agentic workflows
- Build automated test frameworks for functional, integration, regression, and end-to-end testing
- Develop evaluation pipelines to measure agent accuracy, reliability, task completion, and consistency
- Test agent tool usage, API integrations, multi-step reasoning workflows, and interactions with external systems
- Identify, reproduce, and troubleshoot failures such as hallucinations, incorrect tool calls, unexpected outputs, and workflow breakdowns
- Establish quality metrics, evaluation datasets, test scenarios, and benchmarks for AI-driven applications
- Integrate automated testing and quality checks into CI/CD pipelines
- Collaborate with AI engineers, software developers, product teams, and client-facing engineering teams to improve system reliability
- Investigate production issues and use findings to improve agent behavior, test coverage, and application performance
- Evaluate edge cases, failure recovery, security considerations, and resilience in AI-powered systems
Benefits
- Health Insurance: Health, dental, vision, life insurance
- Professional Development: Weekly Lunch & Learns, subsidies on courses and workshops to help you upskill
- Conferences & Events: Constant opportunity to attend conferences and events, tickets on us
- Socials: Friday socials, group hangouts, themed parties, quarterly events!
- Flexible Work Schedule: Ability to fit work into your personal schedule
- Parental Leave: Paid parental leave for moms and dads
- Fitness Perk: Subsidized corporate membership of Class Pass offering an array of classes; Goodlife corporate gym membership which offers 30% discount as well as sign-up fees waived
- Mental Health Perk: Subsidized online therapy sessions and resources with Inkblot
- Long Weekend Perk: The option for extra long weekends to spend with your family & friends
- Take your birthday off: Go ahead, celebrate!
- Inward Strong Partnership: TribalScale has sponsored an Inward Strong platform membership for all our Tribe members
- Vet Perks: 4 Virtual Vet Appointments annually for your Furry Friends
- The ideal candidate is a hands-on engineer who can write code, debug complex systems, and apply software testing principles to applications where outputs may be dynamic, nondeterministic, and dependent on multi-step agent interactions
- Proficiency in Python, JavaScript, TypeScript, or another relevant programming language
- Familiarity with Git, CI/CD pipelines, and modern software development practices
- Experience debugging applications and investigating complex technical issues
- Experience building automated tests and testing frameworks
- Strong understanding of API, integration, regression, and end-to-end testing
- Strong analytical thinking and the ability to design test cases for complex, real-world scenarios
- Professional experience in Software QA, Quality Engineering, SDET, Test Automation, or Software Engineering with a strong testing focus
- Experience testing LLM-based applications, AI agents, or generative AI products
- Familiarity with agentic frameworks such as LangGraph, LangChain, CrewAI, or similar technologies
- Understanding of hallucinations, nondeterministic outputs, prompt injection, and agent tool-use failures
- Familiarity with tools such as Pytest, Playwright, Cypress, or Selenium
- Experience with LLM evaluation, prompt testing, model benchmarking, or AI observability
- Previous experience in Agentic Engineering, Forward Deployed Engineering, Applied AI Engineering, or AI product development
- Experience with cloud platforms, APIs, distributed systems, and production deployments
- Core focus: AI agent reliability, software testing, test automation, and production-ready AI systems
- Candidates with strong programming and automation skills who have tested AI-powered applications are especially relevant. We are also open to Agentic Engineers or FDEs who have meaningful experience building testing frameworks, validating complex workflows, and improving software reliability
- The ideal candidate combines the engineering mindset of an FDE or Agentic Engineer with the quality-focused approach of an experienced QA Automation Engineer or SDET
Skills
Sign in to read and join the discussion.