Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

TribalScale

NewBe an early applicant

Agentic QA Engineer

Posted Updated

Description

  • TribalScale is looking for an Agentic QA Engineer who combines strong software testing and quality engineering expertise with hands-on experience in AI-powered applications and agentic systems
  • This role is similar to a Forward Deployed Engineer (FDE) or Agentic Engineer, with a particular focus on software quality, test automation, and validating AI-driven workflows
  • You will help ensure that AI agents and LLM-powered applications perform reliably in real-world environments by designing testing strategies, building automated evaluation frameworks, and identifying edge cases and failure modes
  • Design and implement testing strategies for AI agents, LLM-powered applications, and agentic workflows
  • Build automated test frameworks for functional, integration, regression, and end-to-end testing
  • Develop evaluation pipelines to measure agent accuracy, reliability, task completion, and consistency
  • Test agent tool usage, API integrations, multi-step reasoning workflows, and interactions with external systems
  • Identify, reproduce, and troubleshoot failures such as hallucinations, incorrect tool calls, unexpected outputs, and workflow breakdowns
  • Establish quality metrics, evaluation datasets, test scenarios, and benchmarks for AI-driven applications
  • Integrate automated testing and quality checks into CI/CD pipelines
  • Collaborate with AI engineers, software developers, product teams, and client-facing engineering teams to improve system reliability
  • Investigate production issues and use findings to improve agent behavior, test coverage, and application performance
  • Evaluate edge cases, failure recovery, security considerations, and resilience in AI-powered systems

Benefits

  • Health Insurance: Health, dental, vision, life insurance
  • Professional Development: Weekly Lunch & Learns, subsidies on courses and workshops to help you upskill
  • Conferences & Events: Constant opportunity to attend conferences and events, tickets on us
  • Socials: Friday socials, group hangouts, themed parties, quarterly events!
  • Flexible Work Schedule: Ability to fit work into your personal schedule
  • Parental Leave: Paid parental leave for moms and dads
  • Fitness Perk: Subsidized corporate membership of Class Pass offering an array of classes; Goodlife corporate gym membership which offers 30% discount as well as sign-up fees waived
  • Mental Health Perk: Subsidized online therapy sessions and resources with Inkblot
  • Long Weekend Perk: The option for extra long weekends to spend with your family & friends
  • Take your birthday off: Go ahead, celebrate!
  • Inward Strong Partnership: TribalScale has sponsored an Inward Strong platform membership for all our Tribe members
  • Vet Perks: 4 Virtual Vet Appointments annually for your Furry Friends
  • The ideal candidate is a hands-on engineer who can write code, debug complex systems, and apply software testing principles to applications where outputs may be dynamic, nondeterministic, and dependent on multi-step agent interactions
  • Proficiency in Python, JavaScript, TypeScript, or another relevant programming language
  • Familiarity with Git, CI/CD pipelines, and modern software development practices
  • Experience debugging applications and investigating complex technical issues
  • Experience building automated tests and testing frameworks
  • Strong understanding of API, integration, regression, and end-to-end testing
  • Strong analytical thinking and the ability to design test cases for complex, real-world scenarios
  • Professional experience in Software QA, Quality Engineering, SDET, Test Automation, or Software Engineering with a strong testing focus
  • Experience testing LLM-based applications, AI agents, or generative AI products
  • Familiarity with agentic frameworks such as LangGraph, LangChain, CrewAI, or similar technologies
  • Understanding of hallucinations, nondeterministic outputs, prompt injection, and agent tool-use failures
  • Familiarity with tools such as Pytest, Playwright, Cypress, or Selenium
  • Experience with LLM evaluation, prompt testing, model benchmarking, or AI observability
  • Previous experience in Agentic Engineering, Forward Deployed Engineering, Applied AI Engineering, or AI product development
  • Experience with cloud platforms, APIs, distributed systems, and production deployments
  • Core focus: AI agent reliability, software testing, test automation, and production-ready AI systems
  • Candidates with strong programming and automation skills who have tested AI-powered applications are especially relevant. We are also open to Agentic Engineers or FDEs who have meaningful experience building testing frameworks, validating complex workflows, and improving software reliability
  • The ideal candidate combines the engineering mindset of an FDE or Agentic Engineer with the quality-focused approach of an experienced QA Automation Engineer or SDET

Skills

Apply

See also

QA jobs by country — openings, pay and top skills →