Conversational AI QA Specialist
Posted Updated
Summary
Remote, near-entry-level QA specialist auditing a debt-collections AI bot: daily bot health checks, disputing/confirming vendor QA scores, listening to calls and reading chat transcripts against SOPs, hunting compliance issues, and testing fixes with repeated test calls/chats. Needs QA or review experience, attention to detail, Excel, and basic LLM familiarity.
- Check bot health every morning: review dashboards and alerts, compare results over time, and report drops or anomalies.
- Audit the vendor's QA reviews: confirm or dispute each pass/fail score, correct it where needed, and note why.
- Review conversations end to end: listen to calls and read chat transcripts, check summaries and payment details, and confirm the account system was updated correctly, all against our QA manuals and SOPs.
- Catch what the rules miss: recognize when a consumer is disputing a debt or describing a hardship without saying it outright, and flag when the bot misses it. Watch for compliance issues such as identity verification, required disclosures, cease and attorney requests, bankruptcy, and active-duty military.
- Hunt for problems proactively: use filters and searches to find compliance errors, silent calls, and other problem conversations nobody flagged.
- Test fixes before they go live: write test cases, run repeated test calls and chats per scenario (typically 10), and track results in Excel.
- Join weekly calibration meetings and help decide whether issues need a manual update, calibration, or a ticket.
- Suggest improvements to the bots and our QA process, and help shape our internal QA tooling as we add automation.
- Keep things moving: communicate with the QA team, the vendor, and developers in meetings and on Teams, and follow up until issues are resolved.
- Some QA or review experience: QA of any kind, AI data annotation or evaluation, content review or moderation (for example, tagging social media or political content), or contact center quality. This role is close to entry level, and we will train you on our products and rules.
- Attention to detail: you catch the small error others miss and take pride in getting it right.
- A strong feel for language: you understand what people mean even when they don't say it directly, and can tell when a bot handled a stressed consumer badly.
- Thinking beyond pass/fail: you follow the manual and also notice what it doesn't cover.
- Organized and thorough: you follow the same steps every day without cutting corners and are willing to dive in when something looks off.
- Initiative balanced with direction: you raise ideas and issues while following the SOPs and priorities the team sets.
- Clear communication and follow-through.
- A basic understanding of large language models, or real curiosity to learn. Comfortable with Excel. You do not need to be technical.
