Chat Support Executive

Job Overview

The Chatbot Log Quality Analyst systematically reviews, validates, and analyses chatbot conversation logs to support accurate, compliant, and customer-focused responses. The role identifies content gaps, incorrect or outdated guidance, hallucinations, policy issues, broken journeys, and improvement opportunities, and translates findings into actionable quality insights for AI, Content, and Operations teams.

Job Description

Chat Log Review & Quality Assurance

• Review daily and weekly chatbot conversation logs across assigned routes.
• Validate responses for accuracy, relevance, completeness, and tone against approved knowledge sources.
• Identify and flag hallucinations, misleading answers, and incorrect policy or process guidance.
• Detect partial or outdated information, broken flows, fallback failures, and looping responses.
• Escalate high-risk inaccuracies and compliance concerns through the defined review process.
• Maintain consistent review coverage and evidence-based quality remarks.
• Apply route, intent, and issue context when assessing each conversation.
• Protect applicants and organizational data while reviewing logs.

Annotation, Feedback & Documentation

• Categorize conversations using defined quality tags such as Correct, Incorrect, Partial, Hallucination, and UX Issue.
• Provide clear, actionable feedback for knowledgebase and website content updates.
• Recommend UI, UX, and conversational journey corrections based on observed failures.
• Maintain structured quality logs, reviewer notes, and supporting evidence for audit purposes.
• Apply consistent annotation standards and participate in calibration reviews.
• Document the root cause and expected correction for recurring issues.
• Coordinate clarifications with AI, Content, Operations, and relevant process owners.
• Support validation of fixes through targeted rechecks and regression review.

Reporting, Insights & Continuous Improvement

• Prepare periodic reports covering accuracy percentage, error types, and hallucination rate.
• Identify top failing intents, recurring applicant queries, and customer sentiment patterns.

Highlight issue trends and improvement opportunities to AI, Content.

Additional Responsibilities (Special Projects)

• Support chatbot UAT, quality calibration, taxonomy refinement, knowledge-base validation, route-level deep dives, and other AI quality initiatives assigned by the reporting manager.

Education

• Graduate in any discipline from a recognized university.

Experience

• 2-4 years of experience in quality assurance, operations, customer service, content review, chat transcript review, or conversational data analysis.
• Experience with chatbots, AI quality review, Excel-based reporting, or regulated customer journeys is preferred.