Senior Product Manager, Conversational AI Chatbot & Agent Quality
Summary
Own and improve a production conversational AI chatbot and agent system, defining quality metrics, evaluation frameworks, and human-in-the-loop workflows to boost accuracy and user satisfaction.
You will build and operate conversational AI products in production, focusing on knowledge quality, agent evaluation, chatbot operations, observability, and human-in-the-loop workflows. You will define evaluation frameworks, improve dialogue flows and retrieval, analyze failures, manage quality metrics, and partner with engineering on runtime and debugging capabilities.
Responsibilities
- Own knowledge base structure, content quality, retrieval coverage, and freshness governance
- Translate business processes into agent flows with edge-case handling and escalation paths
- Own annotation specifications, annotator quality assurance, and training batch impact tracking
- Review logs, triage failures, and maintain knowledge and flow updates
- Own resolution rate, fallback rate, per-intent accuracy, and CSAT metrics
- Define agent evaluation frameworks, test cases, automated scoring criteria, and regression coverage
- Operate the quality feedback loop from harness results to prioritized fixes, re-evaluation, and production deployment
- Define agent runtime requirements including observability, tool-call monitoring, failure alerts, and debugging tools
- Design human-in-the-loop case routing, reviewer interfaces, and correction data capture
- Track agent-version performance and maintain evaluation dashboards
Requirements
- 3–6 years of product management experience
- At least 2 years as the primary owner of a production chatbot or AI agent product
- Quantified business results with baseline metrics and numerical outcomes
- Hands-on knowledge base, labeling, and conversation analysis experience
- Familiarity with at least one chatbot or agent platform such as Coze, Dify, or Dialogflow
- Fluency in Mandarin Chinese
- English proficiency
- Agent evaluation harness design is a nice-to-have
- Internal agent platform product design is a nice-to-have
- LLM-as-judge evaluation experience is a nice-to-have
- Agent observability tooling familiarity is a nice-to-have
- Regression testing experience for non-deterministic systems is a nice-to-have
- Human-in-the-loop workflow specification experience is a nice-to-have
- Customer service, operations, or financial services background is a nice-to-have
Benefits
- L&D programs and education subsidy
- Team building programs and company events
- Wellness and meal allowances
- Comprehensive healthcare schemes for employees and dependants