Gen AI evaluators (Data Analyst)
Summary
Evaluates and improves Gen AI responses by analyzing performance metrics, curating knowledge bases, and refining data sets for accuracy and relevance.
Gen AI evaluators (Data Analyst)
Job Requirement:
- Identify key performance metrics and evaluate the success of Gen AI initiatives.
- Basic understanding of prompt engineering, data pre-processing, and data ingestion techniques in the context of Generative AI systems. (Good to have)
- Proficient in MS Office (Word, Excel, Project and PowerPoint)
- Build and curate a comprehensive knowledge base to serve as a foundation for the Generative AI model.
- Evaluate and analyse AI-generated responses to identify areas of improvement and optimize accuracy.
- Identify, analyse, and assess the quality of AI-generated responses, ensuring they align with company standards and customer expectations.
- Collaborate with the AI Development team to implement necessary adjustments to improve response quality.
- Deconflict and curate data sets across various platforms to ensure a diverse and clean input for the Generative AI model.
- Regularly update and refine data sets to reflect evolving language trends and user preferences.
- Perform daily updates to the knowledge base, integrating the latest product information, industry updates, and relevant content.
- Ensure accuracy and relevance of information provided by collaborating with product owners and subject matter experts.
- Engage in call listening activities to identify emerging demand topics and customer preferences.
- Provide insights and recommendations to the AI Project Manager for enhancing AI-generated responses in response to these demands.
- Establish effective communication channels with product owners to continuously enhance the knowledge base and refine AI responses.
- Actively gather feedback and insights from product owners to improve the accuracy and relevance of AI interactions.