LLM Trainer
Posted Updated 22
views
Description
Role : LLM Trainer
About the Role :
We are looking for an LLM Trainer (Generalist) to evaluate, review, and improve the quality of responses generated by Large Language Models (LLMs). You will assess AI outputs across multiple domains such as reasoning, writing, factual accuracy, coding (basic), mathematics, and safety.
Your evaluations and annotations will directly contribute to training next-generation AI models through high-quality human feedback (RLHF/SFT), helping make AI systems more accurate, reliable, and aligned with human expectations.
Key Responsibilities :
- Evaluate AI-generated responses for correctness, completeness, relevance, and clarity.
- Compare multiple LLM responses and identify the best-performing answer.
- Apply annotation guidelines consistently across diverse tasks.
- Detect factual inaccuracies, hallucinations, logical errors, bias, and unsafe outputs.
- Provide structured feedback and reasoning for evaluations.
- Perform high-quality data labeling and annotation for LLM training datasets.
- Review prompts and model outputs across domains including :
1. General Knowledge
2. Reasoning
3. Writing
4. Math
5. Coding (basic understanding preferred)
6. Safety & Alignment
- Maintain high annotation accuracy with minimal quality control corrections.
Required Experience :
- 1-3 years of experience in one or more of :
1. LLM evaluation
2. AI data annotation
3. RLHF projects
4. Prompt engineering
5. AI quality evaluation
6. Human feedback for LLMs
7. AI content review
- OR experience working with Generative AI products like ChatGPT, Claude, Gemini, Copilot, etc., in a professional setting.
Required Skills :
- Strong English comprehension and writing skills.
- Excellent analytical and logical reasoning ability.
- Ability to compare multiple AI responses objectively.
- Strong attention to detail.
- Ability to follow annotation guidelines consistently.
- Comfortable working with structured datasets (JSON/CSV preferred).
- Basic understanding of LLMs and Generative AI.
Nice to Have :
- Experience with RLHF or SFT projects.
- Prompt engineering experience.
- Python or SQL basics.
- Familiarity with AI evaluation platforms.
- Experience with AI safety or alignment.
- Exposure to coding or mathematics evaluation.
What Success Looks Like :
- High annotation accuracy.
- Consistent judgment across evaluation tasks.
- Clear reasoning for decisions.
- Ability to identify hallucinations and factual errors.
- Minimal QC corrections.
- High productivity while maintaining quality.