AI Evaluation Engineer
Responsibilities
- Develop and maintain AI evaluation frameworks, regression test suites and automated evaluation pipelines.
- Build datasets, quality dashboards and evaluation tools to improve the performance of AI agents and LLM-powered applications.
- Design evaluation methods such as LLM-as-a-Judge, multi-turn conversations, tool evaluation and agent workflow analysis.
- Integrate AI evaluation into CI/CD and model release processes.
- Analyse production issues and improve datasets, prompts, evaluation logic and AI workflows.
- Conduct experiments to optimise model accuracy, retrieval performance, latency and cost.
- Build reusable platform capabilities and developer tools to support multiple AI products.
- Prepare technical documentation and collaborate with cross-functional engineering teams.
Requirements
- Bachelor's Degree in Computer Science, Software Engineering or a related discipline.
- Minimum 2 years of relevant experience in AI Platform Engineering, LLMOps, Machine Learning Engineering or Software Engineering.
- Experience with most of the following technologies: LangSmith, LangGraph / LangChain, FastAPI, Temporal, Grafana, Redis, React / TypeScript, EvalsHub or similar AI evaluation tools
- Experience with LLM evaluation, Retrieval-Augmented Generation (RAG), semantic search, embeddings or model benchmarking is preferred.
- Strong analytical and problem-solving skills.
- Good communication and stakeholder management skills.
- Able to work independently and collaboratively in a fast-paced environment.
Interested candidates are encouraged to submit their resumes outlining their relevant experience and achievements to sherriyang(@)talentvis.com or click apply!
We regret to inform that only shortlisted candidates would be notified