Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Management Solutions International

New

Senior LLM Evaluation Engineer / Applied AI Engineer

Posted Updated

Description

Senior LLM Evaluation Engineer / Applied AI Engineer

Experience: 7–12 Years

Education: Master's Degree or Higher




Location : Doha



About the Role

We are looking for an experienced Senior LLM Evaluation Engineer / Applied AI Engineer to develop and implement evaluation frameworks for AI-powered applications, Large Language Models (LLMs) and autonomous AI agents.




The role focuses on measuring, testing and improving the quality, accuracy, reliability and performance of AI systems in production.




The ideal candidate will combine strong Machine Learning and AI engineering expertise with advanced Python programming, statistical analysis and hands-on experience in LLM evaluation.




You will be responsible for developing automated evaluation pipelines, designing benchmarking frameworks, implementing LLM-as-a-Judge methodologies and conducting statistical experiments to assess and improve AI system performance.




This is an opportunity to work on advanced Generative AI and Agentic AI systems, ensuring that AI applications deliver accurate, relevant, reliable and consistent results.



Key Responsibilities


  • Design, develop and maintain evaluation frameworks for Large Language Models (LLMs) and Generative AI applications.






  • Build automated evaluation pipelines and testing systems using Python.






  • Develop evaluation datasets, benchmarks, ground-truth datasets and annotation frameworks.






  • Implement LLM-as-a-Judge methodologies to assess AI-generated responses.






  • Evaluate AI systems for factual accuracy, relevance, hallucinations, consistency and response quality.






  • Conduct A/B testing and comparative analysis of different models, prompts and AI configurations.






  • Apply statistical methods, hypothesis testing and confidence intervals to validate model improvements.






  • Evaluate AI agents, tool calling, function calling and multi-step agent workflows.






  • Develop offline and online evaluation methodologies for production AI systems.






  • Monitor model performance and identify opportunities to improve AI system quality, reliability and efficiency.






  • Collaborate with ML Engineers, Data Scientists and AI Researchers to improve model performance and evaluation processes.





Required Technical Skills

Programming & Software Engineering






  • Advanced Python programming






  • Object-Oriented Programming






  • Python-based automation and evaluation pipelines






  • NumPy, Pandas and SciPy






Machine Learning & Generative AI






  • Machine Learning and Deep Learning






  • Large Language Models (LLMs)






  • Generative AI






  • GPT / OpenAI APIs






  • RAG (Retrieval-Augmented Generation)






  • PyTorch / TensorFlow / Scikit-learn






LLM Evaluation & Benchmarking






  • LLM Evaluation






  • LLM-as-a-Judge






  • Model Evaluation and Benchmarking






  • Automated and Human Evaluation






  • Offline and Online Evaluation






  • Hallucination Detection






  • RAG Evaluation






  • Evaluation Frameworks such as RAGAS, DeepEval, TruLens and LangSmith






Statistical Analysis & Experimentation






  • A/B Testing






  • Hypothesis Testing






  • Statistical Significance






  • Confidence Intervals






  • Experimental Design






  • Statistical Analysis






  • Inter-Rater Agreement






AI Agents & Prompt Engineering






  • Agentic AI






  • AI Agent Evaluation






  • Tool Calling and Function Calling






  • Prompt Engineering and Optimisation






  • LangChain / LlamaIndex / Semantic Kernel






Cloud & Data Technologies






  • Microsoft Azure






  • Azure OpenAI






  • Azure AI Services






  • PostgreSQL






  • Elasticsearch






  • SQL





Experience & Qualifications


  • 7–12 years of relevant experience in Machine Learning Engineering, Applied AI, AI Research or Generative AI Engineering.






  • Master's degree or higher in Computer Science, Artificial Intelligence, Machine Learning, Data Science, Statistics, Applied Mathematics or a related discipline.






  • Strong hands-on Python programming and software engineering capabilities.






  • Proven experience developing and evaluating ML/LLM-based applications.






  • Practical experience designing evaluation datasets, benchmarking systems and automated evaluation pipelines.






  • Strong understanding of statistical experimentation, A/B testing and hypothesis testing.






  • Experience with LLM evaluation methodologies, including LLM-as-a-Judge and human evaluation.






  • Exposure to production AI systems, RAG applications and AI agent architectures.






  • Experience with Microsoft Azure and cloud-based AI services is preferred.





Ideal Candidate Profile

The ideal candidate will have experience as a Senior Machine Learning Engineer, LLM Engineer, Applied Scientist, AI Research Engineer or Applied AI Engineer, with a strong specialisation in evaluating and improving AI/LLM systems.




We are particularly interested in professionals who can combine hands-on AI engineering with scientific evaluation methodologies to identify model weaknesses, detect hallucinations, benchmark performance and establish measurable improvements in AI system reliability.




Experience with autonomous AI agents, LLM observability, evaluation frameworks and production-grade Generative AI applications will be highly advantageous.




Skills

What Senior AI Engineering jobs ask for — and how much of it you have →
Apply

See also

AI Engineering jobs by country — openings, pay and top skills →