Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Management Solutions International (MSI)

NewBe an early applicant

Artificial Intelligence Engineer

Posted

Description

Senior AI Evaluation Engineer – LLMs & Agentic AI

Location: Qatar

Experience: 7+ Years

Education: Master's Degree or Higher in Computer Science, Software Engineering or a related Engineering discipline.

About the Role

We are looking for an experienced Senior AI Evaluation Engineer to lead the evaluation, testing, validation and continuous improvement of advanced AI and Large Language Model (LLM) systems.

This is not a conventional Machine Learning Engineer or Generative AI Developer position focused primarily on building and deploying AI applications.

The successful candidate will specialize in determining whether AI systems are actually performing correctly, reliably and consistently in real-world production environments.

You will design and implement evaluation frameworks, benchmark datasets, automated testing pipelines and statistical validation methodologies to measure the quality, accuracy, factual reliability and performance of LLM-powered applications and agentic AI systems.

Key Responsibilities

1. LLM & Generative AI Evaluation

  • Design and implement comprehensive evaluation frameworks for Large Language Models (LLMs), Generative AI and RAG-based applications.
  • Develop evaluation methodologies to measure factual accuracy, relevance, hallucinations, contextual understanding, consistency and response quality.
  • Implement and validate LLM-as-a-Judge approaches for automated evaluation and scoring.
  • Evaluate different LLM models, prompts and configurations through structured benchmarking and comparative testing.

2. Evaluation Frameworks & Benchmarking

  • Build and maintain automated evaluation pipelines using frameworks such as RAGAS, DeepEval, TruLens, OpenAI Evals, LangSmith, Arize Phoenix or equivalent tools.
  • Design evaluation datasets, test cases, ground-truth benchmarks and scoring rubrics.
  • Develop repeatable testing processes for assessing model performance across different use cases.
  • Establish evaluation metrics and quality thresholds to determine whether AI systems are ready for production deployment.

3. Statistical Analysis & Experimental Validation

  • Apply statistical methods to assess model performance and reliability.
  • Design and conduct controlled experiments, A/B testing and comparative model evaluations.
  • Perform hypothesis testing, statistical significance analysis and confidence interval calculations.
  • Analyse evaluation results to identify performance gaps, model weaknesses and opportunities for improvement.
  • Apply human


Skills

Apply

See also