Senior QA Engineer - Azure AI
Summary
Senior QA Engineer designs and automates tests for AI/LLM systems on Azure, building Python frameworks and CI/CD pipelines to validate reliability, correctness, and safety of GenAI deployments.
We are seeking a highly skilled Senior QA Engineer specializing in Azure AI to join our client's team in Doha (on-site). This role focuses on ensuring the reliability, accuracy, and safety of AI/LLM systems deployed on Azure, with a strong emphasis on evaluation frameworks, automation, and modern GenAI architectures.
Location: Doha, Qatar - On-site
Role Type: Full Time, Permanent
Language Required: English
- Design and execute QA strategies for LLM, Generative AI, and RAG based systems, ensuring reliability, correctness, and stability.
- Build automated evaluation pipelines using Azure DevOps, integrating model testing into CI/CD workflows.
- Develop advanced Python based testing frameworks, including Pandas workflows, PyTest suites, and custom evaluation scripts.
- Validate AI behaviours using both deterministic assertions and LLM as a Judge evaluation methodologies.
- Collaborate with data engineering and platform teams to maintain evaluation datasets, including ground truth / golden datasets.
- Test and validate Azure AI deployments end to end, including Azure AI Studio, Azure OpenAI, Azure AI Search, and related services.
- Contribute to DataOps processes and continuous monitoring of AI system performance, regression, and drift.
- Support experimentation workflows, ensuring reproducibility, traceability, and quality across multiple model versions.
- Stay updated on emerging GenAI methodologies, including agentic architectures, multi agent workflows, and evaluation best practices.
- 1-2 years of hands on Generative AI experience.
- Practical expertise with RAG architectures (Vector Search + LLM).
- 5+ years of Azure experience.
- Proven deployment/testing experience in Azure AI Studio or Azure OpenAI Service (not just local testing).
- 7+ years of Python experience.
- Ability to write complex, production quality code (Pandas, PyTest, automation frameworks).
- 8+ years of QA experience.
- Strong understanding of LLM evaluation metrics and frameworks.
- 5+ years working with DataOps or CI/CD pipelines.
- Direct experience with Azure DevOps pipelines tied to AI evaluations.
- Technical degree in Computer Science, Engineering, Mathematics, or equivalent professional experience.
- Experience with Agentic AI (LangGraph, AutoGen) or multi-agent systems.
- Experience with Azure AI Search / Cognitive Search including indexing and vector optimization.
- Experience with LangChain or Semantic Kernel.
- Experience building evaluation pipelines using Prompt Flow.
- Experience creating and managing Golden Datasets.
- Master's degree in Data Science or AI related field.
- Experience fine tuning SLMs (e.g., Phi 3, Llama).
- Familiarity with Azure ML, Workspaces, and Compute management.
- Background in C#/.NET for backend integration.
- Exposure to Red Teaming and adversarial testing.
- Experience with MLflow.
- Certification: Azure AI Engineer Associate (AI 102).