Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Qentelli

NewBe an early applicant

AI Engineer

Posted Updated

Description

Job Summary

We are seeking an experienced AI Quality Engineering / AI Test Lead with expertise in testing AI/ML, LLM-based, and Agentic AI systems. The ideal candidate will be responsible for validating the reliability, performance, scalability, safety, and compliance of AI-driven applications and multi-agent architectures. This role involves designing robust test frameworks, evaluating LLM outputs, implementing AI guardrails, and ensuring production-grade quality standards across distributed AI ecosystems.


Roles and Responsibilities

  • Design, develop, and execute comprehensive test strategies for AI/ML, LLM, and agentic AI applications.
  • Build and maintain automated test harnesses, simulators, fixtures, and validation frameworks using Python.
  • Evaluate LLM performance using metrics such as semantic similarity, BLEU, ROUGE, BERTScore, and exact/soft matching techniques.
  • Validate AI guardrails, prompt reliability, content moderation policies, and model safety mechanisms.
  • Perform end-to-end testing of multi-agent workflows, orchestration frameworks, and AI decision-making systems.
  • Conduct distributed systems testing, including latency profiling, resilience validation, failure recovery, and chaos engineering.
  • Implement and optimize CI/CD pipelines for AI testing and deployment validation.
  • Leverage observability tools to monitor AI application behavior, performance, and production readiness.
  • Collaborate with Engineering, Data Science, Operations, and Product teams to define quality standards and release criteria.
  • Ensure compliance with privacy, security, and regulatory requirements related to AI systems, including PII handling and responsible AI practices.
  • Drive continuous improvements in AI testing methodologies, automation frameworks, and evaluation pipelines.

Mandatory Skills

  • 6+ years of experience in AI Development with at least 2+ years focused on AI/ML, Generative AI, or LLM-based systems.
  • Strong expertise in testing Agentic AI and Multi-Agent architectures.
  • Advanced programming skills in Python, including experience building test automation frameworks, simulators, and fixtures.
  • Hands-on experience with LLM evaluation methodologies such as BLEU, ROUGE, BERTScore, semantic similarity, exact/soft matching, prompt testing, and guardrails validation.
  • Expertise in distributed systems testing, latency profiling, resiliency testing, chaos engineering, retries, circuit breakers, and message queue validation.
  • Experience with AI orchestration frameworks such as LangChain, LangGraph, LlamaIndex, DSPy, OpenAI Assistants/Actions, or Azure OpenAI.
  • Strong knowledge of CI/CD pipelines using GitHub Actions, Azure DevOps, or similar platforms.
  • Experience with observability and monitoring tools including OpenTelemetry, Prometheus, Grafana, and Datadog.
  • Understanding of feature flags, canary deployments, and production release validation strategies.
  • Good understanding of AI security, privacy, governance, compliance, and model safety requirements.
  • Strong communication, stakeholder management, and cross-functional collaboration skills.

Preferred Skills

  • Experience with multi-agent simulators, agent graph validation, and tooling latency emulation.
  • Knowledge of MLOps practices including model versioning, dataset management, evaluation pipelines, and model lifecycle management.
  • Experience implementing A/B testing and experimentation frameworks for LLM applications.
  • Strong background in AWS cloud services, serverless technologies, containerization, and event-driven architectures.
  • Experience managing production AI workloads with ownership of cost optimization, latency reduction, scalability, and SLA adherence.
  • Familiarity with Generative AI evaluation frameworks and Responsible AI best practices.
  • Exposure to Kubernetes, Docker, and cloud-native application testing.

Skills

Apply

See also

AI Engineering jobs by country — openings, pay and top skills →