AI Engineer
Posted Updated
Description
Job Summary
We are seeking an experienced AI Quality Engineering / AI Test Lead with expertise in testing AI/ML, LLM-based, and Agentic AI systems. The ideal candidate will be responsible for validating the reliability, performance, scalability, safety, and compliance of AI-driven applications and multi-agent architectures. This role involves designing robust test frameworks, evaluating LLM outputs, implementing AI guardrails, and ensuring production-grade quality standards across distributed AI ecosystems.
Roles and Responsibilities
- Design, develop, and execute comprehensive test strategies for AI/ML, LLM, and agentic AI applications.
- Build and maintain automated test harnesses, simulators, fixtures, and validation frameworks using Python.
- Evaluate LLM performance using metrics such as semantic similarity, BLEU, ROUGE, BERTScore, and exact/soft matching techniques.
- Validate AI guardrails, prompt reliability, content moderation policies, and model safety mechanisms.
- Perform end-to-end testing of multi-agent workflows, orchestration frameworks, and AI decision-making systems.
- Conduct distributed systems testing, including latency profiling, resilience validation, failure recovery, and chaos engineering.
- Implement and optimize CI/CD pipelines for AI testing and deployment validation.
- Leverage observability tools to monitor AI application behavior, performance, and production readiness.
- Collaborate with Engineering, Data Science, Operations, and Product teams to define quality standards and release criteria.
- Ensure compliance with privacy, security, and regulatory requirements related to AI systems, including PII handling and responsible AI practices.
- Drive continuous improvements in AI testing methodologies, automation frameworks, and evaluation pipelines.
Mandatory Skills
- 6+ years of experience in AI Development with at least 2+ years focused on AI/ML, Generative AI, or LLM-based systems.
- Strong expertise in testing Agentic AI and Multi-Agent architectures.
- Advanced programming skills in Python, including experience building test automation frameworks, simulators, and fixtures.
- Hands-on experience with LLM evaluation methodologies such as BLEU, ROUGE, BERTScore, semantic similarity, exact/soft matching, prompt testing, and guardrails validation.
- Expertise in distributed systems testing, latency profiling, resiliency testing, chaos engineering, retries, circuit breakers, and message queue validation.
- Experience with AI orchestration frameworks such as LangChain, LangGraph, LlamaIndex, DSPy, OpenAI Assistants/Actions, or Azure OpenAI.
- Strong knowledge of CI/CD pipelines using GitHub Actions, Azure DevOps, or similar platforms.
- Experience with observability and monitoring tools including OpenTelemetry, Prometheus, Grafana, and Datadog.
- Understanding of feature flags, canary deployments, and production release validation strategies.
- Good understanding of AI security, privacy, governance, compliance, and model safety requirements.
- Strong communication, stakeholder management, and cross-functional collaboration skills.
Preferred Skills
- Experience with multi-agent simulators, agent graph validation, and tooling latency emulation.
- Knowledge of MLOps practices including model versioning, dataset management, evaluation pipelines, and model lifecycle management.
- Experience implementing A/B testing and experimentation frameworks for LLM applications.
- Strong background in AWS cloud services, serverless technologies, containerization, and event-driven architectures.
- Experience managing production AI workloads with ownership of cost optimization, latency reduction, scalability, and SLA adherence.
- Familiarity with Generative AI evaluation frameworks and Responsible AI best practices.
- Exposure to Kubernetes, Docker, and cloud-native application testing.
Skills
- A/B Testing
- Agentic AI
- AI
- Automation
- AWS
- Azure
- Azure DevOps
- CI/CD
- Cloud
- Cloud Native
- Containerization
- Data Science
- Datadog
- DevOps
- Distributed Systems
- Docker
- DSPy
- Event Driven Architecture
- Generative AI
- GitHub
- GitHub Actions
- Grafana
- Kubernetes
- LangChain
- LangGraph
- LlamaIndex
- LLM
- Machine Learning
- MLOps
- Observability
- OpenAI
- OpenTelemetry
- Prometheus
- Python
- Serverless
- Stakeholder Management
- Test Automation
Sign in to read and join the discussion.