AI Quality Engineer (GenAI / RAG / Agentic AI) - Pheonix, AZ - Contract
Description
We at NJ TECH INC are focused on hiring highly skilled professionals who are excited by the opportunity to make a true impact on their careers as well as on our clients' business. We power our clients’ success—and drive our consultants’ career growth.
We are seeking an experienced and outstanding AI Quality Engineer (GenAI / RAG / Agentic AI) for one of our esteemed clients.
Role: AI Quality Engineer (GenAI / RAG / Agentic AI)
Work Location: Pheonix, AZ
Work Mode: Onsite
Employment Type: Long-Term Contract
- Design and implement end-to-end testing strategies for LLM, RAG, and Agentic AI applications.
- Build automated evaluation frameworks to measure answer correctness, relevance, groundedness, faithfulness, hallucination, retrieval quality, tool selection, and agent task completion.
- Develop and maintain golden datasets, benchmark suites, and prompt/model regression test packs.
- Test RAG pipelines covering document ingestion, chunking, embeddings, vector retrieval, reranking, context assembly, and response generation.
- Validate multi-step agent workflows, including planning, reasoning, tool execution, memory, exception handling, guardrails, and human-in-the-loop controls.
- Perform negative, boundary, adversarial, prompt-injection, jailbreak, and sensitive-data leakage testing.
- Validate APIs, microservices, enterprise integrations, and downstream systems accessed by AI agents.
- Automate functional, API, integration, and end-to-end testing using Python, PyTest, Playwright, Selenium, and REST API frameworks.
- Integrate AI evaluations and automated tests into CI/CD pipelines with actionable reporting and quality gates.
- Monitor production quality across accuracy, latency, reliability, token consumption, and cost.
- Identify and troubleshoot failures across prompts, retrieval, data, models, orchestration, tools, and application code.
- Strong understanding of LLMs, Generative AI, RAG, embeddings, vector search, and Agentic AI.
- Experience evaluating non-deterministic AI outputs and defining acceptance thresholds.
- Knowledge of RAG evaluation, prompt regression, hallucination detection, grounding, and agent/tool-call validation.
- Ability to create representative golden datasets and benchmark suites.
- Strong hands-on experience with Python and PyTest.
- REST API testing and automation using Playwright or Selenium.
- Experience building reusable test automation frameworks.
- Experience integrating automated tests with Git-based CI/CD pipelines.
- Working knowledge of SQL, microservices, Docker, and Kubernetes.
- Hands-on exposure to DeepEval, RAGAS, Giskard, LangSmith, Langfuse, or custom Python evaluation frameworks.
- Understanding of LLM-as-a-Judge, deterministic checks, human evaluation, tracing, and production observability.
- Experience with Azure OpenAI / Azure AI Foundry, Amazon Bedrock, or Google Vertex AI.
- Working knowledge of vector databases, model endpoints, identity and access controls, secrets management, and cloud monitoring.
- Response Quality: Answer relevance, correctness, faithfulness, groundedness, and hallucination rate.
- RAG Quality: Context precision/recall, Retrieval Precision@K and Recall@K, and citation accuracy.
- Agent Quality: Task-success rate, tool-call accuracy, workflow completion, and recovery rate.
- Non-functional: P95 latency, availability, throughput, token usage, and cost per transaction.
- Safety: Prompt-injection resistance, jailbreak success rate, toxicity, bias, and PII leakage.
- Regression: Automated pass rate, model/prompt variance, and escaped defect rate.
- Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, Data Science, or a related discipline.
- Experience testing production GenAI or RAG applications in regulated enterprise environments such as banking, insurance, or healthcare.
- Working knowledge of LangChain, LangGraph, Semantic Kernel, AutoGen, CrewAI, or comparable orchestration frameworks.
- Exposure to Responsible AI, model risk management, security testing, and AI governance.
- Relevant cloud, testing, security, or AI certifications.
- Reduced AI defects escaping into production.
- Measurable improvement in grounding, retrieval, and answer quality.
- High automated regression coverage across prompts, models, RAG pipelines, and agent workflows.
- Reliable AI quality gates embedded in delivery pipelines.
- Reduced manual validation effort and faster release cycles without compromising safety or reliability.
NJ TECH INC is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, We cannot sponsor any visa at this time
Skills
- Agentic AI
- AI
- API
- API Testing
- Assembly
- AutoGen
- Automation
- AWS Bedrock
- Azure
- CI/CD
- Cloud
- CrewAI
- Data Science
- Docker
- Embeddings
- Generative AI
- Git
- Kubernetes
- LangChain
- Langfuse
- LangGraph
- LangSmith
- LLM
- Microservices
- Observability
- OpenAI
- Playwright
- pytest
- Python
- RAG
- REST
- Secrets Management
- Selenium
- Semantic Kernel
- SQL
- Test Automation
- Vector Databases
- Vector Search
- Vertex AI
Sign in to read and join the discussion.