Agentic AI Optimization Developer
You build evaluation frameworks for autonomous multi-agent systems and maintain their reliability in production. You curate reference datasets, automate evaluation pipelines, audit agent trajectories, tune prompts and tool calling, optimize RAG systems, and mitigate drift, prompt injection, loops, and hallucinations.
Responsibilities
- Curate and maintain Golden Sets of reference data
- Design automated continuous evaluation pipelines
- Audit multi-step agent reasoning trajectories
- Refine prompts, context windows, and few-shot examples
- Optimize tool and function calling
- Collaborate with AI developers using evaluation insights
- Monitor deployed agents for drift and hallucinations
- Implement production guardrails
- Optimize knowledge bases and RAG pipelines
Requirements
- 3+ years of professional experience in software quality engineering, test automation, or data/ML engineering
- Experience with LLM testing, prompt tuning, or orchestration patterns
- Hands-on experience with LLM orchestration frameworks such as LangGraph or ADKs
- Understanding of JSON schema design, function calling, and structured outputs
- Experience writing automated Python-heavy test scripts
- Experience with tracing and observability for asynchronous systems
- Experience with AI evaluation and observability platforms preferred
- Production experience testing Agentic workflows and GenAI solutions preferred
- Familiarity with Google Cloud AI and UiPath preferred
- Proficiency in Python or TypeScript
- Understanding of asynchronous programming, API design, and microservices architecture
Benefits
- Healthcare packages
- Paid time off