Senior Associate/Assistant Vice President, Agent Assurance Engineer
Posted Updated
Description
Experience and background
- 4–7 years of experience in AI/ML engineering, AI safety research, AI quality assurance, or a related technical discipline, with hands-on experience evaluating, testing, or governing LLM or agentic AI systems.
- Demonstrated expertise in at least two of the following: LLM evaluation frameworks; prompt engineering and context architecture for production agents; AI red-teaming and adversarial testing; or AI system monitoring and observability.
- Experience with production AI systems in regulated or high-stakes environments is strongly preferred, particularly where governance and risk management requirements are significant.
- Familiarity with AI safety and alignment research is advantageous, including RLHF and its limitations, constitutional AI, interpretability research, and agent safety evaluation methodologies.
Technical capabilities
- LLM and agent tooling: Deep familiarity with Anthropic Claude and OpenAI APIs (including tool use, structured outputs, and system prompt design); experience with orchestration frameworks such as LangChain, LangGraph, and AutoGen.
- Evaluation frameworks: Hands-on experience with LLM evaluation tools (e.g. LangSmith Evals, RAGAS, DeepEval, or equivalent); ability to design evaluation datasets, scoring rubrics, and interpret results with statistical rigour.
- Monitoring and observability: Experience with AI observability platforms (e.g. LangFuse, Helicone, Weights & Biases); ability to implement structured logging and behavioural anomaly detection.
- Programming: Strong Python proficiency for evaluation scripting, data analysis, and automation; familiarity with pandas, numpy, and related analytical libraries.
Mindset and approach
- Adversarial thinker: Proactively probes systems for weaknesses, thinks like an attacker, and validates that controls work in practice—not just on paper.
- Rigorous and systematic: Applies a scientific approach to evaluation, with clear hypotheses, controlled testing, reproducible results, and objective reporting of findings.
- Collaborative assurance partner: Partners effectively with AI engineers and product managers to improve agent quality and safety while enabling delivery.
- Continuous learner: Keeps pace with emerging developments in AI evaluation and agent safety, translating research advances into practical production improvements.
Skills
Sign in to read and join the discussion.