Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Argyll Scott

Global Life Insurer - Asia AI Quality Evaluation Engineer

Posted Updated

Description

Argyll Scott in Hong Kong is partnering with a global life insurer to establish a regional AI Centre of Excellence. The AI Quality & Evals Engineer will help build evaluation infrastructure, monitor live AI activity and intervene when needed, spanning governance, safety and cost controls.

You will work hands-on on evaluation frameworks, streaming monitors, and incident reporting, collaborating with Risk, Security and Legal teams, contributing to robust, auditable AI systems across Asia.

Build evaluation frameworks using reference datasets, production failure cases, synthetic tests and versioned evaluation criteria. Design and calibrate LLM-based judges against human assessments, monitoring their consistency as models evolve. Test agent-specific failure modes, including fabricated tool arguments, retrieval failures, uncontrolled loops and unsafe actions. Develop adversarial and red-team suites covering prompt injection, data-exfiltration attempts and privilege escalation. Integrate regression evaluations into release pipelines for changes to prompts, models, tools, retrieval and policies. Build streaming evaluations and risk-based monitors for quality, task success, latency, cost and behavioural drift. Implement guardrails, intervention thresholds, kill switches and safe interruption mechanisms. Automate evaluation results, incidents, asset inventories and control evidence into governance platforms, partnering with Risk, Security, Legal and Internal Audit. Share operational responsibility for the agent control platform, including on-call response and decisions to constrain or halt agent activity. 4+ years in software, ML or data engineering. 2+ years focused on production LLM or agent quality, evaluation or safety.

Skills

Apply

See also