Senior AI Engineer, Agent Evaluation and Governance

Niuro is hiring a Senior AI Engineer for an enterprise AI platform company where AI agents take real actions: sending messages, updating records and preparing decisions. You will own the evaluation and governance layer that decides when an agent capability is ready for production.

You will:
- Build an evaluation framework with datasets, metrics, thresholds and a go-live gate per agent capability.
- Re-evaluate capabilities after every model or instruction change and catch regressions.
- Build a governed agent runtime: accountable owners, human approval where needed and deterministic checks before irreversible actions.
- Size review samples statistically and configure automatic suspension when quality drops.
- Log every model call with model, instruction version, data provenance and checks.
- Keep the platform independent from any single model provider.

You bring:
- Production experience with tool-using, multi-step LLM agents.
- Evaluation sets, metrics, quality thresholds and regression detection in production.
- Strong applied statistics: sample sizing, confidence bounds, error-rate estimation.
- Safety controls enforced in code, such as approvals and kill switches.
- Strong Python or TypeScript, with testing and code review.
- C1 English.

Nice to have: knowledge graphs, ontologies, RAG, human-in-the-loop systems.

Remote from Latin America, full-time contractor.

Location: Remote - Argentina, Bolivia, Brazil, Chile, Colombia, Costa Rica, Cuba, Dominican Republic, Ecuador, El Salvador, Guatemala, Honduras, Mexico, Nicaragua, Panama, Paraguay, Peru, Uruguay, Venezuela

Skills required for this job:

• AI agent orchestration
• AI agents
• Knowledge graph
• Large Language Models
• Observability
• Python
• Software testing
• TypeScript