Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Brilliant Systems

ML Evaluation Architect & Quality Harness Engineer

Posted Updated

Description

Brilliant Systems is seeking an engineer to own the evaluation of ML/LLM systems, turning client requirements into measurable evaluation sets and robust tooling. You will build harnesses, monitor production drift, and ensure evaluation is integral to the product lifecycle.

You must demonstrate statistical literacy, strong Python skills, and a knack for developer-friendly tooling, while candidly reporting results that may be inconvenient.

Turn client requirements into measurable evaluation sets and adversarial cases. Build evaluation harnesses and tooling so engineers can run and read them. Track drift in production and raise it early before clients detect it. Contribute to BotUp where evaluation is part of the product, not just internal tooling. Push back early on unmeasurable scope while changes are inexpensive. You have built evaluation for a machine learning or LLM system that others relied on. You understand what small samples can and cannot tell you. You write code and build tooling that engineers actually adopt. You are willing to report results that nobody wants to hear.

Skills

Apply

See also

Architecture jobs by country — openings, pay and top skills →