ML Evaluation Architect & Quality Harness Engineer
Posted Updated
Description
Brilliant Systems is seeking an engineer to own the evaluation of ML/LLM systems, turning client requirements into measurable evaluation sets and robust tooling. You will build harnesses, monitor production drift, and ensure evaluation is integral to the product lifecycle.
You must demonstrate statistical literacy, strong Python skills, and a knack for developer-friendly tooling, while candidly reporting results that may be inconvenient.
Turn client requirements into measurable evaluation sets and adversarial cases. Build evaluation harnesses and tooling so engineers can run and read them. Track drift in production and raise it early before clients detect it. Contribute to BotUp where evaluation is part of the product, not just internal tooling. Push back early on unmeasurable scope while changes are inexpensive. You have built evaluation for a machine learning or LLM system that others relied on. You understand what small samples can and cannot tell you. You write code and build tooling that engineers actually adopt. You are willing to report results that nobody wants to hear.Skills
Sign in to read and join the discussion.