Client-Facing Language Model Engineer
Posted Updated
Description
Artificial Analysis is seeking a Forward Deployed Engineer to own the language model benchmarking stack, onboard new models, run evaluations, and serve as the technical face for AI labs and enterprise customers.
You’ll operate a Python-based pipeline, debug issues across the stack, and collaborate with engineering to refine processes while staying current with model releases and industry changes. Melbourne on-site with SF collaboration opportunities.
Operate and maintain the Python-based language model benchmarking pipeline end-to-end: onboard new models, configure evaluations, run benchmarks and validate results. Debug issues across the stack: API timeouts, errors and unexpected outputs; resolve quickly. Serve as the primary technical contact for AI lab customers: explain results, answer methodology questions and troubleshoot integration via Slack and video conferencing. Monitor benchmark runs for anomalies and ensure accuracy and integrity of published results. Maintain documentation of processes, known issues, and model-specific configurations. Collaborate with the engineering team to flag pipeline improvements and contribute to refinements. Stay current with new model releases, API changes, and developments across the language model ecosystem. 3+ years in a client-facing technical role (solutions engineering, support engineering, technical consulting, or similar). Strong Python proficiency and ability to work with large codebases you didn’t write. Hands-on experience with AI/ML model APIs (OpenAI, Anthropic, Google, Meta, etc.). Excellent debugging skills; trace issues across APIs, data pipelines and code. EquitySkills
Sign in to read and join the discussion.