Machine Learning Engineer
Summary
Machine learning engineer who owns LLM evaluation end-to-end for Honcho, Plastic Labs' memory and identity layer for agents: designing evals, defining scores, building data pipelines and simulation agents, and reading traces to improve entity representations. Core stack is Python, PyTorch, Hugging Face, and Weights & Biases.
Role Overview:
- You will own the evaluation of Honcho end-to-end, which is the instrument that tells a research-driven team whether their identity representations are improving.
- Plastic Labs builds the memory and identity layer for the agentic world.
- You will operate a complex multi-agent harness with real black-box behavior.
- You will define what better means for representations that change over time without ground truth.
- You will build the machinery that measures these representations.
Key Responsibilities:
- Design the evals.
- Define the scores and decide what better means for an entity representation that changes over time.
- Keep definitions current as the product and methods move.
- Build the pipelines and harnesses.
- Implement data ingestion, labels, versions, reruns, and judges.
- Create the machinery that lets the team ask a new question and get an answer within one week.
- Run evaluations and harvest insights.
- Read traces and results to find what is actually broken rather than what is easy to measure.
- Propose fixes that produce higher-fidelity representations.
- Build simulation agents at scale.
- Increase iteration speed by raising the number of entities that can be modeled and measured at once.
- Own the loop end-to-end, including the question, the pipeline, the rerun, and the write-up.
- Execute tasks independently without waiting on other team sprints.
Requirements:
- Must have designed, built, and ran evaluations end-to-end for LLMs.
- Must have production-quality Python experience, including tests, maintenance, and release cadence.
- Must have 3+ years of experience designing and owning LLM evaluation.
- PyTorch, Hugging Face, and Weights & Biases experience.
- Familiarity with data pipeline tooling including Hydra, Kafka, or SQL is a plus.
Preferred Qualifications:
- Experience with memory, identity, personalization, agent evaluation, or agentic optimization.
- Shipped research experience as a first or co-author at NeurIPS, ICML, ICLR, or equivalent.
- Self-published open-source work of comparable quality.
- Degree in computer science, mathematics, statistics, or a related quantitative field.
Location: New York, United States
Skills required for this job:
- AI evaluation
- Data pipelines
- Hugging Face
- LLM evaluation
- Open source
- PyTorch
- Python