Member of Technical Staff (Answer Quality & Evals)
Posted Updated
- Build the systems and pipelines that enable Search, Product, and other teams to independently access and utilize reliable eval verdicts without bottlenecks
- Take ownership of the “evals-to-product” loop, autonomously determining the best way to turn raw signals into durable datasets that power decision-making across the company
- Build a robust simulator pipeline capable of replaying user interactions with the product in formats legible to LLMs and VLMs, reflecting product changes as they are shipped
- Maintain data trust by implementing monitoring, lineage, and quality checks, ensuring downstream consumers can rely on the results implicitly
- Operate in a small, high-impact team where your work directly shapes how Perplexity measures and improves Answer Quality
- Strong proficiency in Python and SQL, with solid fundamentals in system design, data modeling, and distributed systems
- Experience building big-data systems, including distributed compute, large-scale storage, and high-volume pipelines
- 4+ years of software, data, or machine learning engineering experience shipping and operating production systems
- Demonstrated ownership of ambiguous technical projects from initial design through production operation
- Ability to work effectively with data scientists, engineers, and product partners
- Experience building evaluation, experimentation, observability, or machine learning infrastructure
- Familiarity with LLM and agent systems, including tool use, execution traces, replay, and simulation
- Experience building on top of large-scale data processing platforms such as Databricks, Snowflake, or ClickHouse