Performance Engineer (Inference Systems)
Posted Updated
Description
- Anthropic’s inference fleet serves Claude to millions of users across our own products and the world’s largest cloud platforms
- The stack that makes this possible is deep and tightly coupled: accelerator kernels, model servers, distributed routing, autoscaling, capacity management. Every layer affects the others, often in ways that are hard to see in isolation
- The Inference System Dynamics team is responsible for understanding that whole system and holding it to a high bar across four dimensions: throughput, latency, reliability, and correctness
- We measure how the fleet performs against its theoretical performance frontier, run cross-layer investigations to explain the gaps, and own the correctness checks that make sure Claude’s outputs are right, not just fast, across hardware platforms and serving configurations
- We don’t own the individual components. We instrument and model them, find the highest-leverage opportunities across them, and partner with the owning teams to land the wins
- You’ll work across all four areas. One week that might mean tracing a tail-latency regression from request timing down through routing and batching into a kernel overhead; the next it might mean tightening a correctness eval so it catches an output regression introduced by a quantization change
- We’re looking for performance engineers who treat correctness as part of performance
- Run cross-layer performance investigations across throughput, latency, and reliability, sizing the gap between actual fleet performance and theoretical rooflines, identifying root causes, and quantifying the value of closing them
- Own and improve the correctness evaluation pipeline that validates model output quality across hardware platforms, numerics, and serving configurations, and lead the investigation when it catches a regression
- Build the observability, dashboards, and modeling tools that make throughput, latency, cost, reliability, correctness, and their interactions legible across the stack
- Partner with kernel, serving, routing, autoscaling, and capacity teams to prioritize and land the highest-impact optimizations your analysis surfaces
- Ruthlessly stack-rank a large surface area of opportunities by impact and effort, and say no to the ones that don’t make the cut
- Representative Projects:
- Trace a 350ms latency gap on a new accelerator platform from end-to-end request timing down to a server scheduling overhead, quantify the win, and land the fix directly or with the owning team
- Redesign the correctness eval gate: determine which signals reliably catch real model-output regressions versus noise, and make it the trusted release criterion across hardware backends
- Build a FLOPs funnel that breaks down where compute actually goes across the fleet, exposing the gap between achieved throughput and kernel rooflines
- Root-cause a numerical divergence between two hardware platforms to a specific kernel change, and define the acceptance threshold going forward
- Model the latency–cost impact of changing batch-sizing and utilization targets, and turn the result into the signal the autoscaler uses in production
Benefits
- Comprehensive health, dental, and vision insurance for you and your dependents
- Inclusive fertility benefits via Carrot Fertility
- 22 weeks of paid parental leave
- Flexible paid time off and absence policies
- Mental health support for you and your dependents
- Competitive salary and equity packages
- Optional equity donation matching at a 1:1 ratio, up to 25% of your equity grant
- Retirement plans with competitive matching
- Life and income protection plans
- $500/month flexible wellness and time saver stipend
- Commuter benefits
- Annual education stipend
- Home office stipends
- Relocation support for those moving for Anthropic
- Daily meals and snacks in the office
- Genuine interest in correctness as an engineering discipline: numerics, evaluation design, regression detection
- Hands-on performance engineering experience: profiling, roofline analysis, latency/throughput optimization, and root-cause investigation in complex production systems
- Ability to communicate quantitative results clearly in writing to influence priorities on teams you don’t manage
- Proficiency in Python, with the ability to read, instrument, and contribute to large production codebases you didn’t write
- Solid data analysis skills (e.g. SQL, pandas, or similar) sufficient to turn raw telemetry into clear findings
- Minimum education: Bachelor’s degree or an equivalent combination of education, training, and/or experience
- We encourage you to apply even if you do not believe you meet every single qualification. Not all strong candidates will meet every single qualification as listed
- Experience with ML systems, especially training or inference infrastructure or general LLM serving stacks. Direct large-scale inference experience is a strong plus
- Familiarity with GPU/TPU/accelerator performance concepts (memory bandwidth, kernel overheads, quantization, collective communication). Reasoning about these matters more than having written kernels yourself
- Experience with reliability engineering for high-throughput services: autoscaling, load balancing, request routing, tail latency
- Experience with model evaluation or numerical regression-detection pipelines
- Experience building observability or telemetry for distributed systems
- Comfortable having impact through influence and evidence rather than direct ownership
- Deadline to apply: None. Applications will be reviewed on a rolling basis
Skills
Sign in to read and join the discussion.