Software Engineer Model Routing and Inference
Posted 3
views
Summary
Builds and evolves Cursor's inference platform: an inference gateway that routes AI traffic across model provider APIs, with cross-provider failover, routing backpressure, and admission control. Day-to-day is high-throughput, low-latency distributed systems engineering for AI inference serving.
You will build the inference platform that supports AI interactions in the product. You will evolve an inference gateway across provider APIs, design cross-provider failover, and develop routing backpressure and admission controls to prevent traffic spikes from affecting providers.
Responsibilities
- Build and evolve the inference gateway
- Design cross-provider failover
- Design routing backpressure and admission control
Requirements
- Experience building high-throughput low-latency distributed systems
- Experience with inference serving traffic routing or real-time data pipelines
- Ability to assess cost and performance tradeoffs at scale
- Software engineering fundamentals
- Experience shipping production systems that handle millions of requests
- Judgment regarding reliability cost latency and user experience