Software Engineer, Distributed Systems (AI infra)
Summary
Builds and scales a distributed web-crawling pipeline that indexes hundreds of millions of pages daily for an AI-powered search infrastructure.
A applied AI category leader is building next-generation search infrastructure for the AI era — the layer that has to understand, index, and retrieve over a meaningful fraction of the public internet. This team owns that ingestion pipeline end-to-end: deciding what to crawl, fetching and rendering it correctly, and keeping the index fresh at internet scale. Think distributed crawling at hundreds of millions of pages a day, with all the throughput, politeness, and anti-bot challenges that come with it.
Fits for this team include:
- Distributed systems engineers — deep experience with large-scale systems, throughput/latency optimization, and reliability at internet scale. People who think naturally in terms of "how do we scale this to millions of pages" and "how do we keep this fast and robust."
- Quant, systematic trading, or HFT engineers — the modeling, data pipeline, and low-latency systems instincts from that world map directly onto this problem. Prior crawling experience isn't required.
- Backend / infra-adjacent engineers — solid production experience close to caching, queuing, storage, or observability systems, comfortable owning services end-to-end including on-call.
What's needed:
- Experience with a high-performance systems language (C++, Rust, or similar)
- Familiarity with TypeScript, Playwright, and Chrome DevTools Protocol is a strong plus
- A track record of optimizing systems to an exceptional degree, not just shipping and moving on
- Genuine interest in the problem of making high-quality information findable at scale