LLM KV Cache Systems Engineer
Summary
Systems engineer designing and building KV Cache subsystems for LLM inference: memory allocators, compression, transfer (RDMA/NVLink), GPU/CPU/SSD tiering, and KV-aware scheduling in distributed serving systems like vLLM and SGLang. Core stack: C++, Python, CUDA, Linux.
About the role
We are looking for an LLM KV Cache Systems Engineer to design and build KV Cache subsystems for large-language-model inference. You will work on memory allocation, lifecycle management, reuse, compression, transfer, tiering, and KV-aware scheduling across single-node and distributed serving systems.
This is a systems role. Direct source-level experience with an LLM KV Cache subsystem is more important than familiarity with serving-framework APIs alone.
What you will do
- Design block/page allocators, KV Cache lifecycle management, fragmentation control, eviction, prefix caching, cross-request reuse, and preemption/recovery mechanisms.
- Optimize KV layouts, GQA/MQA handling, quantization, compression, or sparsity while balancing accuracy, capacity, memory bandwidth, and compute overhead.
- Build KV transfer and disaggregated-serving mechanisms, including prefill/decode separation, RDMA/NVLink/CXL transport, and GPU/CPU/SSD memory tiering or offload.
- Develop KV-aware request routing, batching, admission control, scheduling, and multi-tenant isolation under latency and SLA constraints.
- Integrate and productionize KV technologies in systems such as vLLM, SGLang, TensorRT-LLM, LMDeploy, Mooncake, or DistServe, with observability and failure handling.
What we are looking for
- Master’s degree or Ph.D. in Computer Science, Computer Engineering, Distributed Systems, High-Performance Computing, or a related field.
- Deep understanding of Transformer attention, prefill/decode execution, memory behavior, and modern LLM serving architectures.
- Direct source-level design, implementation, or research experience in an LLM KV Cache subsystem.
- Proficiency in C++/Python, Linux, CUDA or equivalent accelerator development, and distributed-system performance engineering.
- Demonstrated results in at least one KV-specific metric: capacity, hit or reuse rate, fragmentation, transfer bandwidth, TTFT, TPOT, throughput, latency, memory cost, or service-level efficiency.
Preferred qualifications
- Experience with disaggregated inference, distributed KV transfer, cache-aware routing, or GPU/CPU/SSD tiered KV storage.
- Core open-source contributions, publications, patents, or large-scale production deployment in LLM serving or KV Cache systems.
Skills
Sign in to read and join the discussion.