Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Huawei Hong Kong Research Center(HKRC)

NewBe an early applicant

LLM KV Cache Systems Engineer

Posted 35 views

Summary

Systems engineer designing and building KV Cache subsystems for LLM inference: memory allocators, compression, transfer (RDMA/NVLink), GPU/CPU/SSD tiering, and KV-aware scheduling in distributed serving systems like vLLM and SGLang. Core stack: C++, Python, CUDA, Linux.

About the role

We are looking for an LLM KV Cache Systems Engineer to design and build KV Cache subsystems for large-language-model inference. You will work on memory allocation, lifecycle management, reuse, compression, transfer, tiering, and KV-aware scheduling across single-node and distributed serving systems.

This is a systems role. Direct source-level experience with an LLM KV Cache subsystem is more important than familiarity with serving-framework APIs alone.


What you will do

  • Design block/page allocators, KV Cache lifecycle management, fragmentation control, eviction, prefix caching, cross-request reuse, and preemption/recovery mechanisms.
  • Optimize KV layouts, GQA/MQA handling, quantization, compression, or sparsity while balancing accuracy, capacity, memory bandwidth, and compute overhead.
  • Build KV transfer and disaggregated-serving mechanisms, including prefill/decode separation, RDMA/NVLink/CXL transport, and GPU/CPU/SSD memory tiering or offload.
  • Develop KV-aware request routing, batching, admission control, scheduling, and multi-tenant isolation under latency and SLA constraints.
  • Integrate and productionize KV technologies in systems such as vLLM, SGLang, TensorRT-LLM, LMDeploy, Mooncake, or DistServe, with observability and failure handling.


What we are looking for

  • Master’s degree or Ph.D. in Computer Science, Computer Engineering, Distributed Systems, High-Performance Computing, or a related field.
  • Deep understanding of Transformer attention, prefill/decode execution, memory behavior, and modern LLM serving architectures.
  • Direct source-level design, implementation, or research experience in an LLM KV Cache subsystem.
  • Proficiency in C++/Python, Linux, CUDA or equivalent accelerator development, and distributed-system performance engineering.
  • Demonstrated results in at least one KV-specific metric: capacity, hit or reuse rate, fragmentation, transfer bandwidth, TTFT, TPOT, throughput, latency, memory cost, or service-level efficiency.


Preferred qualifications

  • Experience with disaggregated inference, distributed KV transfer, cache-aware routing, or GPU/CPU/SSD tiered KV storage.
  • Core open-source contributions, publications, patents, or large-scale production deployment in LLM serving or KV Cache systems.

Skills

Apply

See also

AI Engineering jobs by country — openings, pay and top skills →