Beyond GPU Memory: The Changing Role of KV Cache

Дата публикации: 22-09-2026 06:02:04

Ask an AI assistant a question about a document, and it first processes the document and your question. It then writes its answer in tokens: words, parts of words, or punctuation. Each new token depends on the context that came before it.Transformer-based models avoid repeating much of that work through a key-value cache, or KV cache. It stores intermediate numerical representations of processed tokens, so the model can refer to earlier context without running those tokens through the entire model again.That saves computation, but the stored state takes up memory. As conversations grow and more users arrive, deciding what to keep and where to keep it becomes a serving problem.What Exactly is KVCache ?A transformer uses attention to determine which parts of the available context are useful when processing a token. At each attention layer, it creates three numerical representations:A query, used to look for relevant context.A key, which the query is compared against to calculate attention weights.A value, which supplies the information combined using those weights.The KV cache retains the keys and values for earlier tokens at each layer. Future processing uses a new query to attend to those stored representations. Earlier queries generally do not need to be retained for this purpose.A simple analogy would be to think of it as keeping reusable working material from reading a document, rather than starting the reading process again for every word of an answer. The cache also does not store a finished answer. The model still computes attention and generates a new response.How It Works Today.Generating an answer, also called Inference, typically has two phases:Prefill: the model processes the input prompt, including instructions, documents, and conversation history. It builds the initial KV cache and produces the first output token.Decode: the model processes each newly generated token to produce the next one, reading historical K/V and extending the cache along the way.Three snapshots show four prompt tokens building the initial K and V rows, followed by two decode steps that retain earlier entries and append K/V for the latest processed token.Figure 1. Prefill builds the initial cache. Each decode step processes the latest output token, appends its K/V, and produces the next token. The blocks are illustrative, not actual tensor dimensions.In a common deployment, one worker, a model instance running on one or more GPUs, handles both phases. Active KV usually lives in GPU memory for fast access. Engines such as vLLM use PagedAttention to allocate it in non-contiguous blocks, reducing wasted space and allowing sharing where appropriate.Some engines also retain completed requests' cache blocks for prefix cachingTwo different questions about the same document can share an identical beginning: the instructions and document text. If its cached state remains available and compatible, the second request reuses that prefix and processes the new question. Two aligned prompt strips share instructions and a document but end with different questions. The first request computes all K/V; the second reuses the matching prefix and computes state for the new question. Figure 2. Prefix caching reuses the processing of an unchanged beginning, not a previously generated answer.That reuse is more specific than recognizing similar meaning. Changing an earlier token can invalidate reuse from that point onward.vLLM's prefix-cache design identifies blocks using their tokens and preceding prefix, with additional identifiers for matters such as adapters and isolation.When GPU memory fills upGPU Local caching works well while useful state fits and requests reach the worker holding it. Longer contexts and concurrent users strain both assumptions.The memory adds up quickly. Using Qwen3-32B's published configuration, storing K/V in BF16, a 16-bit numerical format, requires 256 KiB per cached token across its 64 layers and eight KV heads. At 32,768 tokens, that is 8 GiB for one sequence. Sixteen independent sequences without shared prefixes require 128 GiB of KV payload alone, this is before model weights and runtime overhead.One Qwen3-32B sequence requires 8 GiB of BF16 KV payload at 32,768 tokens. Sixteen independent sequences require 128 GiB, excluding weights and runtime overhead.Figure 3. Calculated KV payload, before model weights, buffers, and allocation overhead.A worker eventually evicts retained prefixes to make room for active requests. A returning user then pays for another prefill. Landing on a different worker can have the same effect. Coding assistants face this repeatedly: repository context remains useful across tool calls, even while the model is waiting for a tool to finish.The problem is not that inference lacks caching. It is that useful cached state can outlive the request, worker placement, or GPU-memory budget that originally contained it.Keeping state beyond the GPUOne approach keeps active state close to the GPU and moves retained prefixes to CPU memory, SSDs, or shared storage. A worker can fetch useful state when needed, provided retrieval is worth the cost. Side-by-side comparison: a returning request misses the prefix on worker B in a local-only arrangement; with shared caching, state retained by worker A can be fetched and restored at worker B. Dashed blue lines show request routing and solid green lines show cache movement. Figure 4. Shared caching adds a fetch option. Locality-aware routing and recomputation remain alternatives; paths are conceptual.This is already more than a research proposal. Mooncake's FAST 2025 paper describes Kimi's production serving platform, combining distributed KV storage with scheduling that considers cache locality and latency requirements.NVIDIA Dynamo also provides KV-management and transfer components for offloading to host memory, SSDs, and remote-storage backends.These systems do not make GPU memory unnecessary. They expand where reusable state can live between periods of active computation.Choosing where to runThe emptiest worker may need to rebuild a long prefix that a busier one already holds. Cache-aware routing weighs that saved work against queueing and transfer time. Too much affinity creates hotspots; ignoring locality wastes computation.Separating prefill and decode introduces another placement choice. Prefill is often compute-intensive; decode is frequently limited by memory bandwidth. Different workers can be sized and scheduled for each phase, but they need a KV handoff even when no other request reuses that state.A prefill worker builds prompt K/V and transfers the state to a separate decode worker, which loads it and extends it during generation.Figure 5. Prefill/decode disaggregation allows independent resource allocation at the cost of state transfer.When moving state helpsAn 8-GiB cache takes about 0.69 seconds to cross a 100-Gbit/s link at ideal payload throughput. Storage reads, copies, and queueing add more time. For a short prompt, recomputation may finish first.Sharing also needs boundaries. KV depends on the model and preceding context, so another model cannot automatically consume it. Security research shows that exposed state can reveal prompt information under studied attack conditions. Access controls and tenant isolation matter throughout retention and transfer.The August 2026 paper An Internet for the KV Cache  proposes coordinating this state across datacenters and cloud providers. An interoperable Internet-wide cache remains a vision; managed distributed caching already has production examples.For teams building AI factories, the infrastructure running AI workloads, the immediate task is to test whether reuse improves their own service. Compare cold and cached requests under realistic concurrency. Measure first-token latency and gaps between output tokens, including storage access, host-to-device transfers, network contention, and scheduling. A healthy cluster alone won't tell you whether the cache is helping.The nearer-term shift is concrete: inference systems are learning to manage reusable context independently of a single GPU's memory. The design question expands from "Which accelerator is available?" to "Where should this request run, given the state it can safely reuse?"

Схожие новости

#	Наименование новости	Тональность	Информативность	Дата публикации
1	Как оптимизировать инференс LLM: кеширование, время ответа и GPU-ресурсы	0	11.5	08-07-2026
2	Как я потратил вечер, разбираясь, почему кэш в клоде, gpt и джемини — это три разных зверя	0	6.67	24-07-2026
3	AI inference is getting cheaper, but your agents are getting more expensive	0	12.86	18-08-2026
4	Крупный и средний бизнес за полтора года вчетверо увеличил траты на GPU-серверы — данные Рег.облака	0	7	04-06-2026
5	How Frontier AI Companies Build AI Models	0	5	01-07-2026
6	What is the AI Context Window?	0	9.59	21-07-2026
7	vLLM vs LMDeploy vs Triton: обзор бэкендов для инференса LLM	0	7	18-07-2026
8	NVIDIA выкладывает на Hugging Face по несколько моделей в неделю, ...	0	9.68	28-07-2026
9	As AI Increases Demands on Memory, Storage Steps Up	0	8.18	04-08-2026

Классификация: . Схожих патентов: 0. Схожих новостей: 9. Тональность: 0. Информативность: 7.13. Источник: community.hpe.com.