Like most, we’re chasing new solutions for optimizing tokenomics in AI deployments. We’ve recently been researching KV cache optimization strategies for AI-assisted coding. It’s particularly hard to get high KV-cache reuse rates using standard prefix caching, because with each request, a coding agent needs to load code from one or more solution files into context, often by navigating references. Blocks enter context in virtually every possible order, and the context is altered after every code change, rendering the previous KV cache almost useless.
We realized that if there were a way to treat each file or method in a codebase as an independent block of context that could be recalled from cache anytime an agent needed it, regardless of order, the KV cache hit rate could easily skyrocket above 90%. Through our market analysis we found a technique called CacheBlend designed to do almost exactly that. And as of today, June 9, 2026, the vLLM developer preview documentation indicates that CacheBlend is available for some Llama and Qwen-based models like DeepSeek R1, DeepSeek v3.2, and Qwen 3, as part of their unified Cache Manager (UCM) architecture.
We also found that in early 2026, the team at Tensormesh ran a careful trace analysis of RepoAgent, an open-source AI agent that generates documentation for GitHub repositories. The headline result showed that under standard prompt caching, RepoAgent achieved a 3.4% cache hit rate. Yet most of the tokens it was processing were genuine reuse of content the system had already seen on previous requests. The agent was throwing away almost all of the caching it should have been able to claim. And by moving beyond simple prefix caching, they immediately saw that cache hit rate jump 25x.
Key takeaways
- Prefix caching is on by default in vLLM, SGLang, and most production inference stacks, and it works well for chatbot traffic where each turn cleanly appends to a stable conversation history.
- In agentic and RAG workloads, the text obtained from retrieved documents and tool outputs is used in inconsistent orders. A single early-token change invalidates everything that follows, collapsing cache hit rates to near-zero.
- Real-world impact: RepoAgent runs at a 3.4% hit rate under prefix caching. The OpenClaw agent shows a 5.5-second time-to-first-token with only a 48% hit rate on turn 2, even when roughly 99% of the content is unchanged from the previous turn.
- Non-prefix KV cache reuse, the technique introduced by CacheBlend and awarded Best Paper at ACM EuroSys ’25, lifts hit rates above 85% with no measurable quality loss, by selectively recomputing only the 10-15% of tokens that actually need updating.
How prefix caching works, and why tokenomics suffer for non-chatbot use cases
A quick refresher. When a transformer processes a prompt, it does two distinct things: it performs a prefill pass over the entire input to build the KV cache, then it does decode, generating one token at a time, with each new token attending to everything cached so far. Prefill is the expensive part, especially for long inputs. Decode is comparatively cheap.
The implication, which the Manus team made famous in their context-engineering post, is that production agents have a startling cost profile: an average input-to-output token ratio of about 100:1. And as supportable context lengths grow with new models, this is likely going to get worse for certain use cases. The vast majority of compute goes into prefill. So, anything that lets you skip prefill is enormously beneficial. Caching the KV state from a previous request and reusing it on the next, when the prompts share a common beginning, works swimmingly with prefix caching. vLLM’s own documentation describes it as “almost a free lunch.”
In a chatbot, that’s exactly right. Each new user message appends to a growing conversation history, so the entire cache from the previous turn carries forward and only the new message has to be processed. Agents don’t work that way. Their prompts mix stable elements (system prompt, tool definitions) with content that shifts every turn (retrieved documents, tool outputs, scratchpad state). Because the autoregressive constraint demands an exact match from token zero, every token after the first inconsistency has to be recomputed. The user-visible content might be 99% identical to the previous turn. The cache hit rate can still be near zero.
What is non-prefix KV cache reuse?
Non-prefix KV cache reuse is a serving technique that reuses precomputed key-value cache blocks regardless of where they appear in a prompt, instead of requiring an exact match from the start of the token sequence. It restores attention correctness by selectively recomputing a small subset of tokens whose representations are most affected by position changes.
The “prefix” constraint that this technique relaxes is older than the agent era. It comes directly from the autoregressive math of transformer attention: when a token’s KV representation is computed, it incorporates information from every preceding token. If any of those preceding tokens change, the cached representation is no longer correct, and the safe move is to throw it away. Prefix caching took that conservative reading literally… match exactly from the start, or don’t cache at all. And for a long time, that was good enough. The CacheBlend paper, published at EuroSys ’25 by Yao et al., is one of the first techniques to ship in open source that actually breaks the constraint without sacrificing output quality.
Three concrete failures: RepoAgent, OpenClaw, and the long-context RAG case
RepoAgent
RepoAgent processes a repository in topological dependency order, injecting {project_structure} and {file_path} variables inside the first 200 characters of every prompt. Because every documentation pass is for a different code object, those variables change on every request and break prefix alignment before any cacheable content is reached. Tensormesh measured hit rate climbing from 3.4% to 85.9% under CacheBlend, a 25× improvement.
OpenClaw
OpenClaw, the agentic framework the LMCache team benchmarked, shows the same pattern in a cleaner form. Their evaluation used a two-turn structure: system prompt, then user query, then retrieved supporting document. On turn 2, the supporting document was identical to turn 1, but the different user query between them invalidated the prefix and forced full recomputation. Prefix caching: 5.553 seconds TTFT, 48% hit rate. CacheBlend: 2.325 seconds TTFT, 98% hit rate. A 42% latency reduction.
Long-Context RAG
NetApp’s recent Tech ONTAP post on CacheBlend walked through what happens when you start precomputing KV caches at the corpus level. For a knowledge base of 1,000 documents at 10 pages each, a pretty small collection, the cache lands at roughly 130TB, which has to live on fast CPU memory or NVMe with ultra-high-bandwidth networking between nodes. Even with newer compression techniques like TurboQuant achieving a 5–6× reduction in cache footprint, that’s still dozens of terabytes at extremely high cost. This stops being a GPU-memory problem and starts being a storage-tier problem.
Why “just write better prompts” isn’t the whole answer
The Manus team’s response to this problem (and it’s a good start) is basic context engineering. They argue that small improvements in cache hit rate translate to large savings in production, where the cost of an uncached input token can be 10x the cost of a cached one on frontier APIs.
- Keep your prompt prefix stable.
- Don’t put timestamps at the top of the system prompt.
- Don’t dynamically add and remove tools mid-conversation.
- Use the file system as extended context rather than re-inserting content into the prompt.
All of that is useful, but it still can only get you so far. In RAG, the retrieved content is dynamic by definition. That’s the point of adding retrieval to the mix. In multi-step agent loops, tool outputs are likewise dynamic. The reordering of retrieved chunks isn’t something you can really fix with prompt design alone.
How CacheBlend works under the hood
Why can’t the inference stack just splice precomputed KV caches together in any order? Because each token’s KV representation depends on all the tokens that came before it during prefill. A KV block computed for “Chunk B” in [Chunk A] [Chunk B] [Chunk C] is not the same as the KV block for the same chunk in [Chunk A] [Chunk C] [Chunk B]. Concatenating them in the new order produces attention states that are quietly wrong.
CacheBlend’s insight is that the corruption is concentrated, not diffuse. Not every token in a reused chunk is equally affected by repositioning. The authors call the ones that matter most High-KV-Deviation (HKVD) tokens, and they’re a small fraction of the total, typically 10-15%. By identifying just those tokens and selectively recomputing them while reusing the cache for everything else, CacheBlend produces output that closely approximates a full prefill at a small fraction of the cost: 2.2-3.3x lower TTFT, 2.8-5x higher throughput, and no meaningful degradation in F1 or Rouge-L scores per the published numbers from the paper .
Frequently asked questions
What’s the difference between prefix caching and non-prefix caching?
Prefix caching reuses KV cache only when a new prompt shares an exact match starting from the first token; any change to that prefix invalidates everything after it. Non-prefix caching reuses precomputed KV blocks for matching content regardless of where it appears in the prompt, recomputing only the small subset of tokens whose attention values are most affected by the repositioning.
Does CacheBlend work with vLLM?
Not yet. This is showing up on the feature list of vllm-ascend with support for some Llama and Qwen-based models, but only in their latest developer documentation. It is NOT available now in the latest stable release.
How much can non-prefix caching improve cache hit rate?
On RepoAgent, Tensormesh measured an improvement from 3.4% to 85.9%, a 25x improvement. On the OpenClaw agent benchmark, LMCache measured a jump from 48% to 98%, with TTFT dropping from 5.553 seconds to 2.325 seconds. Gains depend heavily on how much content actually repeats across requests; workloads dominated by genuinely unique content will see less benefit.
Is CacheBlend production-ready?
Mixed. The technique is published and shipping in open source, but engineering maturity for enterprise production isn’t quite there yet. Once this makes it to vLLM’s stable release, this technique will likely see way more adoption.
Does non-prefix caching hurt response quality?
The CacheBlend paper reports comparable or slightly higher quality scores than full recomputation across F1 and Rouge-L benchmarks. The selective-recomputation step targets exactly the tokens whose representations would otherwise be wrong, so the output approximates a full prefill closely. vLLM notes that there isn’t a more than 1-3% quality drop when compared to full KV recompute.
What to watch next
Prefix caching was the right default for the era when LLMs were chatbots. It still works fine for chatbot traffic. But for the workloads that increasingly matter (agents, RAG pipelines, and long-running tool-using systems) it has stopped being sufficient. There’s a huge magnifying glass on cache hit rates due to the sudden focal shift to tokenomics for inferencing, and most teams are measuring it lower than they realize.
Two things worth watching in the next year. First, whether closed inference providers expose non-prefix caching at the API level, which could change the build-versus-buy economics for teams not running their own serving stack. Second, whether vLLM and SGLang converge on a common non-prefix caching standard, or whether the techniques diverge. Either way, the assumption that prefix caching is almost a free lunch is increasingly wrong for the new dominant agentic workloads.
