TL;DR
KV cache, not model weights, is now the dominant cost driver in LLM inference. EverPure’s R&D is combining Google’s TurboQuant algorithm with Pure KVA and FlashBlade to compress the KV cache ~5x on the GPU itself in sub-millisecond time, then offload it over RDMA to external flash. The result: smaller storage footprint, less network bandwidth, fewer GPUs per user, and HBM freed up for bigger models or longer contexts.
Key takeaways:
- TurboQuant compresses KV cache ~5–6x with near-zero accuracy loss and no calibration, unlike earlier quantization methods (KIVI, GEAR)
- Compression happens on the GPU before bytes leave the chassis, so the CPU, PCIe bus, network, and storage all see only the compressed payload
- A 30K-token session shrinks from 1.72 GB (FP16) to 349.5 MB at 3-bit; benchmarks hit higher read bandwidth than FP16 while moving 5x less data
- A five-chassis FlashBlade deployment collapses to one chassis (5U vs 27U)
- The reclaimed HBM lets the same GPU count host larger models or serve longer contexts and more concurrent users
The economics of LLM inference are increasingly being decided not by model weights, but by KV cache, the running memory of every conversation an LLM has ever had open. As context windows of frontier open-source models stretch up to 256K tokens and beyond, that cache has become the single largest consumer of memory in a modern inference cluster, and the single biggest reason AI infrastructure costs keep climbing.
EverPure’s R&D team has been aggressively working on a different answer to that problem. By combining Google Research’s recently published TurboQuant compression algorithm with Pure KVA and FlashBlade, they can compress the KV cache by roughly 5x on the GPU itself in sub-ms time, then move that compressed state across the network and onto external flash. While this is not yet available to the public, this is sure to raise eyebrows in 2026. The rest of this post covers how TurboQuant works at a high level, then turns to what the combined architecture means for the cost of running inference at scale, including contributing factors for ROI measurement.
Near-Lossless Compression
TurboQuant is a KV cache compression algorithm published by Google Research back in March. The headline number is a roughly 5-6x compression of KV cache with near-zero accuracy loss, freeing scarce GPU memory so LLMs can support longer contexts, more users, and bigger models on existing infrastructure. The value of TurboQuant increases as context length grows.
I won’t bore you here with the technical details, as EverPure’s recent writeups do a FANTASTIC job of explaining it. Check the links at the bottom of this article. But what makes it different from earlier low-bit KV cache quantization (KIVI, GEAR, and similar) is that it eliminates the per-block metadata overhead that previously ate one to two bits of the compression budget while maintaining near-lossless accuracy at higher compression ratios and requiring no calibration.
The architecturally important detail is where the compression happens (the 2nd point above). TurboQuant runs on the GPU, before any KV cache bytes leave the chassis. In a conventional offload path, the uncompressed cache traverses every layer in the system… the CPU, the PCIe bus, the network, and the storage subsystem. Each of those components have to be right-sized for an uncompressed payload. With TurboQuant, only the compressed payload ever crosses the wire.
At 30K tokens, a single session goes from 1.72 GB in FP16 to 349.5 MB at 3-bit compression. EverPure’s benchmarks paired that compression with GPU Direct Storage (GDS) over RDMA against a FlashBlade//S system, which removes the CPU from the restore path entirely. Across eight A100 GPUs running Qwen2.5-7B, that combination delivered higher aggregate read bandwidth than the FP16 baseline at every context length tested while moving 5X less data, and an end-to-end session migration at 50,000 tokens produced 50 out of 50 tokens matching the uncompressed baseline. The full benchmark methodology is in Up to 10X Faster KV Cache Restore: TurboQuant Meets FlashBlade.
This technique pairs well with KV-Cache offloading to external storage for several reasons.
- Moving a compressed cache from external storage back to a GPU is significantly faster. So, more value can be obtained from a smaller number of flash blades in terms of storage space and bandwidth.
- Because the compression/decompression happens on the GPU in a way that leans on the GPUs computational strengths (bit manipulation and matrix multiplication/rotation), this can happen in sub-millisecond time, adding virtually no latency to the operation.
- Max context length can increase substantially vs. implementations without TurboQuant and external KV cache. As an example, with a cluster of 32 8-way H200 servers with kimi-k2.5 models loaded on them, only about half of the GPU memory is available for storing cache due to the size of the model, leaving a total of around 16-20 TB of total cache storage in the system, and a cache that is highly fragmented. Moving this to even a single FlashBlade immediately increases the amount of cache context and centralizes access to make all of it available to all servers.
- In the future, this extra GPU memory can be leveraged to store even larger models, which we will continue to see as the frontier labs push the edge.
The Financial Picture
When you compress on the GPU and persist on FlashBlade, the savings stack across three different line items on the same bill:
Less flash needed. A 6-bit TurboQuant cache is roughly one-fifth the size of its standard equivalent with the same context length. Whatever capacity plan a deployment had for KV cache persistence shrinks proportionally. For larger deployments, NVIDIA’s AI factory architecture calls for 16TB of KV cache storage per GPU, defined before TurboQuant existed. At 6-bit compression that number drops to 3.2TB per GPU; across a 1,000-GPU cluster it falls from 16 petabytes to roughly 3.2 petabytes of premium storage — a delta whose dollars, watts, and rack space are not trivial.
Less network bandwidth used. Because compression happens on the GPU before transfer, the network sees only the compressed payload. A 5X reduction in payload size translates directly into 5X more concurrent operations on the same fabric, or the option to provision a less aggressive fabric for the same workload.
Fewer GPUs per User. With more cache retained, cache hit rates increase. This means less tokens have to be computed and the overall number of GPUs needed to support a given number of requests is reduced. As we enter capex territory of $35K+ per GPU, in addition to ongoing power/cooling costs of $3K+ per GPU per year, this stacks VERY quickly.
Same workload, much smaller footprint
Take a deployment previously sized for five fully populated //S200 chassis worth of warm session capacity. In a multi-chassis configuration, that build requires the full XFM-backed fabric: two XFMs to mesh the chassis together, four inter-chassis links per chassis (2 to each XFM), and a full set of uplinks from each XFM out to the SN5610 switch pair. But if we can fit all of that into a single FlashBlade chassis, we no longer need the XFMs and additional optics.
| Component | Before TurboQuant | After TurboQuant |
|---|---|---|
| FB//S200 chassis | 5 | 1 |
| XFM-8400 modules | 2 | 0 (not required for single-chassis) |
| XFM to Switch Links | 16 | 0 (no XFMs needed) |
| Rack units | 27U (5× 5U chassis + 2× 1U XFM) | 5U |
3-year cost picture.
| Before TurboQuant | After TurboQuant | Delta | |
|---|---|---|---|
| Steady-state power | 13,120 W | 2,500 W | −10,620 W |
| Annual energy | 114,931 kWh | 21,900 kWh | −93,031 kWh |
| Annual energy cost @ $0.1437/kWh | ~$16,515 | ~$3,147 | -$13,368/yr |
| 3-year energy cost (IT only) | ~$49,545 | ~$9,441 | -$40,100 |
| 4 x chassis AOC cabling + XFM Uplinks | ~$30,000 | $0 | –$30,000 |
| 2 x XFM-8400 pair | ~$100,000 | $0 | –$100,000 |
| 3-year direct savings (energy + capex) | -$170,100 |
The $40,100 in avoided electricity is IT-only — adding facility cooling at a typical 1.3–1.5x PUE multiplier brings the all-in three-year energy savings to roughly $50,000–$60,000. Stack on the avoided capex (XFM pair and four chassis worth of inter-chassis optics) and the deployment runs $180,000–$190,000 cheaper over three years than the uncompressed equivalent — before counting the four FlashBlade chassis themselves (where you’d easily be dealing with 7-figure implications, regardless of the exact storage configuration), the rack space, the implementation labor, or the operational simplicity of running one chassis instead of five.
What You Do With the HBM You Get Back
This is where the conversation usually shifts from “we saved money” to “we can do something we couldn’t do before.”
The open-source model landscape keeps moving in one direction: bigger. Each new generation of open-weight releases from Llama, Qwen, Mistral, DeepSeek, Kimi, Gemma, and etc. has pushed parameter counts and context windows further than the last. The GPUs people bought 18 months ago were sized for the models of 18 months ago. The KV cache reservation that used to coexist with a 70B model is now competing for the same HBM with the weights of a 120B or 400B successor.
Offloading KV cache to FlashBlade and shrinking what stays resident through TurboQuant gives that fight back to the model weights. A model that previously required eight GPUs to serve at 32K context may now fit on four. Or the same eight GPUs can hold a substantially larger model at the same context length, without a hardware refresh. The headroom can also be spent on the other axis: longer contexts, more concurrent users, sessions that persist for hours instead of being evicted after minutes.
A Tiered Memory Architecture for Inference
The picture this all paints is one EverPure has been advocating for a while: inference workloads need a tiered memory architecture, not just more GPUs. TurboQuant settles the math for compressing the cache itself. FlashBlade with GPU Direct Storage settles the I/O path for moving the cache around. Put them together, and the cost of running long-context, high-concurrency inference looks meaningfully different than it did a year ago.
References
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
https://arxiv.org/abs/2504.19874
Google: TurboQuant: Redefining AI efficiency with extreme compression
https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/
EverPure: TurboQuant Compresses KV Cache by 5X. Does That Mean You Need Less Memory?
https://blog.purestorage.com/purely-technical/turboquant-compresses-kv-cache-by-5x-does-that-mean-you-need-less-memory/
EverPure: Up to 10X Faster KV Cache Restore: TurboQuant Meets FlashBlade
https://blog.purestorage.com/purely-technical/up-to-10x-faster-kv-cache-restore-turboquant-meets-flashblade/
