Intel Moves AI Cache to System Memory in New Inference Tests

Cache Architecture Under Pressure

Intel has demonstrated a technique that addresses one of the most persistent bottlenecks in AI inference workloads: the limited memory available on graphics processors. Tests presented at the OCP APAC Summit 2026 show that relocating the key-value cache used during large language model inference from GPU memory to system DRAM can materially improve serving throughput and enable more simultaneous user requests, according to Intel.

The approach targets a specific constraint. When GPU video memory runs short during inference, the entire serving pipeline stalls. By moving the KV-cache - a data structure that stores intermediate attention states across transformer layers - into the larger pool of system memory, inference servers can handle additional load without requiring more expensive accelerators or larger VRAM configurations.

When the Gains Appear

The performance improvement is not universal. Intel's data indicates that throughput gains are most pronounced when memory capacity is the limiting factor. In scenarios where GPU VRAM fills before compute resources are fully utilized, offloading the cache to DRAM allows the system to accept more concurrent inference requests and process longer prompt sequences.

However, the benefit diminishes as workloads shift from memory-bound to compute-bound regimes. Once the GPU's arithmetic units reach saturation, moving cache data to system memory yields little additional throughput. The memory subsystem is no longer the constraint, and the added latency of accessing DRAM over local VRAM becomes a drag rather than a lever.

This dynamic is familiar to data center architects. Inference workloads for large language models oscillate between two states: prefill, which is compute-intensive and processes input prompts in parallel, and decode, which is memory-intensive and generates tokens one at a time. The KV-cache is central to decode efficiency, and its size grows with both the model's parameter count and the length of the context window.

Practical Implications for Deployment

The Intel tests suggest a pathway for operators running inference at scale, particularly those deploying models with context windows stretching into tens of thousands of tokens. Rather than provisioning additional GPU cards to expand VRAM, teams can configure systems to use existing DRAM as an overflow tier for cache storage.

This approach has direct cost implications. High-bandwidth memory remains scarce and expensive. A server with 192 GB of system DRAM is more economical than one with an equivalent amount of GPU-attached HBM. If the workload profile permits offloading without severe latency penalties, the total cost of ownership per token served declines.

The technique also aligns with emerging inference serving patterns in Asia, where demand for multilingual models and extended context windows is rising. Financial institutions in Singapore, customer service platforms in Seoul, and e-commerce engines in Jakarta are all deploying LLMs that must handle long conversational histories or retrieve context from extensive document corpora. These use cases amplify memory pressure and make cache offload strategies more attractive.

Open Questions on Latency and Bandwidth

What Intel has not fully disclosed is the latency penalty incurred when the KV-cache resides in system memory rather than on-chip. DRAM access is slower than GPU HBM by an order of magnitude in bandwidth and several multiples in latency. If the inference engine must fetch cache data across the PCIe bus repeatedly during decode, per-token generation time will increase, potentially offsetting the throughput gains from handling more requests.

The trade-off depends on request arrival patterns and batching strategies. If the serving layer can batch decode operations effectively and amortize the memory access cost across many tokens, the latency hit may remain tolerable. If requests arrive sporadically and must be processed individually, the overhead becomes more visible to end users.

Intel's presentation at OCP APAC Summit 2026 did not include detailed latency histograms or tail-latency percentiles, metrics that matter in production environments where service-level agreements often hinge on P99 response times rather than average throughput.

Where This Fits in the Inference Stack

The cache offload method is part of a broader set of techniques aimed at stretching GPU resources further. Quantization reduces model weight precision, thereby shrinking memory footprint. Speculative decoding uses smaller draft models to predict tokens before verification by the larger model. Paged attention breaks the KV-cache into fixed-size blocks that can be swapped in and out of memory.

Intel's approach complements these methods. It does not replace on-device optimizations but adds another degree of freedom for system architects. A deployment might combine 4-bit quantization to reduce model size, paged attention to manage cache allocation, and DRAM offload to absorb overflow when context lengths spike.

The fact that Intel is publicly testing and presenting this technique signals that the company sees DRAM offload as a competitive differentiator, particularly for its Xeon server processors paired with discrete GPUs. The method requires tight coordination between CPU memory controllers, system interconnects, and GPU drivers - a domain where Intel has decades of integration experience.

As inference workloads continue to scale and context windows expand, the boundary between GPU and system memory is becoming more fluid. Intel's OCP demonstration is a data point in that evolution, showing that the cache bottleneck can be eased with architectural trade-offs that balance capacity, cost, and latency.