SMART KV Cache Technology - Brochure

Accelerating AI Inference with KV Cache and CXL® Memory Designing a Cost-effective KV Cache using Penguin Solutions’ SMART CXL Memory.

Breaking the AI memory wall: Scalable memory architecture for next-generation AI infrastructure As enterprises deploy large language models (LLMs), copilots, and generative AI applications at scale, the focus of AI infrastructure is shifting from model training to high-performance inference.

Key Benefits

•

Eliminate GPU memory bottlenecks

While GPUs provide enormous compute capability, inference performance is increasingly limited by memory capacity and bandwidth. Architects rely on Key-Value (KV) cache, a memory optimization that stores precomputed keys (K) and values (V) and avoids recomputation of new tokens. Increasingly KV caches are being designed with CXL memory which is an optimal solution to cost effectively increase memory capacity and bandwidth using PCIe slots.

A major driver of this demand is Key-Value (KV) cache, a memory structure used by transformer models to accelerate token generation during inference.

As prompts grow longer, conversations extend across multiple interactions, and thousands of users simultaneously interact with models, memory limitations create a bottleneck, known as the AI memory wall, which prevents GPUs from operating at full efficiency.

In many AI inference environments, GPUs spend a significant portion of time waiting for memory rather than performing computation. As models grow and workloads scale, memory architecture becomes a critical factor in AI performance and infrastructure efficiency.

By combining Compute Express Link® (CXL®) memory expansion technology, Penguin Solutions’ SMART CXL Memory Add-in-Cards (AICs), and Penguin Solutions MemoryAI™ KV Cache Server, organizations can build scalable AI infrastructure capable of supporting modern inference workloads. © 2026 Penguin Solutions KV Cache Technology Brochure - 5.26

•

Enable longer context windows for AI models

•

Support greater inference concurrency

•

Improve GPU utilization and efficiency

•

Scale memory capacity independently from compute

Featured Solutions

•

SMART 1TB CXL Memory Add-in-Cards (AICs)

•

Penguin Solutions MemoryAI™ KV Cache Server

What is KV Cache for AI Inference?

Input Prompts Token

KV cache is a memory optimization in transformer models (e.g.

LLMs) that stores precomputed keys (K) and values (V) from attention layers during inference. In autoregressive generation, it reuses K/V for previous tokens, avoiding recomputation and

Compute K/V for All Tokens

reducing complexity from O(n2) to O(n) per new token.

Expensive GPU memory has historically been used for KV caching, delivering the speed for real-time AI responses critical for long

Store in KV Cache (HBM or CXL)

contexts. However, in large models and extended sequences, it still creates memory bottlenecks.

Using CXL memory, like Penguin’s SMART CXL AICs cost-effectively

Generate Next Token: Reuse Cache + Append K/V

extends capacity and bandwidth, addressing memory wall issues.

Repeat for Sequence: Cache Grows Linearly

Where is KV Cache Used?

Why CXL Memory?

KV cache is fundamental to modern AI applications:

CXL is an industry-standard interconnect that enables servers to attach large pools of memory using PCIe infrastructure. CXL enables a new architecture where memory capacity can scale independently from compute resources.

•

LLM inference

•

Conversational AI and chatbots

•

Enterprise AI copilots and assistants

•

Document summarization and analysis

Key CXL Capabilities:

•

Code generation systems

•

Memory expansion beyond traditional DIMM limits

•

Retrieval-Augmented Generation (RAG)

•

Memory pooling across servers and devices

•

Cache-coherent memory access

•

High-bandwidth PCIe Gen5 connectivity

What’s driving KV cache memory consumption: •

Longer context windows: Models are now processing tens or hundreds of thousands of tokens per request

•

Multi-turn conversations and concurrent users: Chat applications accumulate context across multiple interactions, and each inference session requires its own KV cache allocation

•

Larger model architecture: Greater model complexity increases memory requirements required per token

© 2026 Penguin Solutions KV Cache Technology Brochure - 5.26

Benefits: •

Higher GPU utilization

•

Longer AI context windows

•

Increased inference concurrency

•

Reduced GPU infrastructure requirements

•

Lower power consumption and operational cost

KV Cache Speeds Up Inference Performance Modern transformer-based AI models generate responses one token at a time. During this process, the model evaluates relationships between tokens using an attention mechanism.

Without optimization, the model would need to recompute attention across the entire prompt every time a new token is generated. This would dramatically increase computation requirements and slow inference performance.

To solve this problem, transformer models use (KV) cache. KV cache stores previously computed attention results - known as keys and values - in memory. These stored values are reused during subsequent

Prompt Processing (Prefill Phase) The model processes the input prompt and computes attention keys and values.

KV Cache Storage These values are stored in memory as KV cache.

Token Generation (Decode Phase) When generating new tokens, the model reuses cached values rather than recomputing them.

inference steps, allowing the model to generate new tokens without recomputing earlier calculations.

Cache Growth As additional tokens are generated, new key/value pairs are appended to the cache.

The AI Memory Wall Most AI inference systems store KV cache inside GPU High Bandwidth Memory (HBM). While HBM delivers extremely high throughput, it also has critical challenges: • • •

Limited capacity per GPU High cost per gigabyte Shared with model weights and activations

As AI workloads scale, KV cache quickly consumes available HBM capacity. When GPU memory becomes saturated, inference workloads cannot scale further - even when compute resources remain available. Adding additional GPU servers increases infrastructure cost and energy consumption without addressing the underlying memory constraint. This growing challenge is commonly referred to as the AI memory wall.

Tiered Memory for AI Interference Using CXL memory for KV caches resolves the issues of capacity and cost without adding GPUs or consuming GPU HBM memory. Penguin Solutions’ SMART CXL AICs are positioned in the memory tier for AI inference between main memory (100ns latency) and SSD (100,000ns latency).

© 2026 Penguin Solutions KV Cache Technology Brochure - 5.26

SMART CXL Memory Design with SMART STXPL001TAB8RD5 Penguin Solutions SMART STXPL001TAB8RD5 CXL Memory Add-in-Card is 1TB memory expansion card composed of eight 128GB DDR5 RDIMMs and dual CXL memory controllers. It’s esigned for PCIe Gen5 slot compatibility utilizing the CXL 2.0 memory interface, delivering higher performance than SSDs, increased bandwidth utilization, and tremendous capacity expansion using PCIe slots.

Penguin Solutions Appliances utilizing SMART Modular CXL Deploy the MemoryAI™ KV Cache Server Penguin Solutions MemoryAI KV Cache Server is an Altusbased 4U rackmount server designed specifically for use as a KV cache server during AI inference. It features dual AMD EPYC CPUs and enables up to 11 TB of CXL-based memory (SMART CXA-8F2W). The server provides 3TB main DDR5 memory and up to 8 1TB CXL AICs and uses dual 400Gb/s Ethernet NICs for network connectivity.

Use the Altus® XE4318GT-CXL for Big-memory IMDB Applications The Altus XE4318GT-CXL server is a big memory powerhouse.

Leveraging the latest compute technology from AMD and the latest CXL memory technology from SMART, the Altus XE4318GT-CXL is a fully validated and supported CXL server from Penguin Solutions with CXL memory expansion up to 22TB per server.

Learn More About this CXL Memory Application and Related Servers CXL Memory for KV Cache penguinsolutions.com/en-us/products/smart-modular-cxl-based-solutions/kvcache CXL Memory Expansion Servers penguinsolutions.com/en-us/products/cxl-memory-expansion-servers

The AI Factory Platform Company For sales queries, please contact sales@penguinsolutions.com.

To learn more about Penguin Solutions products, please call 800-221-6588 or visit www.penguinsolutions.com © 2026 Penguin Solutions, All rights reserved.