Architecture & physics · October 3, 2026
The memory wall: why data movement, not compute, limits AI chips
Reading data from external DRAM can cost thousands of times more energy than an integer add. That physical fact explains why on-device large models depend on the data path, not on peak compute.
Compute is cheap. Moving data is expensive.
In his ISSCC 2014 keynote, Stanford professor Mark Horowitz summarized the energy of basic operations in a 45 nm process. The figures are still among the most cited references in chip-architecture discussions:
| Operation (32-bit) | Energy (approx.) |
|---|---|
| Integer add | 0.1 pJ |
| Floating-point multiply | 3.7 pJ |
| On-chip SRAM cache read | 5 pJ |
| External DRAM read | 640 pJ |
One external DRAM read costs about 6,400 times the energy of an integer add, and more than 100 times an on-chip SRAM read. Newer process nodes lower the absolute numbers, but the structure — moving data costs more than computing on it — has not changed.
Large models make it obvious
When a large language model generates text, every token requires reading almost all of its weights. Take a 1.5-billion-parameter model quantized to 4 bits: about 0.75 GB of weights. If the weights sit in external memory and are read once per token at 25.6 GB/s, the theoretical ceiling is about 34 tokens per second — before any other overhead.
Adding more compute units does not help, because they spend most of their time waiting for data. This is usually called the memory wall.
Three common responses
- More external bandwidth: effective, but it adds power, cost and board-design complexity.
- Smaller models and fewer bits: quantization and pruning reduce traffic, but accuracy loss has to be controlled.
- Move compute next to the data: compute-in-memory keeps frequently accessed data next to the compute units and removes traffic by design.
These are not mutually exclusive. Real system design weighs the model, quantization, memory placement and host constraints together.
How VXI approaches it
VXI is built around SRAM compute-in-memory. Frequently accessed data stays on chip, and the compiler plans where weights and activations live, cutting external traffic per token. The efficiency comes from data-path design, not from the most advanced process node.
To estimate the weight and KV memory your model needs, try the model memory estimator. For architecture details, see the technology overview.
The table shows 45 nm estimates compiled by Horowitz (ISSCC 2014) to illustrate orders of magnitude. They are not VXI product specifications.