In computational science, Amdahl’s Law and the Memory Wall dictate that compute cores can only execute operations as fast as operands can be loaded from memory into registers. During autoregressive LLM token generation, the arithmetic intensity is exceptionally low: for every single token generated, the model must read all hundreds of billions of weights from memory into registers.
Consequently, modern AI inference is fundamentally Memory-Bandwidth Bound.
To break this bottleneck, memory manufacturers (SK Hynix, Samsung, Micron) stack DRAM dies vertically using Through-Silicon Vias (TSVs) placed directly adjacent to the compute processor atop a silicon interposer. With the arrival of HBM4, the traditional 1024-bit memory bus doubles to a massive 2048-bit interface, delivering over 2.5 Terabytes per second per stack.
