Research|OCS and DRAM Pooling Expand the Memory Stack, Not Replace HBM

There have recently been market rumors suggesting that Google, in the long-term evolution of its TPU system architecture, is exploring the use of OCS (optical circuit switching) to pool large amounts of DRAM across racks, in order to address the rising cost, supply constraints, and scalability challenges associated with HBM under advanced nodes and 2.5D packaging. This approach is described as reducing system-level reliance on HBM, improving memory capacity elasticity, and lowering overall TCO, leading to the conclusion that demand for OCS, optical modules, and server DRAM would increase. Based on our supply-chain checks, this conclusion is partly correct and partly flawed: the positive implications for OCS, optics, and DRAM are valid, while the conclusion regarding HBM is not.

HBM exists not because “total memory capacity is insufficient,” but because compute accelerators such as GPUs and TPUs depend critically on a memory subsystem with extremely low latency, very high bandwidth, and deterministic access characteristics in order to fully utilize compute throughput. By connecting HBM directly to the compute die through 2.5D packaging, memory access latency is reduced to the order of tens of nanoseconds, while delivering bandwidth density per watt that is orders of magnitude higher than DDR. These characteristics place HBM squarely on the most latency-sensitive and bandwidth-critical paths of model forward passes, backward propagation, and inference, making it a fundamental performance anchor in accelerator systems.