What just happened? In its first major investor event post its split from Western Digital, SanDisk introduced a novel memory architecture aimed at the burgeoning AI market. This new architecture, termed high-bandwidth flash (HBF), combines the extensive storage capability of 3D NAND with the bandwidth levels typically associated with HBM.
This innovative creation features an impressive stack of 16 layers of SanDisk’s latest 3D NAND dies, interconnected via small data pipelines known as through-silicon vias. An additional logic layer contributes to the simultaneous in-and-out data flow across the individual NAND sub-arrays. Consequently, HBF offers 8 to 16 times more capacity per stack compared to current HBM setups.
In a scenario highlighted by SanDisk in one of their examples, a system with eight HBF stacks could accommodate an enormous 4 terabytes for storing large AI models like GPT-4 directly on GPU hardware.
The core innovation appears to be that HBF’s architecture diverges from conventional NAND models by dividing each die into multiple small sub-arrays that allow random parallel access, differing from the traditional block-based planes and pages. This not only facilitates high bandwidth but also retains the cost-effectiveness and high capacity of NAND.
However, the inherent limitation of NAND technology is its higher latency compared to DRAM technologies like HBM. As noted by Tom’s Hardware, HBF follows a similar trend and falls short of matching the rapid speeds of DRAM. This new architecture is intended for read-heavy AI inference tasks that necessitate high bandwidth and capacity but can manage the increased latency, making it unsuitable for applications like gaming.
The technology faces certain challenges yet to be addressed. SanDisk has been reticent on how it plans to overcome the write endurance constraints of NAND and the difficulty of block-based addressing, which could impede randomized access. Specific bandwidth achievements of HBF are still unknown.
Despite unresolved questions, SanDisk envisions substantial potential. The company aims for HBF to be an open standard that complements HBM in hardware such as GPUs. To facilitate this, they are already collaborating with partners and planning three full generations of HBF evolution, indicating a strong commitment to the technology’s future.
In the long term, SanDisk anticipates the technology even trickling down from high-end AI systems to consumer-oriented devices like smartphones.
