Sakana AI has broken the memory bottleneck of deep models; with 1/B memory, large networks can be trained, and the experiments even outperform end-to-end methods.

At ICLR 2026, Sakana AI and the University of Tokyo proposed DiffusionBlocks, which partition the network into blocks and map block updates to the reverse denoising process of diffusion models, achieving block-level independent training. During training, only one block is loaded, while other blocks do not occupy GPU memory, reducing memory usage to 1/B of the original, breaking through the memory bottleneck caused by depth. Experiments show that in visual Transformers, DiT image generation, and text generation tasks, block training can match or even surpass end-to-end training; for Looped Transformers, unidirectional forward updates can also be used, significantly reducing training computation.