The memory bottleneck of online inference has finally been tackled; incremental storage reminds me of git diff—using information gaps to save space, clever.

Research proposes Δ-Mem, an online memory system designed for large language models. By only storing and updating incremental changes in activation, abandoning the complete activation state, it significantly reduces memory usage. Experiments show memory consumption can decrease by up to 70%, with minimal loss in output quality, enhancing online inference and continual learning capabilities in resource-constrained environments.