Byte-level LLMs are finally making serious progress; 1.7 billion parameters can approach tokenization performance, and the vocabulary war may become outdated.

Nous Research's paper states that the long-term reliance of large language models on tokenizers is expected to be replaced in the future. Controlled tests with 1.7 billion parameters show that the benefits of tokenization mechanisms can be simulated at the pure byte level through engineering methods. Experiments indicate that increasing throughput in native byte models and injecting morphological boundaries can significantly narrow the gap with tokenization models; under the same computational power, simulated compression improves single-step gradient processing, becoming the main source of contribution. At the same time, overlaying subword boundaries onto input bytes establishes a long-term inductive bias that does not leak future information. Although the synergistic effects of larger parameters remain to be verified, at 1.7 billion, the benefits of vocabulary parameters and predicting the next subword are limited. This provides a breakthrough idea for tokenization-free large models, and future architectures should focus on increasing throughput and explicitly incorporating morphological priors in a non-leaking manner.