In this Dark Moon phase, Yue An routes the pre-fill to a remote data center to run, with the KV cache moving back and forth over Ethernet. The 54% throughput increase is indeed fierce. At last, long context no longer has to max out local bandwidth.

ME News reports that the Dark Side of the Moon and Tsinghua University proposed Prefill-as-a-Service on arXiv, enabling the pre-filling stage of large model inference to run across data centers. By using a hybrid attention model, they significantly reduce KV cache throughput, allowing the cache to be transmitted over Ethernet and returned to the local cluster for decoding. The PrfaaS architecture builds an independent prefill cluster that only routes requests for long context misses, while short requests stay on the local PD; it also introduces length threshold routing and bandwidth-aware scheduling. In tests with a 1T parameter hybrid model, throughput increased by 54% compared to homogeneous PDs and by 32% compared to naive heterogeneous setups.