Tsinghua + Moon's Dark Side split the pre-fill process out to run separately, KV cache can handle it over Ethernet, a 54% throughput increase is indeed impressive, the cost structure for long-context reasoning is about to change.

ME News reports that the Dark Side of the Moon and Tsinghua University proposed Prefill-as-a-Service on arXiv, enabling the pre-filling stage of large model inference to run across data centers. By using a hybrid attention model, they significantly reduce KV cache throughput, allowing the cache to be transmitted over Ethernet and returned to the local cluster for decoding. The PrfaaS architecture builds an independent prefill cluster that only routes requests for long contexts that miss, while short requests stay on the local PD; it also introduces length threshold routing and bandwidth-aware scheduling. In tests with a 1T parameter hybrid model, throughput increased by 54% compared to homogeneous PD, and by 32% compared to naive heterogeneous setups.