PrfaaS: This architecture is interesting—only when long-context requests miss does it route to an independent pre-fill cluster; short requests are handled locally by PD; and bandwidth-aware scheduling prevents unnecessary congestion.

ME News reports that the Dark Side of the Moon and Tsinghua University proposed Prefill-as-a-Service on arXiv, enabling the pre-filling stage of large model inference to run across data centers.
By using a hybrid attention model, it significantly reduces KV cache throughput, allowing the cache to be transmitted over Ethernet and returned to the local cluster for decoding.
The PrfaaS architecture builds an independent prefill cluster that only routes requests for long context misses, while short requests stay on the local PD; and introduces length threshold routing and bandwidth-aware scheduling.
In tests with a 1T parameter hybrid model, throughput improved by 54% over homogeneous PD, and by 32% over naive heterogeneous setups.