ME News Report, April 18 (UTC+8), according to Beating Monitoring from Dongcha, Moonshot AI and Tsinghua University published a new paper on arXiv on April 16 titled "Prefill-as-a-Service," proposing to run the prefill stage of large model inference across data centers. Large model inference consists of two steps: prefill, which reads input data once and generates a KV cache; and decode, which outputs results token by token based on this cache. The hardware requirements for the two steps are completely different—prefill consumes computational power, while decode requires GPU memory bandwidth. The industry’s mainstream approach is to split these two steps onto different machines (PD separation), but this requires RDMA interconnection within the same data center, because the dense attention model’s KV cache outputs dozens of Gbps per second, and if transmission slows, GPUs idle. The breakthrough comes from a new generation of hybrid attention models. The paper’s experiments show that models like Kimi Linear, MiMo-V2-Flash, Ring-2.5-1T, etc., combine a few full attention layers with many linear layers, reducing KV cache throughput by about an order of magnitude, with Ring-2.5-1T achieving a total compression ratio of 36 times. At this point, the KV cache can be transferred from a dedicated RDMA network to a regular Ethernet network for upload. The specific approach of PrfaaS involves establishing an independent "prefill cluster" that only routes requests with long contexts or unhit prefixes, while short requests stay in the local PD cluster; after prefill, the KV cache is transmitted back via Ethernet to the local cluster for decoding. It also introduces length threshold routing, bandwidth-aware schedulers, and hybrid prefix cache pools. The paper reports experimental results using an internal 1T parameter hybrid model (based on Kimi Linear architecture), showing an overall service throughput 54% higher than a homogeneous PD deployment, and 32% higher than a naive heterogeneous scheme, with each machine only using moderate cross-data-center bandwidth. (Source: BlockBeats)

View Original

This page may contain third-party content, which is provided for information purposes only (not representations/warranties) and should not be considered as an endorsement of its views by Gate, nor as financial or professional advice. See Disclaimer for details.

13 Likes

Reward
13
7
1
Share

Comment

Add a comment

GateUser-57ab9c02

· 3h ago

A short request to keep it local is very reasonable, avoiding making a mountain out of a molehill.

View OriginalReply0

Cream-ColoredCross-ChainBridge

· 3h ago

Naive heterogeneity can be outperformed by 32%, with a significant gap in infrastructure quality

View OriginalReply0

CandleSitter

· 3h ago

PD separation reaches new heights of innovation

View OriginalReply0

MidnightReconciler

· 3h ago

The Dark Side of the Moon's recent technical output is impressive.

View OriginalReply0

WatercolorGlassBottle

· 3h ago

Bandwidth-aware scheduling shows they've really been through the pitfalls

View OriginalReply0

MistValleySignpost

· 4h ago

How is the synchronization mechanism for feedback decoding designed?

View OriginalReply0

SeaSaltMintCandy

· 4h ago

Feels especially friendly to long context scenarios

View OriginalReply0

Trending Topics
View More
#
WinGoldBarsWithGrowthPoints
1.22M Popularity
#
WTICrudeFallsBelow90Dollars
1.18M Popularity
#
StockTradingChallengeUpTo17000U
187.65K Popularity
#
USIranNegotiationGame
9.41M Popularity
#
TradeCFDWinGold
3.2M Popularity

Pinned

Sitemap

The Dark Side of the Moon and Tsinghua's New Paper: LLM Pre-Filling Can Cross Data Centers, 1T Model Throughput Increased by 54%

Trending Topics

WinGoldBarsWithGrowthPoints

WTICrudeFallsBelow90Dollars

StockTradingChallengeUpTo17000U

USIranNegotiationGame

TradeCFDWinGold

Pinned