Kimi's move to decouple across data centers is quite impressive. If the inference costs can really be brought down, future large model invocation fees might become dirt cheap.

ME News Report, April 18 (UTC+8), the Moonshot AI team recently announced that their decoupling technology for Prefill and Decode has successfully expanded from a single cluster to cross-data center and heterogeneous hardware environments. According to the article, this move is expected to significantly reduce the inference cost per token. Previously, the expansion of this technology was hindered by KV cache transmission overhead issues. The breakthrough was made possible primarily due to their hybrid model Kimi.