原帖引用 · AI 翻译
标题：“为什么推测解码在稀疏 MoE 模型上的效果好于预期”
@cohere 的一篇新论文发现，即使在此前研究预测收益较小的批量大小为 1 的情况下，推测解码在细粒度 MoE 模型上仍表现良好。
研究发现，在混合专家模型中，相邻 token 有 20% 至 45% 的专家是共用的，而预期比例为 4% 至 12%，这使得推测解码的成本低于预测。
推测解码让一个小型草稿模型提出 token，再由大模型用 1 次前向计算进行核验。此前的研究认为，对于 MoE 模型，核验多个 token 会加载许多额外的专家，从而限制低批量大小下的收益。
他们测试了参数量从 30B 到 1T 的 10 个 MoE 模型，并在 H100 GPU 上分析了它们的性能。连续的 token 会复用许多专家，而小型专家能充分利用 GPU 带宽，因此核验额外 token 的成本很低。
相邻 token 有 20% 至 45% 的专家是共用的，而独立路由下的预期比例为 4% 至 12%。对于采用 FP8 的 Qwen3-235B，核验阶段的运行速度达到普通解码步骤的 89%，而预测值为 52%。
对于批量大小较低的 MoE 服务场景，例如智能体工作负载，应在自己的硬件上对推测解码进行基准测试，而不要直接排除它。
原文
New @cohere paper finds that speculative decoding works well on fine-grained MoE models even at batch size 1, where prior work predicted small gains. Finds that in mixture-of-experts models, neighboring tokens share 20% to 45% of their experts, against 4% to 12% expected, which makes speculative decoding cheaper than predicted. Speculative decoding lets a small drafter propose tokens that the big model checks in 1 pass. For MoE models, prior work said checking several tokens loads many extra experts, limiting gains at low batch sizes. They tested 10 MoE models from 30B to 1T parameters and profiled them on H100 GPUs. Consecutive tokens reuse many experts, and small experts use GPU bandwidth well, so checking extra tokens costs little. Adjacent tokens shared 20% to 45% of their experts, against 4% to 12% expected from independent routing. For Qwen3-235B in FP8, the verification pass ran at 89% of a normal decoding step's speed, against 52% predicted. For low-batch MoE serving, such as agent workloads, benchmark speculative decoding on your hardware instead of ruling it out.