Original post · AI translation
标题：“超越专家边界理解混合专家语言模型”
腾讯的一篇新论文表明，混合专家模型中的领域技能（如数学或编程）分布在不同专家中分散的神经元里。
因此，剪除作用较弱的神经元，而不是整个专家，可以在保留大部分技能的同时缩小模型。
在他们的实验中，混合专家 AI 模型剪除了作用最小的 25% 的专家神经元后，仍保留了原有性能的 91% 至 98%。
MoE 模型将每个 token 发送给少数几个专家，这似乎意味着每个专家负责数学或代码等一个领域。但其他研究发现，路由主要追踪的是 token 类型。
他们用一种名为 RAMP 的指标，对每个神经元经路由后的贡献进行评分。针对同一领域的不同样本选出的神经元有 58% 至 74% 重合，而跨领域的重合度为 15% 至 24%；这些神经元分布在混合的专家中。
RAMPr 根据这些评分在所有专家中剪枝。在 6 个模型中，保留 75% 的神经元后，模型仍保留了原始性能的 91% 至 98%。保留 50% 时，GPU 内存占用下降了 36% 至 43%，批量预填充吞吐量提高了 17% 至 39%。
如果要压缩 MoE 模型，应根据神经元经路由后的贡献来剪除专家内部的神经元，而不是将每个专家视为一个整体。
Original
New Tencent paper shows domain skills (like math or coding ) in Mixture of Experts models sit in scattered neurons across expert. so cutting weak neurons instead of whole experts shrinks the models while keeping most skills. In their experiment, Mixture of Experts AI models kept 91% to 98% of their performance after their least useful 25% of expert neurons were cut. MoE models send each token to a few experts, which suggests each expert handles a domain like math or code. Other work finds routing mostly tracks token type instead. They scored each neuron's routed contribution with a measure called RAMP. Neurons picked for one domain overlapped 58% to 74% across samples of that domain, versus 15% to 24% across domains, and sat in mixed experts. RAMPr prunes by these scores across all experts and kept 91% to 98% of original performance with 75% of neurons left, across 6 models. With 50% left, GPU memory fell 36% to 43% and batched prefill throughput rose 17% to 39%. If you compress MoE models, prune neurons inside experts by their routed contribution instead of treating each expert as a single unit.