Original post · Original
The article text is unavailable in this language; an existing version is shown.
New Tencent paper shows domain skills (like math or coding ) in Mixture of Experts models sit in scattered neurons across expert.
so cutting weak neurons instead of whole experts shrinks the models while keeping most skills.
In their experiment, Mixture of Experts AI models kept 91% to 98% of their performance after their least useful 25% of expert neurons were cut.
MoE models send each token to a few experts, which suggests each expert handles a domain like math or code. Other work finds routing mostly tracks token type instead.
They scored each neuron's routed contribution with a measure called RAMP. Neurons picked for one domain overlapped 58% to 74% across samples of that domain, versus 15% to 24% across domains, and sat in mixed experts.
RAMPr prunes by these scores across all experts and kept 91% to 98% of original performance with 75% of neurons left, across 6 models. With 50% left, GPU memory fell 36% to 43% and batched prefill throughput rose 17% to 39%.
If you compress MoE models, prune neurons inside experts by their routed contribution instead of treating each expert as a single unit.