Original post · AI translation
标题：“分词器的迷思及其出处”
斯坦福大学的一篇新论文发现，没有任何指标能在所有意义上都不受分词器影响，因此比较分词器的人不应只相信单一分数，还应检查模型是否正确作答。
为了比较使用不同分词器的模型，研究人员通常报告每字节比特数（BPB），即按字节而非词元计算损失。许多论文称它不受分词器影响。
他们给未经训练、进行均匀随机猜测的模型，以及采用相同训练方案但使用不同分词器的已发布模型打分。他们还重新核查了一项已发表的说法，即另一种据称不受分词器影响的指标与 BPB 一致。
即使模型什么也没学到，其 BPB 仍会因分词器而异；对英语文本，词汇量接近 24K 时的分数最佳，但准确率仍处于随机猜测水平。在使用不同分词器训练的模型中，BPB 与任务得分之间的相关系数仅在 +0.02 至 −0.30 之间。
比较分词器时，不要把较低的 BPB 当作模型更好的证据；在选择词汇表前，还应在相同文本上检查任务准确率。
Original
New Stanford paper finds that no metric can be tokenizer-agnostic in every sense, so anyone comparing tokenizers should not trust a single score and should also check whether the model answers correctly. To compare models with different tokenizers, researchers usually report bits-per-byte, or BPB, which divides the loss by bytes instead of tokens. Many papers call it tokenizer-agnostic. They scored untrained models that guess uniformly, plus released models that share one recipe but use different tokenizers. They also rechecked a published claim that another supposedly tokenizer-neutral metric agrees with BPB. Models that learned nothing still got different BPB by tokenizer, with English text scoring best near a 24K vocabulary, while accuracy stayed at chance. On models trained with different tokenizers, correlations between BPB and task scores ran only from +0.02 to −0.30. When comparing tokenizers, don't treat lower BPB as proof of a better model, and check task accuracy on the same text before picking a vocabulary.