Original post · AI translation
标题：“用于 RAG 声明核验的 Jev-as-a-Judge：在非生成式核验器与前沿大语言模型之间基于置信度门控的级联”
这篇论文发现，一款低成本核验器在核查 RAG 声明时，准确率可与前沿大语言模型相当，但会放过更多缺乏证据支持的声明。
RAG 流程需要反复判断证据是否支持某项声明，这是一个简单的是非问题。大多数团队却使用生成文本的大语言模型来回答，并为此支付生成文本的费用。
他们在 495 项经人工标注的声明上，将只返回标签和概率的 Jev 与 GPT-6 Astra 进行比较。他们还构建了一套级联系统：由 Jev 处理置信度高的声明，其余交给 Astra。
Jev 的准确率与 Astra 相差不到 0.6 个百分点，成本却低 187 倍；但它将 23.2% 缺乏证据支持的声明判为得到支持，而 Astra 的这一比例为 14.1%。级联系统也降低了 Astra 的费用，但随机将相同比例的声明交给 Astra，节省的费用也差不多。
在为 RAG 产品选择声明核验工具之前，应衡量它将缺乏证据支持的声明判为得到支持的频率，因为总体准确率会掩盖这一点。
Original
This paper finds that a cheap verifier can match a frontier LLM's accuracy at checking RAG claims, while letting more unsupported claims slip through. RAG pipelines keep asking whether evidence supports a claim, a simple yes or no. Most teams get that answer from a text-generating LLM at generation prices. They tested Jev, which returns only a label and a probability, against GPT-6 Astra on 495 human-labelled claims. They also built a cascade where Jev answers confident claims and passes the rest to Astra. Jev landed within 0.6 points of Astra's accuracy at 187 times lower cost, but it approved 23.2% of unsupported claims against Astra's 14.1%. The cascade also cut Astra's bill, yet sending the same share of claims to Astra at random saved about as much. Before picking a claim checker for a RAG product, measure how often it approves unsupported claims, since headline accuracy hides that.