Ai2 发布 XYBench，测试 AI 助手能否识别提问中的错误假设Chinese only
Ai2 发布包含 8,115 道真实场景问题的 XYBench，评估助手能否从用户提出的做法中识别错误假设，转而解决其实际目标。在单轮测试中，五款模型都比人类参考答案更少引导用户放弃错误做法；该基准目前仅涵盖英语提问，且以操作类问题为主。
1 developments2 reports2 sources
Event developments 1
Ai2 releases XYBench to test whether AI assistants spot mistaken assumptions in questions
Ai2 has released XYBench, a benchmark of 8,115 real-world questions that tests whether assistants can identify mistaken assumptions in a user's proposed approach and address the underlying goal. In single-turn tests, all five evaluated models were less likely than human reference answers to steer users away from the mistaken approach. The benchmark currently covers English-language queries and mainly procedural problems.
View source reports · 2 sources · 2 reports
- Allen Institute for AI 研究动态 · Ai2 releases XYBench to test whether AI assistants spot mistaken assumptions in questions
- Hugging Face 官方博客 · Researchers release XYBench to test whether AI spots mistaken assumptions in user questions
All reports 2
Researchers release XYBench to test whether AI spots mistaken assumptions in user questions
XYBench contains 8,115 real-world questions testing whether assistants can move beyond a user's mistaken approach and address the underlying goal. In single-turn tests, five evaluated models were less likely than human reference answers to steer users away from that approach. The benchmark currently covers English-language queries and mainly procedural problems.
1 more source reported this
- Allen Institute for AI 研究动态
Ai2 releases XYBench to test whether AI assistants spot mistaken assumptions in questions
Ai2 has released XYBench, a benchmark of 8,115 real-world questions that tests whether assistants can identify mistaken assumptions in a user's proposed approach and address the underlying goal. In single-turn tests, all five evaluated models were less likely than human reference answers to steer users away from the mistaken approach. The benchmark currently covers English-language queries and mainly procedural problems.
1 more source reported this
- Hugging Face 官方博客