Ai2 releases XYBench to test whether AI assistants spot mistaken assumptions in questionsMachine translation

Ai2 has released XYBench, a benchmark of 8,115 real-world questions that tests whether assistants can identify mistaken assumptions in a user's proposed approach and address the underlying goal. In single-turn tests, all five evaluated models were less likely than human reference answers to steer users away from the mistaken approach. The benchmark currently covers English-language queries and mainly procedural problems.