Ai2 introduces StateBench to test how language models track changing states in long documentsMachine translation

Ai2 introduced StateBench, which divides state tracking in long documents into four difficulty levels and compares model architectures on corresponding tasks. In its tests, four frontier models’ accuracy on the hardest tasks fell sharply as inputs grew, reaching 13% or less on the longest inputs; recurrent and hybrid models trained directly on the tasks handled much longer sequences than those used in training at some levels. The framework currently assumes a fixed number of variables.