Original post · Original
The article text is unavailable in this language; an existing version is shown.
Rohan Paul@rohanpaul_ai
Title: "AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models"
People rate the exact same AI chatbots more harshly each month, so a chatbot that never changes starts to look worse.
They studied 1,260,967 blind votes on Chatbot Arena, where people compare answers from 2 anonymous models. They kept only pairs of frozen models, such as dated snapshots and fixed open weights, so the models never changed.
On the same pair, "both are bad" votes rose 1.71 percentage points a month, on a 17.1% average. Yet in 2024 the stronger model won about the same share of decisive votes, so rankings held.
View this post on X ↗
· UTC+8