Study finds users rate unchanged AI chatbots more harshly over timeMachine translation

People rate the exact same AI chatbots more harshly each month, so a chatbot that never changes starts to look worse. They studied 1,260,967 blind votes on Chatbot Arena, where people compare answers from 2 anonymous models. They kept only pairs of frozen models, such as dated snapshots and fixed open weights, so the models never changed. On the same pair, "both are bad" votes rose 1.71 percentage points a month, on a 17.1% average. Yet in 2024 the stronger model won about the same share of decisive votes, so rankings held.