aifollow.news
Back Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · Original publication time AI score54

Study finds that low model capability scores can be precisely manufacturedMachine translation

AI introduction

A study used a LoRA adjustment to steer a Qwen model toward low target scores on unseen tests, coming within 1.9 points; prompting models to underperform missed by 42 to 47 points. The adjustment left a telltale pattern: on multiple-choice questions, up to 97% of its new errors fell on one answer letter. The study suggests checking which items a model misses and how its errors are distributed before treating a low score as evidence of low dangerous capability.

Original post · Original

@rohanpaul_ai

This paper finds that a low benchmark score can be manufactured precisely, so a low score alone says little about what a model can do.

Safety tests often read a low score on a dangerous-capability probe as reassurance. But a low score can come from a model being held back, so the number alone proves little.

They trained a LoRA "dial" that pulls a Qwen model toward a smaller Qwen model, then compared its errors with the real small model's. They also tried prompting models to underperform, raising temperature, and adding weight noise.

The dial hit unseen target scores within 1.9 points, while prompting missed by 42 to 47 points. But on multiple choice, up to 97% of its new errors landed on 1 letter, a share no natural model in their pool approached.

Before reading a low score as low capability, check which items the model fails and how its wrong answers are spread.

Source: Rohan Paul · x.com

Research
Found an error? Send a correction