Article · AI translation
spaCy creator blasts research in lengthy article
Jay reporting from Aofeisi
QbitAI | WeChat account QbitAI
Jay reporting from Aofeisi
QbitAI | WeChat account QbitAI
Astra edits a small model’s weights? That viral MIT study has been publicly challenged by a prominent expert.
Just a couple of days ago, Astra took the AI community by storm with a startling ability: directly editing another AI model’s weights without training or calculating gradients.
The MIT team claimed that by changing just 0.016% of Qwen3.5-0.8B’s weights, Astra boosted its math score from 7.9% to 48.1%, more than 6 times the original score.
One commenter even marveled that AI had begun to understand another AI’s brain.
But the story now appears to have taken a turn.
A prominent figure in the field has published an article raising doubts.
This study is seriously misleading.
This study is seriously misleading.
The critic is Matthew Honnibal, creator of spaCy and co-founder of the AI company Explosion.
A magician’s trick
So what exactly happened?
Honnibal used a pointed analogy to mock what the MIT researchers had done: sleight of hand.
Stage magicians often perform the same trick in several different ways.
The first time, a magician might use a special prop; the next time, they show you the prop is ordinary but conceal something on their person; the third time, they might use some form of forced choice.
As you watch, you assume there is only 1 technique, so each performance seems to disprove your theory of how the trick works.
Stage magicians often perform the same trick in several different ways.
The first time, a magician might use a special prop; the next time, they show you the prop is ordinary but conceal something on their person; the third time, they might use some form of forced choice.
As you watch, you assume there is only 1 technique, so each performance seems to disprove your theory of how the trick works.
Honnibal believes the way the Astra study presents its experiments closely resembles such a magic show.
Turning back to the paper, it actually demonstrates 3 different capabilities:
Reading weights: Identifying what task a small neural network is performing simply by looking at the raw weight values.
Writing weights: Directly generating or modifying the parameters of a small network from scratch, without feedback from the environment, so it can play simple games such as Breakout.
Repairing a large model: Greatly improving Qwen’s performance on math tests by modifying some of its weights.
But the paper’s abstract leaves a central impression—
that Astra can understand and modify another model “just by observing its raw weights.”
At a glance, it is hard not to take that to mean:
Both identifying a small network’s function and repairing a large language model’s math ability come from the same remarkable “mind-reading” technique.
But is that true?
After carefully reviewing the paper’s 8 sets of experiments, Honnibal found that the experimental conditions used to achieve these 3 things were vastly different.
In other words—
It is a blatant bait and switch.
2 technical approaches
So what exactly is going on across these different experiments?
First, in experiments such as identifying a small network’s task and repairing a stuck racing model, Astra really did work by “looking only at raw weights.”
That is because the experimental conditions were extremely strict:
The weight values (raw floats) were fed directly into Astra’s context as input.
Astra was not allowed to run Python code.
Astra was not allowed to execute the target network to observe its behavior.
In this mode, the models whose parameters Astra handled were all very small, ranging from 17,000 to 172,000 parameters. That is impressive, but it is still a long way from truly understanding a modern large language model.
But the most eye-catching and exciting experiment, repairing Qwen’s math ability, was an entirely different matter.
When repairing the Qwen model with 800 million parameters, Astra did not directly read the raw weights.
Instead, Astra was allowed to program in Python and run up to 50 forward-pass tests. In each test, it could have the model process a batch of practice problems, then analyze and modify the weights by observing the model’s performance.
These are two completely different things...
Honnibal pointedly observed that presenting these 2 methods together could easily lead readers to think:
Oh, that “mind-reading” technique demonstrated on small models can also be scaled up to large models with 800 million parameters.
But in fact, when working with a genuine large language model, Astra is more like a “sideline coach,” intervening through a series of engineering steps such as programming, running the model, and observing it.
The real value
Does this mean the MIT research has no value?
Of course not.
It must be acknowledged that, even without the aura of “mind reading,” Astra’s demonstrated abilities remain remarkable.
Honnibal also acknowledged that prompting an AI model to design and carry out tasks previously reserved for human machine learning engineers—writing code snippets to analyze and even directly modify network weights—is itself a huge advance.
AI models from 1 or 2 years ago would have failed at all these tasks.
AI models from 1 or 2 years ago would have failed at all these tasks.
This shows that today’s leading large models already have fairly sophisticated tool use and reasoning abilities, allowing them to act as “AI engineers” that diagnose and repair other models. That alone is an engineering breakthrough worth watching.
The real problem lies in how the results are presented.
They vaguely package 2 distinctly different capabilities as 1 unified “superpower” for reading and writing AI brains.
Honestly, at first I thought this sounded like something Company A had done...
It also reflects, indirectly, today’s AI hype frenzy.
Original article: https://honnibal.dev/blog/astra-read-write-misleading