Article · AI translation
Instruction-following ability also improved.
By Luyu, reporting from Aofei Temple
QbitAI | WeChat public account QbitAI
By Luyu, reporting from Aofei Temple
QbitAI | WeChat public account QbitAI
GPT-6 Astra can now directly read and edit the weights of small models.
MIT has just conducted an experiment that instantly set the large-model community abuzz—
Astra was given the 0.8B-parameter “source god” Qwen3.5; by changing just 0.016% of its weights, it dramatically improved the model's performance.
No training or gradient descent was used throughout.
Under the same sampling settings, its MATH-500 score surged from 7.9% to 48.1%, making its mathematical ability about 6 times what it was.
Even its instruction-following ability, which was not originally targeted for improvement, rose 3-fold.
And Astra did more than fix a model: it also wrote the weights of a small game-playing network from scratch, with good results.
It submitted its answer in a single response and scored more than 20 times as high as a PPO baseline trained for 500 million steps.
The team also gave this approach a name:
Weight-Space Programming.
Astra radically rewrites model weights
Put simply, a large model directly analyzes neural-network weights, determines what the network is doing and where it is going wrong, then modifies it as requested or even writes a network from scratch.
For the specifics, start with the most straightforward experiment, involving Qwen.
The researchers gave Astra a Python environment containing all the weights of Qwen3.5-0.8B.
There was just one instruction: improve the model's mathematical ability as much as possible.
Ordinarily, training a model involves calculating a loss from the gap between its output and the target, obtaining gradients through backpropagation, and having an optimizer update the parameters.
This time, Astra had to analyze the model and decide exactly what to change and how.
It found that the small model's math performance was being dragged down substantially by reasoning loops.
Faced with a problem, it would repeatedly generate similar text without producing an answer. Of its first 96 practice responses, 85 ran all the way to the 32768-token length limit.
Qwen's own model card had even warned about this problem...
Tracing the failures, Astra located the loops in the feedforward network and changed the relevant weights in 64 output columns each in layers 15 and 19.
In the end, the proportion of MATH-500 responses hitting the length limit fell from 93.0% to 33.1%.
The researchers then gave Astra budgets of 10, 25 and 50 forward calls; the modified model's MATH-500 scores were 12.3%, 14.1% and 48.1%, respectively.
Astra dispensed with gradient-based training, but model diagnosis and validation were still part of the process.
So how did it find these clues in the weights?
First, consider its ability to read them.
The researchers trained 4 small networks to play different games, then gave Astra their raw weights as text. All 4 networks had identical architectures, input channels and action options.
Astra correctly identified the 4 games: Breakout, Pong, Pac-Man and Freeway.
For Breakout, Astra inferred the game from where the weights were concentrated.
With an image-generating diffusion model, it could also identify circles, squares and triangles from the weights.
It got one detail wrong, however: the model actually generated filled shapes, but Astra read them all as outlines.
It got the shapes right but missed the fill.
The researchers then raised the difficulty: they took 3 small language models with 7.2 million parameters each, trained respectively on Python code, romanized Japanese and English, and removed their input and output layers, leaving only the 4 middle Transformer blocks.
At that point, there was no way to know which number corresponded to which character, and the models could no longer be run directly on text. Astra used Python to analyze the weights, but could not feed data into the networks or their layers to observe their outputs.
Even so, within 2 attempts it correctly identified the languages learned by all 3 models and found every induction head.
The path from identifying a network's purpose, to locating its internal components, to changing its behavior was coming together.
Another experiment, in which a small car had to climb a hill, made the effect of modifying weights directly visible in its actions.
The network originally trained with PPO would always charge uphill and get stuck; to succeed, the car had to reverse to gain some distance, then accelerate up the slope.
Given the weights, a description of the scenario and a record of a failed attempt, Astra wrote a patch in a single response without using tools. By changing just 3.5% of the weights, it made the car succeed in all 1024 tests.
Astra wrote the sequence of actions that the model had failed to learn through training directly into its weights.
Is training obsolete?
Since Astra could add logic to an existing network, the researchers took the task a step further:
Given a network whose weights are all zero, write it directly into a usable one.
In the MinAtar game experiment, Astra received the game rules, source code and a specified network architecture. It had no training data, loss values, gradients or tools, and could not first run the network in the environment to try it out.
It had to devise a strategy itself, translate that strategy into the weights of each layer, and submit the result in a single response.
In the end, the network Astra generated achieved an average score of 517.4.
For comparison, PPO scored 15.8 after 100 million training steps and 25.3 after 500 million steps; Astra's result was more than 20 times higher.
What stands out about this work is that an existing general-purpose frontier model analyzed and rewrote weights itself to pursue a goal stated in natural language.
Astra's own knowledge and abilities still come from its earlier training; analyzing weights, generating code, running inference in batches and evaluating results also consume computing resources.
GPUs do not need to retire just yet.
But the next time a small model performs poorly, developers may genuinely be able to ask one more question first:
Does this require retraining, or will directly changing the weights do the job?
References: [1]
https://x.com/i/trending/2117328869283840499 [2]
https://rikiyat.github.io/blog/weight-space-programming/ [3]
https://x.com/rikiyatakehi/status/2117313742891327994