正文 · AI 翻译
Parse Everything
Jay reporting from Aofei Temple
QbitAI | WeChat official account QbitAI
Jay reporting from Aofei Temple
QbitAI | WeChat official account QbitAI
Wait, RSI applied to OCR works this well??
Recently, a Chinese open-source model with just 1.3B parameters has blown document parsing out of the water.
First place on all three: OmniDocBench, olmOCR-Bench, and DocParsingBench.
Enough about benchmark scores; let's look straight at what it can do.
Old-school handwritten advanced math notes, converted to MD in one click.
It can even draw chemical structure diagrams...
It's absurd—it can parse just about anything.
What I didn't expect was for it to be incredibly fast, too???
On a single RTX PRO 6000, end-to-end parsing of 100 document pages takes just 101 seconds.
Uh-oh, Company A's efficiency at distilling printed books is about to soar (just kidding).
It's also caught fire in overseas developer communities, already earning 1.1K+ GitHub stars.
At one point, it even became the hottest Daily News item on Twitter.
And behind all of this is the hottest concept in AI right now—
RSI.
This new, self-evolving breed of OCR is called OXR.
Optical “Everything” Recognition.
Document parsing is a mountain that may not sound very technical, but still hasn't been fully conquered.
For traditional OCR, a seemingly ordinary PDF can often turn into a complete disaster.
Two-column papers get read in the wrong sequence, handwritten formulas turn into gibberish, large tables spanning pages lose their structure entirely, and many tools simply discard chemical structures as random lines...
Feed extraction this poor into RAG, and no matter how smart the model is, all the downstream tool calls grind to a halt.
Now, the barrier preventing AI from getting high-quality context has finally been smashed.
OmniDocBench: 97.18, first place (open-sourced by the Shanghai AI Laboratory; covers 10 types of complex documents)
olmOCR-Bench: 87.84, first place (released by the team at the well-known overseas AI organization Ai2; more than 7000 tough, unit-test-level questions)
DocParsingBench: 87.76, first place (#1; covers 5 major industries and 14 real business scenarios)
An open-source document parsing model called OXR has topped all three world-class leaderboards.
And the entire evaluation process and code are fully open-source and reproducible.
I gave it a quick try myself, and it really is different. Finally, some AI that isn't artificially stupid.
With earlier tools, I basically only expected them to turn images into text; I left the rest for AI to infer on its own.
OXR, though, can handle 13 types of document elements in one go:
Tables, formulas, code blocks, images, chemical equations, footnotes, headers, footers, stamps...
Here's what that looks like in specific use cases—
Magazines: output structured MD that preserves the full semantic structure and reading order.
Math and science formulas: convert them directly into standard LaTeX code.
Research papers?
No need to make an LLM slowly call tools to read them itself; even a two-column layout doesn't affect quality.
It can even parse math exam papers???
Looks like GPT will be better at helping elementary school students finish their summer homework from now on o(╥﹏╥)o.
Of course, what's most exciting is that it's open-source.
You have to realize that open-sourcing small, specialized models like this means something quite different from open-sourcing LLMs.
Honestly, open-source LLMs are great, but they have little to do with ordinary users.
Even a model with only 27B parameters is absurdly expensive to run locally; you'd need at least an M5 Pro with 64G, and that's before considering output speed.
But this OXR thing is just 1.3B...
What does that mean?
With an RTX 3060, you can basically run it at full capacity and use it as much as you like.
Overseas developers are delighted, too.
Pretty impressive.
So how does a 1.3B document parsing model achieve state-of-the-art accuracy?
The secret lies entirely in post-training—
It's actually RSI???
For a long time, everyone has held the same assumption:
For AI to achieve the highest possible precision, you have to pile on models with tens of billions of parameters and vast amounts of human-labeled data.
But honestly, in document parsing, labeled data is a bottomless pit.
Layouts vary wildly, and specialized formulas, code, and charts are intricate. Relying entirely on human labeling is not only enormously expensive; the human labeling error rate itself is frighteningly high.
You can never exhaust all the long-tail bad cases in the real world. Engineers painstakingly pick out bad samples, correct them by hand, and retrain. They finally release a new version, only for another bizarre layout to break it.
The pace of data preparation keeps a stranglehold on the model's evolution.
The team explored a fairly aggressive approach to this.
They realized that document parsing has long had an easily overlooked characteristic:
Every input has an objectively determined correct answer.
Whether the text is right, whether table cells span the correct rows and columns, and whether formula symbols are missing can all be measured precisely.
If you can judge right and wrong objectively, why wait for humans to clean up the mess?
After this thought experiment, OXR made a decision:
Introduce a recursive self-improvement (RSI) training paradigm for document parsing during post-training.
Without relying on additional manual human labeling, let the AI system find its own mistakes, analyze the causes, prepare the next round of training data, and iteratively improve itself.
This RSI process can roughly be divided into three stages:
1. Prediction and Classification
OXR first parses a vast number of real, unlabeled documents.
Next, a general vision-language model (General VLM) steps in and uses the original image, the parsing result, and the evaluation criteria to divide samples into three categories:
Correct: completely right; pass directly.
Wrong: clearly incorrect; send for in-depth analysis.
Uncertain: predictions are unstable or lie at the edge of the distribution; send for analysis as well.
Keeping the Uncertain category is a somewhat counterintuitive piece of know-how.
If you give the judging model only the black-and-white choices of Correct and Wrong, its hesitation gets forced into a definitive label, and you end up feeding that poison into the next training round.
With Uncertain, samples that can't be judged confidently have a place to go.
Besides using the VLM, OXR also performs multiple rollout samples of its own outputs.
If repeated runs on one document produce wildly different results, the model lacks confidence. Even if it happens to guess right once, the document goes straight into the Uncertain set.
Finally, while keeping the meaning entirely unchanged, the system applies slight image perturbations to these high-disagreement samples (minor scaling, compression, slight blurring, or translation).
If disagreement grows after a perturbation, that shows the model is overly sensitive to this kind of harmless visual change. The perturbation is recorded on the spot as a key cause and directly guides targeted improvements in the next round.
2. Attribution and Training Planning
Once you've identified incorrect and weak samples, you can't just feed them indiscriminately into the model. You have to understand exactly where it fails.
At this stage, the General VLM diagnoses samples one by one like an experienced doctor: did it fail to align cells spanning columns, or misread connected handwriting?
The natural-language reasons from the diagnoses are fed into SigLIP2 to extract feature vectors, then clustered by a separate clustering model into clear groups of causes (Reason clusters).
For example, a financial statement and a chemical experiment log may look and read nothing alike. But if attribution clustering finds that both fail on a particular rare structure spanning rows and columns, that becomes a shared direction for constructing data.
That's targeted intervention: strengthen whatever is weak.
3. Data Generation
With clear clusters of causes, the generator knows what kinds of problems to practice.
OXR's specific approach is this: cause clusters supply structural challenges, while the pool of real samples supplies layout and visual style references.
Whenever new data is generated, the system takes a layout style from real samples as an anchor and naturally incorporates the structural challenge it needs to overcome.
Visual weaknesses found in the earlier perturbation tests are also systematically built into the synthesis pipeline, forcing the model to repeatedly become less sensitive to them through practice.
Finally, the generated data undergoes strict standards validation and feeds directly back into training the next generation of OXR.
After one round, the model's weaknesses are addressed. Run another batch of real documents, and new weaknesses emerge, driving the next iteration.
And so RSI takes shape.
The data distribution becomes more complete and more robust with every iteration.
That's also why OXR can achieve such high intelligence density.
Once data quality reaches a certain level, the model no longer needs bloated parameter counts for rote memorization.
Having covered the data and algorithm, let's look at the next question.
I can understand it ranking first in accuracy.
But how is it also first in speed???
On a single RTX PRO 6000, OXR parses 100 document pages end to end in just 101.0 seconds, ranking first among open-source models.
At first, I thought it was because the model was small.
But I later realized that a small model alone isn't enough for document parsing.
There's a lot of counterintuitive know-how behind this.
Looking back at the technical evolution of document parsing over the past 2 years, the industry has actually gone through several stages.
At first, everyone built pipelines, using a separate specialized model for detection, recognition, and ordering. Put together, they were extremely bulky and enormously costly to maintain.
Then VLMs took off. People realized that all these tasks are essentially mappings from images to sequences, so they began trying to handle everything with an end-to-end large model, having it generate the text and layout for an entire page directly.
Although elegant, this approach is slow as a snail, because autoregressively generating long sequences demands so much compute.
Later, PaddleOCR-VL made a smart move:
It combined layout analysis (LA) and reading order (RO) in one small model, followed by a recognition model, creating a two-model, two-stage setup.
This brought layout detection back to the millisecond-level speed of traditional object detection and directly became the industry's standard solution over the past 2 years.
But was it perfect?
No. Every additional layout annotation has to be paired with a reading-order annotation, directly doubling the difficulty of data processing.
After working through the problem in depth, the OXR team proposed a completely different way to restructure the tasks:
A separate lightweight model handles layout analysis (LA), while the main OXR model handles both content recognition (Recognition) and reading-order decisions (RO).
In other words:
Layout Analysis–>Recognition+Reading Order
This seemingly simple division and combination brings 2 major changes in engineering terms:
First, it completely eliminates the performance cost of determining reading order.
Throughout the pipeline, determining reading order already took far less time than text recognition.
After OXR folded reading-order decisions into the main model, it added no extra time. Instead, because the main model has content-aware semantic understanding, reading-order accuracy improved substantially.
At the same time, freed from the complicated burden of ordering, the earlier layout analysis model (LA) has a very focused task and can use a lighter, more efficient detection model, sending overall throughput soaring.
Second, the layout analysis module is independent and fully interchangeable.
Different businesses have completely conflicting layout needs. This is a long-tail problem:
For an area mixing text and images, a content team wants to extract it as a single image, while a knowledge-base team wants to pull out the text as body copy;
For a densely arranged 4-panel comic, business A wants to crop it as a whole, while business B wants to split it into 4 smaller images;
...
In short, it's very hard to define in a way that satisfies everyone.
But in OXR's architecture, layout analysis is a completely independent, interchangeable module—
enabling maximum freedom to adapt to business needs with minimal modification costs.
If your business has highly specialized layout segmentation needs, such as processing ancient texts or medical forms, you can seamlessly swap in your own privately trained layout analysis model,
and still use OXR's powerful content recognition capabilities on the back end for free.
Of course, since it was built for the developer community, no matter how impressive a model sounds, it's useless if you can't deploy it.
On that front, the team provides a free API that requires neither registration nor an API key application.
Yes, completely free. You can just use it for nothing...
The team says it will keep maintaining the free, open-source service at least as long as the server load remains manageable.
How could I pass up an opportunity like that!
So I hurried to integrate the API into my writing app.
Honestly, its PDF parsing is vastly better than the solution I used before.
And if you want to deploy it locally, you can do that too. OXR supports the major inference frameworks.
You don't have to go through the trouble of recompiling vLLM yourself. Just mount the plugin in the official Docker image or CLI, and you can start the service with one command.
OXR's code, weights, and the full procedure for reproducing its benchmark evaluations are now public.
The links are at the end of this article for anyone interested.
Of course, if you have more demanding needs, such as processing additional formats including Word, PPT, and Excel, or want more extensive post-processing customization, you can also look into the commercial-grade engine maintained by the team.
It's called SoMark.
That's also the name of the team behind OXR.
Finally, I have a few thoughts I'd like to share.
Why are we still putting so much effort into tackling document parsing today?
At a time when agents are taking over workflows across industries, most of the most essential, specialized private knowledge accumulated by people over the past several decades still lies quietly in PDFs, scans, and reports with inconsistent formatting and complex layouts.
No matter how smart a large language model is, if you feed it a jumble of characters with the order scrambled and tables broken apart, all it can produce is hallucinations.
In the past, many people treated OCR as low-skill grunt work, but today, from OCR to OXR, its mission has changed completely.
It is a bridge between humans and machines, reconstructing all the complex multimodal documents in the human world, fully and without loss, as structured assets that agents can readily use.
Honestly, it's exciting to see a team in China working on things so closely tied to practical deployment.
Too many people focus on fancy narratives, chasing cutting-edge technology every day, and in the process we've forgotten something important—
What do users actually need?
At this stage of LLM development, the answer is definitely not more model intelligence.
The bigger bottleneck lies in areas no one has yet explored, where people need to get their hands dirty.
If we can reach a consensus on this—
the pieces of the infrastructure for agents will visibly fall into place, one by one.
GitHub: https://github.com/SoMarkAI/OXR
Hugging Face: https://huggingface.co/SoMarkAI/OXR-1.0
Online demo:
https://free-oxr.somark.cn/
Free API:
https://free-oxr.somark.cn/api/v1
SoMark official website: https://www.somark.cn