正文 · AI 翻译
Noah reporting from Aofei Temple
QbitAI | WeChat public account QbitAI
Noah reporting from Aofei Temple
QbitAI | WeChat public account QbitAI
Reinforcement learning environments (RL Envs) are becoming a new business.
As large models move toward tool use, software operation, and multistep tasks, the way models are trained is also changing. Models no longer learn only from existing data and answers; they also need to enter specific tasks, take actions, observe the results, and continually adjust their strategies based on feedback.
For model teams, algorithms determine how a model learns, while environments determine what it can practice, how many attempts it can make, and how to judge whether an attempt was effective. Finding enough tasks that are realistic and have assessable outcomes is becoming an important part of post-training.
The market has already shown how strong this demand is. Mercor, Surge AI, Handshake... Over the past year, a group of new companies and businesses focused on large-model training and post-training have rapidly come to Silicon Valley's attention.
On November 10, Mercor announced that it had completed a $750 million Series D funding round at a valuation of $20 billion. The company operates an expert network of more than 5 million people, supplying evaluations and post-training data to frontier labs and enterprises. Earlier this year, Mercor also acquired Deeptune, a team that spent 2 years rebuilding hundreds of enterprise applications as training environments.
Surge AI, meanwhile, demonstrated the link between task environments and model training. The company built RL environments around long-horizon office tasks such as documents, spreadsheets, web research, planning, and tool use, and used Qwen3.5-122B-A10B for post-training. The model subsequently improved by 5.7 percentage points on SWE-Bench Pro, which it had never seen before.
Handshake AI focuses on experts and scenarios, providing frontier labs with expert data, RL environments, and graders across areas including finance, life sciences, and alignment. According to its CEO, as cited by Time magazine, this business reached $1 billion in annualized revenue about 1 year after launch.
These companies' business models are not exactly the same: some focus on specialized tasks, some provide training environments, and some combine expert experience, environment construction, and evaluation.
But they point to the same shift: the core inputs needed for model post-training have expanded from algorithms and computing power to training data and environments—“what tasks to do, where to practice, and how to judge the results.”
As models begin to take on long-horizon tasks, the role of environments becomes increasingly concrete: they determine which jobs models can practice, what results their actions produce, and whether training can continue to iterate.
When these environments remain in the digital world, models operate software, modify files, and run code. The changes caused by their actions can usually be recorded, and the results are relatively easy to judge.
But when models move from screens into the physical world, training environments extend from digital tasks to physical interaction, which will undoubtedly be more complex.
Progress in large-model capabilities is changing what is imaginable in robotics research and development.
On September 3, OpenAI released GPT-6 Astra, positioning it as a frontier model for computer use and programming.
Within 1 month, third-party evaluations began to emerge: across RoboDojo's 42 simulation tasks, Astra achieved an average success rate of 22.48% without any robotics-specific fine-tuning, surpassing all 40 trained robot policies on the leaderboard.
Investor Freda Duan posted on X that Astra was a “GPT-3 moment for robotics”; former OpenAI researcher Will Depue put it more directly: specialized models had learned a “bitter lesson.”
Of course, OpenAI has never officially published robotics scores, and tests on physical robots also show clear strengths and weaknesses: placing a block in a bowl succeeded 19 times out of 20, while a precision peg-insertion task succeeded only 2 times out of 20.
The shift these discussions point to is real: advances by general-purpose models in reasoning, tool use, and multistep tasks have turned the idea of “foundation models entering the physical world directly” from a vision into reality.
And in moving from the digital world to the physical world, reinforcement learning training faces a different kind of feedback.
In digital environments, a software agent's actions change files, web pages, or program states, and the results are usually clear. But a robot's actions affect an object's position, orientation, and contact state; where and how forcefully it makes contact, and the result of that contact, also affect subsequent operations. Therefore, besides understanding instructions and planning a workflow, a robot must continually adjust its actions during interaction.
This also means that the RL Envs needed for physical AI cannot simply replicate digital environments. They must support specific work tasks, let robots act repeatedly from different initial states, and make key objects respond credibly to the policy's actions.
The choice of training venues narrows accordingly. Trial and error on physical robots is constrained by equipment, task resets, and on-site conditions, making it difficult to scale. Simulation can provide repeatable training conditions, provided that the environment can support specific tasks and continually give feedback on a policy's actions.
Building physically realistic RL Envs has therefore become an essential path for foundation models entering the physical world.
During CoRL2026, Guanglun Intelligence released RoboFinals RL Environments (hereafter, RoboFinals RL Envs), providing interactive reinforcement learning simulation environments for robot models.
Unlike traditional training scenarios designed around a single action, RoboFinals RL Envs is oriented toward real work tasks. Its initial demonstration featured a long-horizon simulation task lasting more than 8 minutes: extended automotive repair work.
This task brings dexterous-hand manipulation, tool use, multistage workflows, and interaction with flexible objects into the same work scenario, involving different kinds of objects such as hoses, cables, and gaskets.
The complexity it presents lies not only in the precision of individual actions, but also in how multiple physical interactions become coupled within the same task sequence.
First, dexterous manipulation: degrees of freedom and contact points multiply.
Most robot manipulation tasks use parallel grippers, which have only 1 degree of freedom for opening and closing. Dexterous hands usually have more than a dozen to over 20 degrees of freedom, with each finger moving independently.
For the policy, the dimensionality of the action space rises sharply. For simulation, multiple contact points exist simultaneously between the fingers and the object, and the forces at each point must be solved. If contact is calculated inaccurately, the object will slip or be penetrated, and the grasping method the policy learns will not work in the real world.
Second, long-horizon tasks: errors accumulate over time.
Previously, common task designs in robot reinforcement learning environments involved relatively independent actions such as grasping and placing.
In Pick-and-Place, for example, a robot must grasp an object and put it in a designated location. But actual work environments usually involve tasks made up of multiple sequences of actions. To enter real workplaces, robots inevitably need training on long-horizon tasks.
The longer the task, the harder the training. The effects of early actions may appear only much later, while small deviations at each step continue to accumulate.
This is an important distinction between a long-horizon task and a simple concatenation of multiple independent tasks.
A common approach is to split a long task into several segments, each starting from a manually set initial state. This reduces the difficulty, but also removes a key condition of real work: the robot does not actually have to deal with the state changes caused by its preceding action.
In RoboFinals RL Envs tasks, earlier and later stages are linked. Changes in an object's position, orientation, and contact state caused by an earlier step carry over to subsequent steps, so the policy must learn to complete the task from imperfect intermediate states.
Third, interaction with deformable objects: states are hard to describe and parameters are hard to determine.
Hoses, cables, and gaskets are deformable soft bodies. In simulation, they must be discretized into many nodes and calculated one by one, placing far greater demands on computation and numerical stability than rigid bodies do.
Their behavior also depends on parameters such as stiffness, damping, and friction. Slight errors in those parameters can make soft bodies appear too soft or too rigid in simulation, or produce jitter that does not exist in the real world. The shape response after interaction with a soft body directly affects the later course of the task.
The fingers of a dexterous hand must make contact with objects; tool use involves precise control of position and orientation; a hose bends and swings when pulled; and a gasket deforms under pressure.
These objects have different physical properties and modes of interaction. Showing the motion of any one of them on its own is not difficult. The challenge is to place them in the same task environment so that a robot can interact with different objects during continuous operations.
The focus of Guanglun's demonstration is not just an automotive repair simulation scenario, but a robot reinforcement learning environment that combines precise manipulation, complex object interactions, and long-horizon tasks. Achieving all of these at once requires starting with the underlying systems.
Guanglun Intelligence's fully self-developed simulation platform, SimFoundry, covers 3 stages: “solving, measurement, and generation”:
The solver is responsible for accurately calculating interactions among rigid bodies, articulated bodies, and soft bodies;
The measurement stage ensures that objects' physical parameters come from real data;
The generation stage is driven by an AI Agent, which uses real tasks to reconstruct assets, model their physics, and define tasks. With an Agent driving the generation pipeline, real-world scenarios and tasks can enter simulation in batches.
Guanglun Intelligence calls this set of standards RL Ready: semantically, each task stage has a clear criterion for completion; in physical interaction, feedback under trial-and-error conditions is reliable; and the feedback signals can be used directly for training. Once connected, a model can begin trial and error and iteration.
The reinforcement learning environments needed for physical AI place demanding requirements on those who build them:
Assets must be visually realistic and physically realistic. Building a high-quality simulation environment requires high-quality assets, a high-precision physics solver, and expert experience. Without any one of these 3, long-horizon tasks will stall.
The problem is that it is difficult for 1 team to bring all 3 capabilities together.
Asset makers do not work on solvers; sellers of simulation software do not have real-world scenarios on hand; and model teams know what counts as correct but have little incentive to build environments.
Anyone can piece together a one-off demo, but after assembling it, they find it cannot keep running. This is exactly the bottleneck that has kept long-horizon RL Envs from being released.
What distinguishes Guanglun Intelligence is that it has all 3 capabilities in-house.
As the only Chinese company on the Newton simulation technical committee, Guanglun Intelligence helps define the SimReady simulation standard and is a core contributor to the open-source SimReady Central Library of simulation resources.
In previously demonstrated simulation tasks such as plugging and unplugging cables and grasping plastic bags, the team showed its ability to handle the deformation, contact, and state changes of flexible objects, which are far more difficult than rigid bodies.
Although the objects have changed in the automotive repair task, it addresses the same kind of problem. Assets, scenarios, and a self-developed solver are core capabilities the company has refined bit by bit over the years.
Having all the capabilities is not enough; they must work together to produce a trainable environment.
That is what the Agentic calibration workflow does: the Agent proposes directions, compares results, and organizes the direction of the next round of corrections, while Guanglun Intelligence's simulation experts make decisions at key points and continually refine the process.
A specific case from this release illustrates how the workflow operates.
In the automotive repair task, the bending and swinging of an inflation device's hose when pulled affects the robot's operation, and the initial simulation showed abnormal jitter.
Through successive rounds of automated calibration, the Agent gradually identified Young's modulus as the parameter with the greatest effect on the hose's response. Once candidate parameters and trajectories were produced, human experts compared the simulated bending and swinging with the behavior of the actual hose to help complete the final calibration.
In the example, the maximum positional deviation shrank by about 3 millimeters, the previously obvious jitter was also reduced, and the hose's response became closer to reality.
With a clearly defined target for physical parameter adjustment and results that can be compared, this provides a basis for reproducible automated calibration work.
Whether the ability to build environments can scale will determine how far the environment library can go.
Lightwheel Intelligence plans for RL Envs to support a wider range of robot tasks across different levels of difficulty and task durations, with the environment library's distribution following that of real-world tasks. Scene types cover homes, industry, retail, healthcare and more; operation types and difficulty range from grasping and placing to alignment, tool use and force control, and then to precise closed-loop operations; asset types include rigid bodies, articulated bodies, soft bodies and fluids.
Once the environments are built, it is the model team's turn to ask: How do I know it has learned?
Training environments need a corresponding set of evaluations. RoboFinals RL Envs and RoboFinals Benchmark share task semantics, scene assets and capability definitions, so training and evaluation use the same task requirements. Training tasks, the process of tuning rewards, and evaluation instances are kept strictly separate, leaving an independent check on learning outcomes.
In the industry, what can be used for training often cannot be used for evaluation, while those that can evaluate do not have environments of their own. A design with a common foundation is possible only within the same product line.
The final layer is delivery.
Open-source tools have lowered the barrier to starting simulations, but the work of adapting assets, calibrating physics and checking runs remains.
Lightwheel has packaged these into a deliverable product, so model teams can connect their policies and start training. Environment capabilities are no longer “work assembled internally by a particular team” but standard components that can be purchased and reused.
In the industry, some make assets, open-source solvers are available, and organizations also conduct evaluations. Viewed separately, none of these is exclusive. Lightwheel Intelligence has been the first to complete the entire path from “environment construction–calibration–evaluation–delivery.”
The value of RL Envs must ultimately be reflected in whether a policy can continue acting in an environment and adjust its behavior based on the results of each interaction.
At this launch, Lightwheel Intelligence also showed how the same task changed at different stages of policy iteration:
In the first stage (5 steps), the robotic hand's movements were still rather shaky, and it had not completed the grasp;
In the second stage (150 steps), the policy had begun attempting to grasp the screw, but the screw slipped during the operation;
In the third stage (175 steps), the policy successfully picked up the screw, completing this step of the task.
This shows that RoboFinals RL Env can support trial and error and behavioral changes during policy training, allowing the iterative changes in model performance to be observed and recorded in a repeatable simulation task.
The development of large models has already shown that improvements in model capability depend not only on the algorithms themselves but are also closely related to training tasks, interaction conditions and feedback mechanisms.
For robots, this issue is more complex.
As models move from completing a single action to carrying out multistage workflows, environments must not only support the tasks but also handle fine-grained operations, physical interactions, continuous state changes and long-term feedback. What robots can practice and how they receive feedback are becoming factors that affect training outcomes.
Lightwheel Intelligence had previously built end-to-end capabilities around physical AI spanning “data–evaluation–deployment.” The newly launched RoboFinals RL Envs further extend its product capabilities into interactive training environments.
Built on the SimFoundry simulation platform, Lightwheel organizes complex tasks, object assets and interaction rules into runnable simulation environments, enabling models to directly control robots to perform tasks in the environments, observe changes in state and explore different policies through repeated attempts.
For complex operational tasks where post-training data is difficult to obtain at scale, this also offers a new way to acquire learning experience.
After training, models need evaluation and diagnosis to identify directions for the next round of capability improvements. RoboFinals RL Envs and RoboFinals Benchmark serve policy training and capability evaluation, respectively. Models can interact and optimize their policies in the environments, then undergo independent evaluation to test task performance and identify capability limits.
From data supply and simulation technology to evaluation at scale and reinforcement learning environments, Lightwheel Intelligence's product portfolio also reflects changing needs for physical AI infrastructure: expanding from the data and evaluation resources needed for model development to ongoing interaction between models and the physical environment.
As robots begin to take on more complex, more continuous work tasks, how to build learning environments that can support those tasks is also becoming an important question for physical AI infrastructure to answer.