Article · AI translation
Thanks to IT Home reader HH_KK for the tip!
IT Home reported on November 7 that, on November 4 local time, Apple's research team submitted a paper titled “Perceptive Humanoid Motion from Simulation” to the preprint platform arXiv, revealing HALO, an autonomous control system for humanoid robots.
The research team also released code for data generation, model training, and deployment, covering motion generation, motion tracking, task orchestration, and operation in simulation and on real robots.
For real-world deployment, the research team used a Unitree Robotics G1 humanoid robot fitted with a custom 3D-printed protective enclosure on its back. Inside was a Mac mini (M4 Pro chip, 64GB of unified memory), which handled real-time, computationally intensive inference for the Orchestrator (VLM) and Motion Generator (DiT) on the robot; the G1's built-in onboard chip focused on running the Tracker (RL policy network) at high frequency.
When carrying out tasks, HALO is controlled through coordination among 3 layers. At the top is a task orchestration module based on a vision-language model (VLM), which uses Alibaba's Qwen3.6-35B-A3B by default; the system can also call a cloud-based model through an API. The middle motion-generation layer uses another Qwen model, Qwen3.5-2B. At the bottom is a motion tracker trained through reinforcement learning, policy distillation, and other steps, running at a frequency of 50 times per second.
According to the paper, the research team conducted 360 single-task tests across 6 real-world environments that had not been scanned or mapped in advance, achieving an average success rate of 91%.
For long-sequence composite tasks, the team conducted 10 tests of each combination in a single real-world setting, with the 3 combinations averaging a 70% success rate. The most complex sequence, “bend down to pass an obstacle — walk toward the target — push a box,” succeeded in 5 of 10 tests, a success rate of 50%.
According to the team, it used 250 Nvidia H100 GPUs for about 48 hours to synthesize the main training data. The motion generator was trained through imitation learning using 64 B200 GPUs; the motion tracker was trained through reinforcement learning, policy distillation, and other steps using 4 H100 GPUs.
However, the generation of simulation data has not entirely eliminated human involvement. The team said that adding new physical tasks still requires people to write prompts and design motion keyframes, while some motions also used human video or motion-capture material.
The research team also noted that the robot currently relies primarily on visual perception and has difficulty directly sensing contact. Performance on tasks involving substantial contact declines when objects leave the camera's field of view or are occluded. Adding tactile perception is one direction for future improvement.
IT Home provides the relevant links below:
Paper: https://arxiv.org/abs/2611.06908
Project homepage: https://apple-aiml-research.github.io/ml-halo/
GitHub repository: https://github.com/apple-aiml-research/ml-halo
Advertising disclosure: External links in this article (including but not limited to hyperlinks, QR codes, and access codes) are provided to convey more information and save time in selecting information. Their contents are for reference only. All IT Home articles containing external links include this disclosure.