Executive Summary
A data engine that rewrites work footage a person shot on smart glasses or a wrist camera into trajectories 16 kinds of robot can follow is now public. It is Chorus, posted to a research page on October 26 by Action Intelligence (行元智能), a Chinese robot foundation model company. Video that arrives in a different shape from every device is first converged into a single intermediate representation — task semantics and spatial motion — and that representation is then fitted to the body of each robot and handed over as training data.
At the center of the numbers the company puts forward is the conversion rate. One screened hour of human footage converts across 16 robot types at an average rate of 85%, which comes to roughly 13.6 hours of trajectory. A single clip gets used sixteen times, so the processing cost per trajectory hour falls by the same factor. Not one line of the research page, though, shows whether a robot trained on the data this produces actually got better.
Sections 1 through 7 report what the research page and the primary sources set down. Section 8 is this article's reading.
Key Numbers
The numbers Chorus has published fall into three groups: the breadth of the conversion, its cost, and the box still left empty.
Source: the Action Intelligence Chorus research page, published October 26, 2026.
16
Robot embodiments supported
One motion of a human hand is solved separately into the joint angles of 16 robots
85%
Average usable conversion rate
The share that clears joint limits and collisions and remains as usable trajectory
1 → 13.6 hours
What a screened hour becomes
Processing runs about RMB 200 per input hour and about RMB 14.7 per trajectory hour
0
Published learning results
No success rate or generalization figure for a robot trained on this data appears
Six Kinds of Rig, One Shared Representation
Footage of people picking things up, carrying them and wiping surfaces is already piling up across a range of devices. Smart glasses, chest-mounted cameras, wrist cameras, neckbands, head-worn rigs and the UMI-style handheld devices used in labs each record the same motion at a different angle and in a different coordinate frame. What Chorus takes as its input is footage stacked up this unevenly. The capture setups it accepts run from a single monocular camera through stereo, four-camera and six-camera arrangements.
Different devices produce differently shaped video. Chorus first converges all of it into a common intermediate representation, built on two axes: the task semantics of what the person is doing, and the spatial motion of how the hands and objects traveled through space. A robot downstream reads this intermediate form rather than the original footage. Add one more capture device and only the path from that device into the representation has to be built; everything behind it stays as it was.
The research page lays the processing out in six stages: convergence into the common form, semantic annotation, spatial reconstruction, adaptation to a particular robot, synthesis of observation video with that robot in frame, and a final scoring pass over the data.
The company filed this piece as the second chapter of its research archive. Chapter one was Continuo, on robots that stay alongside people, and Chorus takes the seat next to it: what those robots learn from. The company also states that bringing this range of human demonstration sources together in a single framework is, as far as it knows, a first. The phrase on the page is "to the best of our knowledge," and the right way to read it is as the company's claim rather than as a fact verified from outside.
What the Person Did and How the Hands Moved
Showing a robot a video accomplishes nothing by itself. Footage becomes training material only once what the person did, and which coordinates the hand passed through at what speed, are written down as text and numbers. Chorus hands those two jobs to its second and third stages.
2.1Semantic Annotation
The "what" gets attached by a 4-billion-parameter model the company built itself. It sweeps the video at 10 frames per second and recognizes atomic actions frame by frame: reaching, grasping, turning, units that cannot be broken down further. The research page puts the cost of this work at roughly one twentieth of a Gemini 3.8 Flash baseline. Running a small dedicated model instead of calling a general-purpose one decides the unit price of this stage.
The annotations attached then multiply across three layers. Atomic actions, the subtasks they group into, and the whole task the clip carries out each get their own instructions. Instructions at different granularities coming out of one clip bring the instruction volume to roughly three times, by the research page's account.
2.2Spatial Reconstruction
The "how" gets recovered through depth estimation, hand localization and object segmentation. Video shot by a person usually carries no camera calibration parameters and no record of camera pose, and where those values are missing Chorus estimates them back out of the recording. Stretches where the hand moves fast or an object blocks the view and the trajectory breaks are filled in by an optimization that weighs time and geometry together.
This stage makes one choice that matters later. It does not pass on the reconstruction alone; it passes on how uncertain that reconstruction is. When joint angles get computed further down the line, a stretch reconstructed hazily and a stretch reconstructed sharply should not be handled at the same weight.
Moving One Human Arm Onto 16 Robot Arms
A human arm and a robot arm differ in length and in the number of joints. A robot can rarely trace the hand trajectory a person drew exactly as it was drawn. So the fourth stage remakes the same motion to fit the robot's body, the job robotics calls retargeting.
The order runs like this. The robot's base is placed in the scene first. The motion of the human hand is then carried over to the pose of the robot's end effector. That pose goes through inverse kinematics, which finds joint angles that stay inside the joint limits. The angles found are run back through forward kinematics and compared against the original target trajectory, and a confidence score is attached joint by joint. Clutter standing where the robot arm would pass is cleared away by scene editing, while geometry the task needs — the object being manipulated, the surface supporting it — stays in place.
An average of 85% comes through this procedure as usable data. 16 robot types are supported at present, and the research page gives the reason as optimizing the inference engineering. The arithmetic of the conversion is plain. One screened source hour, times 16 embodiments, times 85%, comes to roughly 13.6 hours of trajectory. A scene a person shot once ends up translated into sixteen differently built robot bodies.
For that arithmetic to hold, the 16 trajectories drawn from one human clip have to be learning signals different enough from one another. The research page records the converted volume in hours, and it does not separately measure how much the sixteen overlap.
The fifth stage builds the picture the robot appears in. A robot is projected onto the original footage, the region left bare by removing the human hand is filled by background repair, and the compositing holds the occlusion relationships between the robot and the objects around it. The work makes the observation video used for training show a robot arm instead of a human hand.
Cost is given on two scales. Processing input video runs about RMB 200 per hour, and dividing that by the trajectory volume produced gives about RMB 14.7 per trajectory hour. The second number looks small because of the multiplication by 16 in front of it. Teleoperation, where someone moves a robot directly to gather data, has a different cost structure, one that climbs with the number of robots and the wages of the operators.
Two things are named as what comes next: support for more camera perspectives than now, and handling data from real robot execution.
Four Quality Axes and a Fifth Called Action Surprise
Not every converted trajectory turns out to be worth keeping. At the last stage Chorus attaches scores to the data it has made. Four axes cover quality: whether task semantics are attached completely, whether the 3D reconstruction is accurate and stable, whether the motion holds together physically, and whether the visual quality of the frames is good enough. Effective duration and action counts ride alongside as secondary indicators.
The interesting axis stands apart from those four. The company named it action surprise. A language model is given the context up to that point and asked to predict the next action, and the measurement is how far that prediction diverges from the action actually observed in the video. The larger the divergence, the more likely that stretch holds an unexpected choice, a change of strategy, or a corrective move fixing a mistake. Chorus pushes stretches like these up the review queue.
The four quality axes and this one measure different things. The first four ask whether the data is clean. Action surprise asks how far the data widens a robot's range of behavior. Underneath the axis lies the idea that ten thousand hours of clean footage repeating one motion may be worth less for learning than rougher footage carrying situations that differ from one another.
The Hundred-to-One Gap Between Human and Robot Footage
Language models grew by reading text that had already accumulated on the internet. Robots have no record stacked up the same way. Learning requires records in which observation and action are paired, and records of that kind only come into being once someone actually moves a robot. One operator teleoperating one robot to gather demonstrations remains close to the standard, and the cost of that method rises directly with the number of robots and the hours people put in.
On the other side lies a pile of an entirely different size: the task footage people have shot from a first-person view. As wearable cameras and smart glasses spread, that side outweighs anything a single robot could demonstrate in a lifetime. The distance is written in orders of magnitude. Public teleoperation footage with precise action labels runs to the ten-thousand-hour range, while unlabeled first-person human footage runs to the million-hour range. That gap is roughly a hundredfold, a figure an earlier Pebblous report on robot data interfaces set down by citing a table in the primary source. The trouble is that nothing on the million-hour side carries joint angles, end-effector poses or task labels.
Chorus stands between the two sides. Rather than buying more robots, it proposes to attach coordinates and labels a robot can read to footage that has already been shot. The idea does not belong to Chorus alone.
Five Other Routes From Demonstration to Trajectory
Four columns do the comparing. What goes in, what comes out, how many robot types it spans, and whether the data it makes gets filtered by score. The table below splits five lines of work confirmed from primary sources along those four. Chorus itself is public only through a company research page, with no academic paper and no open repository, so the figures below differ in kind from one another and should be read with that in mind.
| Name | Input | Output | Robots | Quality scoring |
|---|---|---|---|---|
| Chorus | Free-form human demonstration shot on six kinds of device | Per-robot trajectories and synthesized observation video | 16 | Four quality axes plus action surprise |
| UMI (Stanford and others, arXiv:2402.10329) | Demonstrations a person shoots with a camera on a 3D-printed gripper | Relative end-effector trajectories and a policy trained on them | No specific count claimed (compatible where the gripper spec fits) | Only whether kinematic constraints are satisfied |
| EgoMimic (Georgia Tech) | First-person smart-glasses video together with robot demonstrations | Human-robot co-training rather than a converted dataset | 1 (a bimanual robot designed to narrow the kinematic gap) | Not confirmed in the paper |
| Phantom (Stanford, arXiv:2503.00779) | Human demonstration video only, with no robot data | Observation-action pairs with the human arm erased and a robot drawn in | A single setup | Not stated |
| MimicGen · DexMimicGen | A small number of robot teleoperation demonstrations gathered in simulation | Synthetic robot trajectories multiplied by recombination and augmentation | 4 arms (DexMimicGen covers bimanual and humanoid) | A binary filter on task success |
| NVIDIA GR00T synthetic flow | Demonstrations in which a person teleoperates a robot inside simulation | Synthetic trajectories put through domain randomization and photorealistic transfer | Humanoid-centered, with no multi-embodiment figure | Physical plausibility scored on a 5-point scale, with only the top band kept |
▲ Only the range stated in each system's paper or official page is carried here. Items that material does not confirm are marked as such
The columns, read downward, show where Chorus parts company with the rest. Phantom and EgoMimic take free-form human demonstration as input; the MimicGen line multiplies one demonstration across several robots; the GR00T synthetic flow filters generated data by score. No system in the primary sources holds all three inside a single pipeline. That combination is what the company claims to have reached first.
The principle behind the multiplication differs too. MimicGen built more than 50,000 trajectories out of roughly 200 robot demonstrations gathered in simulation, and DexMimicGen made some 21,000 out of 60. The figure NVIDIA has published for the GR00T synthetic flow is 780,000 trajectories. Those numbers, though, came from recombining the same motion inside simulation while object positions and scenes were varied. Chorus, carrying a scene a person shot once into sixteen bodies for a 13.6-fold gain, draws its volume from the number of robot bodies rather than the number of virtual scenes.
Where publication stops is another dividing line. UMI trained a robot-agnostic diffusion policy on top of the trajectories it converted; EgoMimic went as far as aligning human footage and robot demonstrations into one coordinate frame and co-training on both; Phantom put a policy trained on human video alone, with no robot data, onto a real robot zero-shot. All three carry on past conversion and publish the results of learning from that data as well. Chorus has made public only as far as conversion and scoring.
First-person datasets are not comparison targets, but they belong beside this table. Ego4D gathered more than 3,670 hours of video shot by 923 people in nine countries, and Ego-Exo4D gathered 1,286 hours of synchronized first- and third-person video from 740 participants. Both are research assets that collected human first-person footage and annotation at scale, and neither includes a pipeline that turns the footage into robot data. An engine like Chorus can take datasets of this kind as its input.
The Older Research Chorus Is Built On
Most of the parts that make up Chorus came out of older research. The new thing is not the parts but the arrangement that strings them into one pipeline. Below is the research line each stage leans on, traced back to the original papers.
| Stage in Chorus | Research line it leans on |
|---|---|
| Carrying human motion over to a robot | Retargeting in computer graphics — Gleicher, SIGGRAPH 1998 |
| Computing joint angles | Inverse kinematics techniques — Aristidou et al., Computer Graphics Forum 2018 |
| Handling first-person task video | Egocentric vision — EPIC-KITCHENS, ECCV 2018 |
| Recovering hand pose | 3D hand reconstruction — HaMeR, CVPR 2024 |
| Getting depth from a single camera | Monocular depth estimation — Depth Anything, CVPR 2024 |
| Filling in what was erased | Video inpainting — ProPainter, ICCV 2023 |
| Finding the unexpected stretches | Curiosity work that measures novelty by prediction error — Schmidhuber 1991, Pathak et al., ICM, ICML 2017 |
▲ Left, the stages the research page lists. Right, a representative primary source for the technique each stage uses
The last row needs reading a little differently. Treating the size of a prediction error as a signal of novelty started in curiosity research in the 1990s and carried through to the intrinsic curiosity module of 2017. That line spent the signal on exploration, and Chorus spends it on data selection. The research page does not cite the lineage, so the accurate reading is of an idea moving in the same direction and landing on the data side.
Why Pebblous Is Watching This Release
8.1Converted Data and Learnable Data Are Different Things
Chorus has made a finely built conversion pipeline public. Footage becomes coordinates, joint limits are respected, and the output gets scored and filtered. The page describes what each box in the pipeline does in fair detail. One box is empty. How much better a robot trained on the selected data performs than one trained without it never appears. That is the line dividing Chorus from the systems in the section 6 table that published policy-learning results alongside their pipelines.
The blank does not belong to Chorus alone. The physical-plausibility scoring in the GR00T synthetic flow does not disclose what percentage gets filtered out, or how learning outcomes change before and after the filter runs. It explains the generation and scoring procedure and leaves out the number that would settle whether the scoring was right. The shape recurs among those who put scoring inside the pipeline.
Adding one more scoring axis and having that axis actually pick out learnable signal are two different jobs. Whether stretches with high action surprise really help learning can only be known by comparing a model trained with those stretches against one trained without them. The conversion pipeline does not settle the value of data; the score a model trained on it earns does. Until that score exists, 13.6 hours states a volume, not a learning value.
8.2Where Pebblous Works on the Same Problem
The first task Pebblous took up when it was founded in 2021 was precision-targeted synthetic data: the thought that a small amount of data aimed exactly at a hard problem closes the gap that remains. That requires knowing where the hard places are to begin with. DataClinic spreads data out in a high-dimensional space to find its density and its empty spots, and makes that aim possible. The diagnostic method is registered as US patent 12,481,720, and the generation of synthetic data that fills the empty spots as US 11,967,308.
Data Greenhouse strings those tools into a single loop. Diagnose, synthesize, prune, diagnose again. PebbloSim produces robot manipulation data in simulation and feeds it into that loop. It faces the data scoring and selection stage of Chorus across the table, from a different starting point. Chorus finishes converting and then picks by score; Pebblous finds where the gaps are first and then builds aimed at those spots.
Pebblous is not yet at the stage of making data and selling it either. The question this article puts to Chorus comes back to Pebblous unchanged. Whether the empty spot a diagnosis pointed at really moved a model's score can be proven by learning outcomes and nothing else. So there is one sentence Pebblous keeps about data. We do not hunt for data. We cultivate it.
Three pieces are worth reading next. The Data Greenhouse strategy sets the diagnose-and-synthesize loop down as an autonomous data operating system; Data for Physical AI covers where the hunting-versus-cultivating figure came from; the ISO/IEC 5259 roadmap is the work of writing data quality in the language of an international standard.
References
R.1Primary Sources
- 1.Action Intelligence Research. (2026-10-26). "Chorus: A Cross-Embodiment Data Engine for Robot Pretraining." Action Intelligence Research.
- 2.Action Intelligence. "About — Action Intelligence (行元智能)." Action Intelligence.
R.2Comparable Systems
- 3.Chi, Xu, Pan et al. (2024). "Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots." RSS 2024.
- 4.Kareer et al. (2024). "EgoMimic: Scaling Imitation Learning via Egocentric Video." arXiv:2410.24221.
- 5.Lepert, Fang, Bohg. (2025). "Phantom: Training Robots Without Robots Using Human Videos." CoRL 2025.
- 6.NVIDIA / UT Austin. (2023). "MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations." CoRL 2023.
- 7.NVIDIA / UT Austin. (2024). "DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning." ICRA 2025.
- 8.NVIDIA GEAR Team. (2025). "GR00T N1: An Open Foundation Model for Generalist Humanoid Robots." arXiv:2503.14734.
- 9.NVIDIA. "Building a Synthetic Motion Generation Pipeline for Humanoid Robot Learning." NVIDIA Developer Blog.
- 10.NVIDIA. "GR00T Dreams — Post-Training Recipe." NVIDIA Cosmos Cookbook.
- 11.Grauman, K. et al. (2021). "Ego4D: Around the World in 3,000 Hours of Egocentric Video." CVPR 2022.
- 12.Ego-Exo4D Consortium. (2024). "Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives." CVPR 2024.
R.3Technical Lineage
- 13.Gleicher, M. (1998). "Retargetting Motion to New Characters." SIGGRAPH 1998, pp.33–42.
- 14.Aristidou, A., Lasenby, J., Chrysanthou, Y., Shamir, A. (2018). "Inverse Kinematics Techniques in Computer Graphics: A Survey." Computer Graphics Forum 37(6).
- 15.Damen, D. et al. (2018). "Scaling Egocentric Vision: The EPIC-KITCHENS Dataset." ECCV 2018.
- 16.Pavlakos, G. et al. (2023). "Reconstructing Hands in 3D with Transformers." CVPR 2024.
- 17.Yang, L. et al. (2024). "Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data." CVPR 2024.
- 18.Zhou, S., Li, C., Chan, K.C.K., Loy, C.C. (2023). "ProPainter: Improving Propagation and Transformer for Video Inpainting." ICCV 2023.
- 19.Schmidhuber, J. (1991). "A Possibility for Implementing Curiosity and Boredom in Model-Building Neural Controllers." SAB 1991, pp.222–227.
- 20.Pathak, D., Agrawal, P., Efros, A.A., Darrell, T. (2017). "Curiosity-driven Exploration by Self-supervised Prediction." ICML 2017.