Executive Summary

Key Numbers

The first three numbers are what the audit gained. The last one records that the gain did not arrive evenly across every stage.

Source: arXiv:2608.07895, Table VII and the LIBERO results

A human teleoperates the robot to produce a demonstration, and a sentence describing the motion gets attached to that episode. Vision-language-action (VLA) policies learn from instructions and trajectories paired this way. The pairing comes apart during bulk collection. Several tasks are recorded back to back and the ordering slips, an annotator picks the wrong entry from a list of similar tasks, or a hand slides after hours of labeling.

The trouble is that the result looks fine. A demonstration where the robot drops an object announces itself the moment you play the video, and pipelines that flag failures automatically already exist. A demonstration with only the instruction wrong executes cleanly from start to finish. It passes the same eyeball quality check a human performs while scrolling through a dataset.

The damage to the policy is a different kind of damage as well. A failed rollout adds bad motion to the mix, but a misattached instruction inverts the correspondence between language and action itself. Show a policy enough cases where the word for picking is paired with the motion of placing, and it loses any basis for telling the two words apart. The harm is largest in settings where the instruction is what distinguishes one task from another.

To reproduce the defect experimentally, the authors injected corruption two ways. One raises the probability of a flipped label for borderline episodes whose nearest neighbor belongs to a different task. The other splits each task's demonstrations into five chunks and moves an entire chunk under a different task label. The corruption rate was set to 30% in both cases. The first mimics an annotator who is genuinely confused; the second mimics one collection session mislabeled as a block. Set next to AgiBotWorld labeling the robot's fumbles, where failed demonstrations were annotated rather than discarded and turned into an asset, this defect sits on the opposite side. Here the demonstration succeeded and the sentence attached to it is wrong.

MMPF (Multimodal Probabilistic Fusion) looks at a single episode through three channels: the head-mounted camera, the wrist camera, and the proprioceptive record of joint positions and actions. Each channel is treated as an independent expert and asked, on its own, which task label this episode should be carrying.

A single expert's judgment blends two signals. The first is local neighborhood agreement. It finds the 30 nearest episodes in embedding space and reads the distribution of their labels, giving closer neighbors a larger vote. The second is global prototype similarity. The mean embedding of every episode carrying a given task label becomes that task's prototype, and the expert measures which prototype the current episode sits closest to. Neighbors alone get swept along by corruption that clusters locally; prototypes alone miss the variation inside a task. The two signals were mixed evenly at λ=0.5.

In the final step the three experts' judgments are multiplied together. How sharply peaked each expert's distribution is decides how much it gets to say, so the confident ones speak louder. When something occludes the wrist camera and it can no longer distinguish anything, the wrist camera's vote shrinks on its own. There was a real stretch of data that needed this. The authors seeded one real-robot dataset with cases where the operator neglected the head-camera view, at a rate of 10%, and in those episodes the head camera's judgment goes blurry while the remaining two experts carry more of the decision.

The pipeline as a whole trains nothing. Visual embeddings come from the public Cosmos-Embed1 model as is, and the proprioceptive side, which has no standard pretrained encoder available, uses a lightweight encoder trained separately with masked reconstruction and inverse dynamics. That encoder never saw a task label. When the thing under audit is the label itself, using a classifier trained on those labels as the auditor means learning the corruption along with everything else, so judging by statistics over representations that never saw a label has a structural advantage on this problem.

Audit performance is counted with two separate metrics. Detection is the job of pointing at which episodes are corrupted; correction is the job of guessing the label the episode should have carried in the first place. Doing the former well without the latter leaves deletion as the only available option. Read the two together and the conclusion changes.

Measurement happened in three places. LIBERO is a simulation benchmark widely used for robot manipulation learning, and its four suites, each with a different character, were combined with the two corruption schemes above to produce eight settings. The other two are collected on real hardware. Table holds 1,260 episodes of four basic actions performed on paper and mugs: fixing the gaze, approaching, picking, and placing. Bottle-mug holds 2,980 episodes of a two-stage manipulation, picking up a bottle or mug and setting it at a target spot. All three record the head camera, the wrist camera, and proprioception together.

Values in %. LIBERO combines 4 suites with 2 corruption schemes for 8 settings. Source: arXiv:2608.07895

The 100% correction accuracy comes from a simulation benchmark, and it is measured only over the true mismatches that were detected. Episodes missed at the detection stage never enter that denominator. It has to be read alongside the detection F1, which swings between 88.9% and 100% depending on the setting.

There were three points of comparison. Confident Learning flags suspects using the predicted probabilities of a classifier trained on the labels. Retrieval matches instruction against video using CLIP-style vision-language similarity. LEMoN reasons from neighbors in the representation space and neighbors in the label embedding space at once. How the three scored depended heavily on the character of the corruption. Confident Learning, the standard tool for label error detection, fell to 64.8% F1 on LIBERO-Spatial, where object placement varies, once it met borderline-case corruption. MMPF scored 99.6% under the same conditions. When labels flip at the boundaries that are easy to confuse, a method that reasons from the predictions of a classifier trained on those labels ends up learning the corruption as if it were normal. For a fair comparison, the authors reimplemented every baseline on the same embeddings.

Once the audit finishes, a decision remains. Pull the suspect demonstrations out of the dataset, or leave them in place and change the label attached to them. The authors trained policies on 30% corrupted real-robot data and compared the two strategies head to head. Real-robot evaluation was counted over 10 rollouts per task, and the policy was trained with the open VLA framework pi0.5. The Table results split across the four actions named earlier: fixing the gaze on the target (Focus), moving toward it (Navigation), grasping it (Pick), and setting it down (Place).

On overall success rate alone, relabeling wins outright. It went from 73.8% to 90.0%, 11.2 points above the 78.8% that filtering produced. On the second dataset, Bottle-mug, relabeling pulled 62.5% up to 77.5%.

Break it down by stage and the character of each strategy shows through. On the early stages of fixing the gaze and approaching, both strategies hit 100%. As long as the instruction is attached correctly, there is nothing to confuse about which task this is. At the Place stage, where the robot sets an object down, filtering sank to 40%. That is below the 50% of doing nothing at all. Deleting the corrupted demonstrations deleted the perfectly good trajectories inside them, and the policy was left short of examples for learning low-level control.

The judgment criterion the authors wrote into the paper lives right at this point. When the dataset is large enough and the mislabeled demonstrations are hard to trust down to the trajectory, filtering is the better move. When the dataset is small enough that losing control trajectories hurts and the motion itself is sound, repairing is better even if correction accuracy is short of perfect. Neither branch carries a numerical threshold.

The question a team working with robot data actually faces is not what the detection rate is. It is what to delete and what to repair, standing in front of the list of demonstrations the audit flagged. The most practical result this paper leaves behind is that the decision moves overall success and per-stage success in opposite directions. Same audit output, same tooling, and which lever you pull is worth 11.2 points.

Making that decision requires knowing a few things about the dataset.

• Is the trajectory of the flagged demonstration sound on its own, or are label and trajectory both suspect
• How many demonstrations of that task remain. Which stage runs short of examples for low-level control once you delete
• Does the instruction do the work of distinguishing tasks, or do the observations distinguish them already

All three questions are answered by the dataset's own records, not by the audit tool. Whether the original instruction survives alongside the corrected one, which session a demonstration came from and in what order, how many demonstrations each task holds. Those have to be attached to the record for the questions to have answers. A related problem came up in We Mixed in Flawless Data, and the Robot Got Worse: how hard it is to verify in closed loop what happens to policy performance when you change mixture ratios and demonstration selection. The final experiment in this paper reads as a case of filling part of that gap with measurement, narrowed to two strategies.

The method has clear limits of its own. Its judgments lean on the quality of the trajectory embedding, and unlike video, robot proprioception still lacks usable pretrained resources. It assumes the list of task labels is fixed in advance, so it does not transfer as is to datasets where instructions are written freely. The authors listed replacing rigid task classes with soft neighborhood relations in instruction embedding space as the next problem to solve.

• 1.Holk, S., Takanami, R., Matsushima, T., Iwasawa, Y., Matsuo, Y., Wu, Y.-H., & Ota, K. (2026). "Auditing Instruction-Trajectory Mismatches in Multimodal Robot Demonstrations." arXiv:2608.07895 (accepted, IEEE RA-L).