Skip to main content

From imitation to imagination

Research

World models connect video prediction to robot action. Their results show why training stages, action signals and evaluation need explicit data contracts.

By Emal Alwis

6 min read
Share

World models ask what may happen next, and robot-learning systems can use those predictions to choose actions. The opportunity is substantial. So is the need to distinguish a convincing video from an executable action, and a promising experiment from a general capability.

The 1XWM report illustrates that distinction. Its video backbone receives 900 hours of egocentric mid-training and 70 hours of robot fine-tuning, after web-scale video pretraining. A separate inverse-dynamics model uses 400 hours of robot data to turn generated frames into actions. The report does not establish that these stage-specific sets are disjoint, so adding them would not establish a unique-hour total.

The authors also report about 11 seconds for video generation, followed by 1 second for action extraction, for a five-second rollout in their setup. These are vendor-reported training and inference details, not Avala measurements or a general recipe for every robot.

The lesson is more useful than a human-to-robot hour ratio: identify what each training stage consumes, what it learns, and how its output reaches the controller.

Prediction and action are different jobs

A vision-language-action model connects observations and instructions to actions. A world model predicts aspects of the environment, sometimes conditioned on candidate actions. A control system can use that prediction in planning, policy training or evaluation. The labels overlap, and different implementations make different choices about representation and feedback.

Neither approach eliminates error accumulation. A policy can visit states poorly represented in training. A learned dynamics model can make inaccurate predictions, and planning against those predictions can amplify the error. The practical question is which failure modes a particular system exposes and how the evaluation detects them.

Video-only data and robot trajectories also provide different signals. Human footage may show an interaction without recording robot commands, measured joint state, force or controller context. A conversion or inverse-dynamics model estimates missing information; it does not turn that estimate into a sensor measurement. Keep the distinction in the schema.

What a controlled comparison can show

The V-JEPA 2 paper studies latent-space prediction and action-conditioned planning. It reports less than 62 hours of robot-video post-training after broad video pretraining. The selected results below average two labs, each with ten trials per task, using Franka arms with RobotiQ grippers and image goals.

Selected manipulation results reported in V-JEPA 2, Table 2:

TaskOctoV-JEPA 2-AC
Grasp a cup15%65%
Pick and place a cup15%80%
Pick and place a box10%65%

Source: V-JEPA 2, Table 2. These are author-reported results under that protocol, not a universal ranking or an Avala benchmark. Both methods run closed loop; the comparison does not isolate feedback versus no feedback. Other tasks in the table are omitted from this selected view.

The paper also reports camera-position sensitivity, reliance on image goals, and error accumulation in longer predicted rollouts. Its planning-time comparison is per action under specified hardware and search settings, not the time to complete an entire task. Those limits matter as much as the successful trials when deciding whether to reproduce the approach.

Keep the data contract attached to the result

A training-hour count is useful only with its role and contents. Before comparing programs, separate these questions:

  • What is observed? Record the actual cameras, timing, calibration and other measured channels. Unknown or missing fields should remain explicit.
  • What is commanded? Identify action representations, units, coordinate frames and controller conventions. A hand-pose estimate is not a recorded robot command.
  • What is derived? Label reconstructed geometry, retargeted motion, generated captions and synthetic views by their provenance and validation method.
  • What happened? Preserve success, failure, intervention and recovery under defined review criteria. Technical validity and behavioral outcome answer different questions.
  • What was evaluated? Link the model version and dataset release to the task, environment, test split, metric and limitations.

The appropriate contract depends on the learning objective. A video-prediction experiment does not automatically require LiDAR, radar, force and every other possible channel. Conversely, calling an artifact “multimodal” does not prove it contains the action and state fields a policy loader needs.

Reconstruction is an estimate, not certainty

A repeatable pipeline can produce the same reconstruction twice and still be wrong. Deterministic processing is not a guarantee of geometric accuracy, complete visibility or correct temporal alignment.

Evaluate reconstruction against the question it is meant to answer. Check calibration, motion, occlusions and the relevant reference measurements. Carry uncertainty and known gaps into downstream review rather than replacing them with visually plausible geometry.

The same principle applies to model-generated labels. Record the proposing model and version, the human review applied and the acceptance criteria. Automation share is an operational measurement for a defined workload; it cannot stand in for task quality or justify a universal percentage promise.

Evaluate the system that will act

Separate visual plausibility, action prediction and physical task performance. A generated sequence may look reasonable while requiring an infeasible action or missing a contact event. A successful action on one embodiment does not establish transfer to another.

Use a representative input slice to test the actual handoff: observations, action/state fields, timestamps, coordinate conventions and episode boundaries should survive the conversion and reach the intended loader. Then evaluate the candidate system on held-out conditions appropriate to the deployment question. Report failures as well as successes, with the denominator and protocol attached.

A model choice can change the data required; better data can change which model works. These examples do not establish that architecture no longer matters or that one model family has won. They make the connection between data and control more concrete.

The infrastructure underneath both

Avala builds Fleet Learning Infrastructure for Physical AI, helping teams connect the recordings, annotations, review and dataset handoff around their learning objective. A program should define supported inputs, required outputs and acceptance checks before making claims about scale or downstream improvement.

Our view is that the useful unit is an inspectable release, not an unqualified hour count. The release should make clear what was captured, what was estimated, what was reviewed and what remains unknown. That argument applies whether the customer uses imitation learning, predictive planning or a combination.

Physical AI can expand what people can do. Turning that ambition into a more capable machine requires evidence about what it has learned and where it still fails. The work is to preserve that evidence through the next round of collection, review and evaluation.

Explore the robotics workflow or book a working session to discuss your data requirements.


Emal Alwis is the founder and CEO of Avala.

  • Physical AI
  • World Models
  • Robotics
  • Data Infrastructure