Skip to main content
Research

How language models became robot brains

How RT-2, π0, and OpenVLA connected visual and language pretraining to robot actions, and why their progress still depends on data and evaluation.

By Avala Research

6 min read
0
Share

A robot can recognize an object without knowing how to grasp it. It can describe a task without being able to complete it. Vision-language-action models, or VLAs, connect these problems: they learn to produce robot actions from observations and instructions, drawing on representations learned from images and language.

Three research systems help explain that transition. RT-2 represented actions as tokens. π0 used flow matching to generate continuous actions. OpenVLA made a model and fine-tuning tools available for further experimentation. They offer different answers to the interface between understanding and movement, not proof that one architecture has solved robotics.

For teams building physical systems, the useful question is what that transition changes about the data they record and the experiments they run. Here is our reading of the research, followed by its implications for the learning loop.

From instructions to actions

There is a practical distinction between choosing a task and executing it. “Pick up the cup” leaves many decisions unresolved: which pose to approach from, how far to close the gripper, and what to do if the cup moves. A high-level instruction does not specify all the actions needed to perform it.

A VLA learns a connection between observations, instructions, and actions. That connection still has an interface. The action representation must mean something to the robot controller, and the observations must contain enough information for the policy's task. Calling the model a robot brain is a metaphor, not an account of the whole deployed system.

RT-2 made actions part of the model's output

The RT-2 paper, submitted in July 2023, describes co-fine-tuning vision-language models on robot trajectories and web vision-language tasks. It expresses robot actions as text tokens so that they can be learned alongside language outputs.

The authors report improved generalization to unfamiliar objects and interpretation of instructions absent from the robot training data. Their examples include choosing an object by its relative size or position. These are results within the paper's evaluation, not evidence that the robot can execute any task it can describe.

The important interface change was to learn action outputs within the model. It did not remove the need to define the actions, collect robot trajectories, or execute commands through a robot system.

π0 explored continuous action generation

The π0 paper, first submitted in October 2024, takes a different approach. It builds a flow-matching architecture on a pretrained vision-language model and trains with data from multiple robot platforms, including single-arm robots, dual-arm robots, and mobile manipulators.

The reported evaluations include instruction following, fine-tuning for new skills, and tasks such as laundry folding and table cleaning. The paper treats data, generalization, and robustness as continuing challenges.

The contrast with RT-2 matters more than a nervous-system analogy. Action generation has design choices of its own. A model's representation, its training procedure, and its deployment interface need to be evaluated together. A successful demonstration does not establish reliability across different hardware, environments, or operating conditions.

Open models made experimentation more accessible

OpenVLA, first submitted in June 2024, reports a model trained on diverse real-world robot demonstrations and releases checkpoints, code, and fine-tuning notebooks. Its authors attribute the reported improvements to both data diversity and new model components. The paper also examines efficient adaptation to new settings.

That is evidence for progress in several parts of the system at once. It does not isolate data as the only cause, establish that model design has stopped mattering, or guarantee the same result on a new robot.

Our interpretation is that accessible models make a controlled experiment easier to start. They do not remove the experiment. A team still needs to choose a baseline, define the task, and test whether its changes improve outcomes on held-out conditions.

Record what the action means

For a learning system, video of motion and recorded actions are different inputs. A camera can show a gripper closing without establishing which command was sent, when the controller received it, or what the robot measured afterward.

A useful recording contract distinguishes observations, commanded actions, measured state, and derived estimates. It specifies timebases, units, coordinate frames, missing values, and the control mode. Those details depend on the robot and training method; there is no universal camera count or sample rate that makes a recording suitable for every VLA.

Labels should answer the questions the experiment needs. Task boundaries, outcomes, interventions, and recovery segments can help describe what happened. They do not turn an estimated pose into measured state or make every frame require a manual annotation. See our VLA training-data requirements for a closer look at the handoff.

Preserve meaning across the handoffs

Recording, review, export, and training may use different tools. The engineering question is whether the transitions preserve the information the experiment depends on.

Start with a representative recording. Check its timestamps, action definitions, calibration references, and missing channels. Carry those definitions into the export, then load that export through the intended training path. A file that opens successfully can still contain a time shift or an action convention the consumer interprets incorrectly.

Keep the recording identity, annotation revision, conversion settings, and dataset release connected. That record helps reproduce a training input and investigate a failure. It does not, by itself, identify which training example caused a particular model decision. Causal attribution requires additional analysis.

Close the loop with an evaluation

A deployment can reveal a useful failure case. Whether collecting more examples of that case helps is a hypothesis to test, not an automatic consequence of storing more data.

Define the failure condition, select or collect relevant recordings, and record the changes made to the training set. Compare the candidate policy with a baseline on held-out conditions. Examine regressions as well as gains. Keep model versions and evaluation settings attached to the result so the next experiment starts from an interpretable record.

This is why the VLA story matters to Avala. We build fleet learning infrastructure around the connections between sensor context, review, dataset delivery, and model feedback. The customer owns the model and decides what to train and release. The papers above did not evaluate Avala, and their results are not evidence of our performance.

Our thesis is that a dependable loop makes research easier to turn into repeatable engineering. Better architectures and better data can both contribute. The evaluation determines whether either helped. Explore Avala's robotics workflow or book a working session to discuss the data handoffs in your program.

  • Physical AI
  • Robotics
  • Foundation Models
  • Data Infrastructure
  • VLA