How Vision-Language-Action Robot Models Work

1. A vision-language model that answers with movements

Google DeepMind's RT-2 paper of July 2023 named the category. Its authors took vision-language models (VLMs) already trained on internet images and text, PaLI-X at 5B and 55B parameters and PaLM-E at 12B, and fine-tuned them so that, asked "what action should the robot take to [task]?" about a camera image, they reply with a robot action instead of a sentence. They called the category vision-language-action models.¹ The bet is that knowledge learned from the web carries over to control: RT-2's evaluation (about 6,000 trials) reports better handling of unseen objects and commands that never appear in its robot data, such as placing an object on a particular number or icon.¹

Every model in the table keeps that shape: camera images, a language instruction and usually the robot's own joint state go in; a vision-language backbone digests them; something at the output end produces motor commands. What differs is that last stage, how fast each part runs, and what the model was trained on.

2. Actions as tokens

The simplest way to make a language model emit actions is to make actions look like words. RT-2 describes each step as the end-effector's change in position and rotation (6 degrees of freedom), how far the gripper is open, and a "terminate" flag; each continuous dimension is cut into 256 equal bins, so an action becomes eight integers, written as a string such as "1 128 91 241 5 101 127". For PaLI-X, which already has a token per integer up to 1,000, the bins reuse those tokens; for PaLM-E, the 256 least-used tokens in the vocabulary are overwritten. At run time the output is restricted to valid action tokens.¹

OpenVLA (June 2024, 7B parameters, built on Llama 2 with DINOv2 and SigLIP image encoders) uses the same 256 bins, but sets them between the 1st and 99th percentile of the training actions so outliers don't stretch the scale, and overwrites the last 256 tokens of the Llama tokenizer.² Ai2's MolmoAct also uses 256 percentile bins, but first has the model write depth "perception tokens" and a 2D trajectory trace of where the end-effector should go, so the plan can be read and edited before the action tokens follow.¹⁰

Per-dimension binning has a known weakness. The FAST paper (January 2025, from Physical Intelligence, UC Berkeley and Stanford) reports that it "typically perform[s] poorly" on dexterous, high-frequency robot data, and proposes compressing a whole sequence of actions with the discrete cosine transform before tokenizing; its FAST+ tokenizer was trained on 1M real robot trajectories, and paired with π0 it matched diffusion-style VLAs while cutting training time by up to 5x, per the authors.⁸

3. Continuous actions: flow matching and diffusion heads

The other route skips tokens for the output and generates continuous numbers with a small generative network bolted onto the backbone. Physical Intelligence's π0 (October 2024) starts from the 3B-parameter PaliGemma VLM and adds a 300M-parameter "action expert", 3.3B parameters in total. The expert is trained with flow matching, which its paper calls "a variant of diffusion": it learns to turn random noise into a chunk of 50 future actions, here in 10 integration steps, while reusing the cached backbone output at each step.³

NVIDIA's GR00T N1 (March 2025) uses a diffusion transformer (DiT) trained with action flow matching that cross-attends to the VLM's output; its 2.2B-parameter checkpoint (1.34B in the VLM) samples a chunk of 16 actions in 63.9 ms on an L40 GPU.⁵ The GR00T N1.7 card keeps the flow-matching DiT and swaps the backbone for Cosmos-Reason2-2B, for 3B parameters in all.²¹ Hugging Face's SmolVLA (June 2025) applies the same recipe at 450M parameters.⁹ π0.5 (April 2025) uses both: it is pre-trained on FAST action tokens for 280k steps, then a randomly initialized flow-matching action expert is added and trained for 80k more.⁴

4. A slow thinker and a fast controller

A billion-parameter backbone is too slow to drive motors directly, so several designs split the work, borrowing Kahneman's "System 2" (deliberate) and "System 1" (fast) labels. In GR00T N1 the VLM (System 2) runs at 10 Hz on an L40 GPU and the action module (System 1) produces motor actions at 120 Hz; the two are trained together end to end.⁵ Figure's Helix (February 2025) pairs a 7B-parameter open-weight VLM running at 7–9 Hz (S2) with an 80M-parameter transformer that outputs upper-body control at 200 Hz (S1); S2 passes S1 a single latent vector, and the two run on separate embedded GPUs on the robot.¹¹ Helix 02 (January 2026) adds a third layer, S0: a 10M-parameter network trained in simulation on over 1,000 hours of human motion data that turns S1's full-body joint targets into actuator commands at 1 kHz.¹²

Google DeepMind's first Gemini Robotics model split along the network: a backbone distilled from Gemini Robotics-ER runs in the cloud with query-to-response latency under 160 ms, and a local action decoder on the robot compensates, for about 250 ms from observation to action chunk and an effective 50 Hz control rate.⁶ From Gemini Robotics 1.5, the split is between models: an embodied-reasoning model (ER 1.5, then ER 2) plans the task and hands the VLA natural-language instructions one step at a time.¹⁵¹⁶

5. Where the training data comes from

Robot demonstrations are scarce next to web text, so every VLA mixes sources. The shared starting point is Open X-Embodiment, pooled from 22 robots by 21 institutions and covering 527 skills (160,266 tasks).⁷ OpenVLA trained on 970k of its demonstrations, using 64 A100 GPUs for 14 days (21,500 A100-hours).² π0 added more than 10,000 hours of its own data from 7 robot configurations and 68 tasks.³ π0.5 had about 400 hours of mobile robots in real homes, but 97.6% of its first-phase training examples came from elsewhere: other robots, web data such as captioning and question answering, and high-level task labels.⁴ Co-training with web data goes back to RT-2, whose authors found it generalized better than fine-tuning on robot data alone.¹

Others lean on synthetic data. NVIDIA describes a "data pyramid" of web data and human video at the base, simulation and video-model output in the middle and real robot data at the top; it generated 780,000 simulated trajectories, equal to 6,500 hours of demonstrations, in 11 hours, and labeled action-less human video with learned "latent actions".⁵ SmolVLA pre-trained on 481 community datasets (22.9K episodes, 10.6M frames), Helix on about 500 hours of teleoperation, and Gemini Robotics on thousands of hours collected on a fleet of ALOHA 2 robots over 12 months.⁹¹¹⁶

6. Control rates, chunks and where the model runs

Speed decides what a VLA can do. RT-2-PaLI-X-55B ran at 1–3 Hz from a multi-TPU cloud service; the 5B version at about 5 Hz.¹ OpenVLA needs 15 GB of GPU memory in bfloat16 and runs at about 6 Hz on one RTX 4090; its authors report that 4-bit quantization kept success rates similar while using under half the memory.²

Action chunking is how slower models drive faster robots. π0 takes 73 ms to produce a chunk on an RTX 4090 (86 ms when run off-board over Wi-Fi), then plays it open-loop: on its 50 Hz robots it re-plans every 0.5 seconds, after 25 actions.³ π0.5 sends arm, gripper and lift targets and base velocities at 50 Hz to simple PD controllers, with no separate trajectory planner.⁴ On-robot versions avoid network latency: Google DeepMind says Gemini Robotics On-Device runs locally and adapts to new tasks with 50 to 100 demonstrations, and that On-Device 2 adapts to a new bi-arm robot in a few hours, typically with fewer than 200 examples.¹⁴¹⁶

7. Open and closed weights

The field is split. OpenVLA's checkpoint is MIT-licensed;¹⁸ π0 and π0.5 base checkpoints are on Hugging Face under the Gemma license inherited from PaliGemma, though the LeRobot π0.5 card says only the flow-matching head is included, not the subtask prediction or action tokenization from the paper;¹⁹ NVIDIA publishes GR00T under its Open Model License;²¹ SmolVLA and MolmoAct use Apache-2.0, and Ai2 says it releases MolmoAct's training code and dataset as well.²³²⁴ Google DeepMind's VLAs reach outsiders through programs: over 100 trusted testers, per its model page, with the VLA and On-Device 2 open to early-access partners and only the reasoning model, ER 2, available in Google AI Studio.¹⁷¹⁶ Figure's Helix pages describe the system but offer no weights, although S2 itself is built on an open-weight VLM.¹¹

8. How VLAs are evaluated

There is no shared leaderboard. Each paper picks its own tasks and robots: RT-2 ran about 6,000 trials; OpenVLA tested 29 tasks on WidowX and Google robots and fine-tuned on the LIBERO simulation suites; π0 scored tasks such as laundry folding with per-task rubrics.¹²³ GR00T N1 used two open simulation benchmarks plus its own tabletop suite and a Fourier GR-1 humanoid; SmolVLA used LIBERO and Meta-World; MolmoAct reports results on SimplerEnv and LIBERO; NVIDIA's N1.7 card lists checkpoints post-trained for SimplerEnv, DROID and LIBERO.⁵⁹¹⁰²¹ For the reasoning side, Google DeepMind introduced ERQA, 400 multiple-choice questions on embodied reasoning.⁶ All of these numbers are run by the labs themselves, with different robots, harnesses and fine-tuning budgets, so MetaPrompt doesn't rank VLAs against each other on them.