From Seeing to Acting: How Vision-Language-Action Models Control Robots

Vision-language models (VLMs) have demonstrated impressive proficiency at visual question-answering tasks. Given an image of a table with a red cup, for example, a VLM could reason that the instruction “put the cup in the bowl” would result in a new arrangement of objects on the table.

However, that is not the same thing as controlling a robot to perform this same task.

A robot must decide where to move, how far to move, how to orient the gripper, when to open or close it, and what to do if the attempt fails.

It must act on the world, not merely reason about it.

The question, then, is how do we transition from vision-language reasoning to action?

What Is a Vision-Language-Action Model?

A vision-language-action model, or VLA, is similar to a VLM but extends its functionality to produce actions:

vision + language → action

Given a camera viewpoint and a language instruction, a VLA must determine what robotic action to perform.

The general pipeline for VLAs is:

camera observation → visual features

language instruction → text representation

visual features + text representation → robot action

The idea is that a robot requires both knowledge of the world—what objects are where—and an ability to perform actions, such as moving the gripper to pick up an object. Vision-language pretraining provides some understanding of the world and how to reason about it, while robot data provides an opportunity to learn how to act.

VLAs typically operate in a closed loop:

observe → predict → act → observe

Rather than inputting a detailed instruction or plan for what the robot should do, these models continually observe the world, predict an action, perform it, and observe the results.

Why Vision-Language-Action Models Matter

Robotic data is challenging to obtain.

Language models can be trained on large text corpora, vision models on large image datasets, and vision-language models on web-scale image-text data.

However, robotic learning requires physical hardware, which also requires operators, maintenance, and time.

Collecting examples that cover every possible object, environment, language instruction, and failure a robot might encounter is difficult.

Yet, these are precisely the sorts of situations a robotic policy should be able to handle gracefully.

Can Robot Actions Be Treated Like Language?

A vision-language model can reason about an image and an instruction, but how can those representations be converted into physical actions?

RT-2 explores this idea by treating robotic actions as tokens in a discrete action space.

Rather than predicting a natural-language sentence, the model predicts a sequence of tokens representing a low-level robotic action. Think of it as an LLM-inspired, autoregressive approach to robotics.

However, the challenge with such an approach is evident: actions are typically continuous, while language models operate on discrete tokens.

RT-2 addresses this problem by discretizing the action space it learns, with each dimension divided into 256 bins.

For example, an action might be represented as:

translation → bin 129 rotation → bin 113 gripper → bin 255

The same approach is used for translation, rotation, and gripper control, and the resulting bins are concatenated into a sequence of action tokens:

image + instruction → action tokens → continuous action values

When the model is deployed, these tokens are then converted back to continuous values.

RT-2 also uses co-fine-tuning, in which the model is trained on robotic tasks as well as web-scale vision-language tasks.

The result is a policy that retains much of the semantic reasoning power of its pretrained components, while also learning unique physical skills.

What Would Make VLAs Practical?

If robotic data is so difficult and expensive to collect, what would it take to evaluate, adapt, or build on a VLA?

Earlier models were largely closed, with only limited analysis possible because of the lack of details about weights, training procedures, data distributions, and available adaptation techniques.

OpenVLA, a 7-billion-parameter open-source foundation model trained on approximately 970,000 real-world robot demonstrations, is much more accessible and easier to adapt.

OpenVLA has three main components.

The first is a fused vision encoder built from DINOv2 and SigLIP. DINOv2 is a self-supervised vision model, while SigLIP is trained through vision-language alignment. OpenVLA combines their complementary patch-level features.

The second component is an MLP projector, which projects the concatenated visual features into the language model’s embedding space.

The third component is a Llama 2 7B backbone, which takes in the projected visual tokens and a language instruction and outputs robot action tokens.

OpenVLA is a discrete action-token policy similar to RT-2. Each action dimension is discretized into 256 bins, just as in RT-2.

The difference is in how OpenVLA selects the bin edges.

Rather than using the full action range, OpenVLA uses the 1st and 99th percentiles from the robot training data for each dimension, thereby reducing the influence of outliers.

π₀: Do Robot Actions Need to Be Tokens at All?

RT-2 and OpenVLA both attempt to control a robot by predicting a sequence of discrete actions.

However, as we have seen, robot actions tend to be continuous.

π₀ asks a different question: do robot actions need to be tokens at all?

The answer, it argues, is no—actions can be conditioned on language using flow matching.

At a high level, π₀ takes a pretrained PaliGemma vision-language model and adds a set of weights specific to robotics, called an action expert.

Image and language inputs are processed through the vision-language model weights, while robot-state and action inputs use the action-expert weights. Their representations interact through the transformer’s attention layers to generate continuous actions.

π₀ receives multiple camera images, a language instruction, and the robot’s proprioceptive state, such as its joint angles, and it must determine a robot action.

However, π₀ outputs not only a single action but multiple actions:

Aₜ = [aₜ, aₜ₊₁, ..., aₜ₊ₕ₋₁]

The model can execute these actions in rapid succession, which is advantageous for many different robotic tasks involving quick movements.

The key architectural difference is its use of flow matching rather than autoregressive action-token prediction.

At a high level, the process begins with random noise. During training, the model learns how to gradually transform that noise into realistic robot actions based on the images, language instruction, and robot state.

At inference, it uses this learned process to generate a continuous chunk of actions for the robot to execute.

Practical Implications

The inference numbers make the architectural difference concrete. The largest RT-2 model operated at roughly 1–3 Hz, while OpenVLA can reach around 6 Hz on an RTX 4090. π₀ instead predicts a chunk of continuous actions at once, allowing the robot to execute controls at up to 50 Hz.

This does not mean the full model runs 50 times per second. For its 50 Hz robots, π₀ predicts a 50-action chunk but reruns inference every 0.5 seconds, after executing 25 actions from the current chunk.

On an RTX 4090, generating a chunk takes about 73 milliseconds. By combining continuous outputs with action chunking, π₀ supports smoother, more dexterous control than earlier autoregressive VLAs.

What’s Next?

Each model raises intriguing questions about the future of vision-language-action models.

The first issue is data.

Robotic demonstrations can be expensive and time-consuming to obtain, and even the largest robot datasets are comparatively small.

The second issue is reliability and robustness.

A language model may generate incorrect text, but a robot policy that has learned the wrong behavior may do far worse, potentially damaging objects, the surrounding environment, or itself and requiring human intervention.

The third issue is embodiment.

Robots come in a variety of forms with different camera arrays, joints, degrees of freedom, actuation frequencies, and control loops. A generalist policy must be able to adapt across these differences.

While there is clearly more work to be done, researchers have already begun thinking about these challenges.

A vision-language model understands what an instruction says and what the world looks like, but it still has to determine how to act within that world.