Robotics Foundation Models: How RT-2 and OpenVLA Tokenize Action

The Robotic Tokenization Breakthrough: From Words to Motor Actions

For decades, robotics software was fundamentally partitioned into disconnected, brittle sub-systems: - Perception: Object detectors (YOLO, Mask R-CNN) estimating 3D bounding boxes. - Motion Planning: Algorithmic kinematic solvers (RRT*, PRM) computing collision-free paths. - Low-Level Control: PID controllers firing joint actuator voltages.

If a coffee mug was rotated by forty-five degrees, or if an unfamiliar fruit was placed on a counter, the entire stack failed because the perception system had no semantic understanding of human concepts.

In 2023 and 2024, robotics research underwent a historic convergence with foundation models through Vision-Language-Action (VLA) architectures—spearheaded by Google DeepMind's RT-2 (Robotics Transformer 2) (*RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control*) and open-source models like OpenVLA (Stanford/Berkeley) and Octo.

The foundational breakthrough of VLA models was shockingly elegant: treating robotic motor actions as ordinary tokens in a language model's vocabulary.

Camera Image (RGB) + Natural Language Task ("Pick up the ripe banana") │ ▼ ┌────────────────────────────────────────────────────────┐ │ Pre-Trained Vision-Language Backbone (VLM) │ │ - PaLM-E (RT-2) or Llama 2 / Prismatic VLM (OpenVLA) │ │ - Web-scale knowledge of objects, physics, and concepts│ └────────────────────────┬───────────────────────────────┘ │ Autoregressive Decoding ▼ ┌────────────────────────────────────────────────────────┐ │ Discretized Action Tokens (7-DoF End-Effector) │ │ [x, y, z, roll, pitch, yaw, gripper_close] │ │ Example: Token IDs: [20481, 20512, 20490, ..., 20550] │ └────────────────────────┬───────────────────────────────┘ │ Continuous Vector De-quantization ▼ Physical Robot Arm Controller (UR5e, Franka Emika, WidowX)

---

How VLAs Tokenize Continuous 7-DoF Action Spaces

A standard robotic manipulator operates in a continuous physical space characterized by 7 Degrees of Freedom (7-DoF): - 3D Cartesian Position: $(\Delta x, \Delta y, \Delta z)$ - 3D Orientation (Euler Angles): $(\Delta \text{roll}, \Delta \text{pitch}, \Delta \text{yaw})$ - Gripper Actuation: $\text{gripper} \in [0, 1]$ (Open to Closed)

To integrate continuous motor actions into an autoregressive transformer that predicts discrete vocabulary tokens, RT-2 and OpenVLA employ uniform action discretization:

1. Range Clamping: Continuous displacements are normalized and bounded to an operating envelope, typically $[-d_{\text{max}}, +d_{\text{max}}]$. 2. Bin Discretization: The continuous span is partitioned into $B = 256$ discrete uniform bins:

$$\text{Token}(a_i) = \text{round}\left( 255 \cdot \frac{a_i - a_{\text{min}}}{a_{\text{max}} - a_{\text{min}}} \right)$$

3. Vocabulary Extension: These 256 integers are assigned to the least-frequently-used token IDs in the language model's existing tokenizer (or appended as dedicated action tokens $\langle \text{act}_0 \rangle \dots \langle \text{act}_{255} \rangle$).

During inference, predicting a physical robot movement is executed identically to generating the next word in an email: the model emits 7 sequential tokens representing the next motor displacement vector.

---

Emergent Semantic Reasoning and Physical Generalization

Because VLA models build directly upon pre-trained vision-language foundation models (trained on hundreds of billions of web images and text documents), they inherit semantic world knowledge:

• Symbolic Reasoning: If instructed to *"Move the apple to the smallest coin,"* an RT-2 model correctly identifies a dime among nickels and quarters, moves its arm, and places the apple—despite never being explicitly trained on coins during robot teleoperation.
• Multilingual Understanding: Commands can be issued in German, Mandarin, or French; the underlying multilingual representations parse intent and map it to physical joint trajectories.
• Abstract Physics: In response to *"Knock over the object that can spill water,"* the model targets an open plastic cup rather than a solid wooden block.

---

The Open X-Embodiment (OXE) Dataset & OpenVLA

Robotics historically suffered from the "hardware silo" problem: data collected on a Franka Panda robot could not be used to train a Google Everyday Robot.

In late 2023, a consortium of 33 academic laboratories released the Open X-Embodiment (OXE) dataset: - Over 1,000,000 robotic manipulation demonstrations. - Spans 22 distinct robot morphologies, from low-cost WidowX arms to dual-arm bipedal humanoids. - Standardized into unified action formats.

Using OXE, researchers built OpenVLA (7B) (*OpenVLA: An Open-Source Vision-Language-Action Model*, Kim et al.): - An open-weights 7B VLA model based on Llama 2 and DINOv2 / SigLIP visual backbones. - Matches or exceeds proprietary RT-2 performance on standardized tabletop manipulation tasks while remaining freely downloadable on Hugging Face Spaces.

---

Architectural Comparison: Robotics Foundation Models

| Model Architecture | Parameter Scale | Base VLM Backbone | Action Formulation | Latency / Frequency | Open Weights? | | :--- | :---: | :---: | :---: | :---: | :---: | | Google RT-1 | 35M | Custom CNN | Discrete Bins | 3 Hz | Code Only | | Google RT-2 | 55B | PaLM-E / PaLI-X | Discrete Language Tokens | 1 - 3 Hz | Proprietary | | Octo (UC Berkeley)| 93M | Transformer Diffusion | Diffusion Action Chunking | 10 - 20 Hz | Yes (Apache 2.0) | | OpenVLA (Stanford)| 7B | Llama 2 + SigLIP | Discrete Language Tokens | 5 Hz (Quantized) | Yes (Apache 2.0) | | Physical Intelligence $pi_0$| Undisclosed | Custom VLM | Continuous Flow Matching| 50 Hz | Research Preview |

---

The Latency Barrier: Moving Toward High-Frequency Action Chunks

While discrete language tokenization proved that internet pre-training transfers to physical actuation, autoregressive decoding introduces an operational bottleneck: - Predicting 7 individual tokens sequentially through a 7B parameter transformer takes between 200ms and 500ms, capping control frequency at 2 to 5 Hz. - Rapid physical stabilization—such as catching a slipping glass—demands 20Hz to 50Hz low-latency control.

To bridge this gap, next-generation architectures (like Octo and Physical Intelligence $pi_0$) are coupling high-level VLM semantic planners with continuous high-frequency diffusion action heads, cementing the foundation model as the universal brain for modern robotics.