Open X-Embodiment: Cross-Robot Pretraining Dataset

A practitioner's guide to Open X-Embodiment (OXE) — the 1M-trajectory, 22-embodiment, 60-dataset corpus that made modern robot foundation models possible.

TL;DR

Metric	Value
Task count	160,000 tasks spanning 527 distinct skills
Robots	22 embodiments (Franka, UR5, WidowX, xArm, Google Robot, Sawyer, Kuka iiwa, and 15 more)
Modalities	RGB (1-4 cameras), depth (subset), proprioception, gripper state, natural language
License	Apache 2.0 for most contributions (per-subset licenses vary)
Size	1,000,000+ real-robot episodes, ~4 TB RLDS
Hosting	Google Cloud Storage (gs://gresearch/robotics) and HuggingFace

What is Open X-Embodiment?

Open X-Embodiment (OXE) is the first serious attempt to unify the fragmented world of real-robot demonstration data into a single pretraining corpus. Released in late 2023 by Google DeepMind and a coalition of 33 academic and industry labs, the dataset aggregates 60 pre-existing robot learning datasets, re-encodes every trajectory into the standardized RLDS (Reinforcement Learning Datasets) format, and ships them behind one download endpoint. The headline numbers are 1,000,000+ robot trajectories, 22 robot embodiments, 527 skills, and 160,000 distinct tasks.

The intellectual argument behind OXE is that robotics has been stuck at the "per-lab, per-robot" scale of training data while vision and language models scaled to the entire internet. By normalizing action and observation spaces across embodiments, the collaboration showed that a single transformer — RT-1-X and later RT-2-X — could be pretrained on the full corpus and then zero-shot-transferred to new robots with positive transfer. This is the result that eventually produced OpenVLA, Octo, and the current generation of VLA foundation models, all of which use OXE (or a curated subset called the OpenX mix) as their pretraining base.

OXE is not a single clean dataset — it is a union of heterogeneous sources, from large-scale teleoperation corpora like BridgeData V2 and RT-1 to laboratory-scale datasets like Jaco Play and NYU Franka Play. Each contribution retains its original license (most are Apache 2.0 or CC-BY, a few are research-only), and researchers typically filter the mix per-experiment rather than training on the raw union.

How to download & load

OXE is distributed as RLDS shards on Google Cloud Storage. You can stream individual sub-datasets without downloading the full 4 TB:

# Install the RLDS/TFDS tooling
pip install tensorflow_datasets rlds

# List the available sub-datasets
python -c "from tensorflow_datasets.robotics import rtx; print(rtx.RTX_BUILDERS)"

# Stream the BridgeData V2 subset directly from GCS
import tensorflow_datasets as tfds
ds = tfds.load(
    "bridge",
    data_dir="gs://gresearch/robotics",
    split="train[:1000]",
)
for ep in ds.take(1):
    for step in ep["steps"]:
        print(step["observation"]["image"].shape, step["action"].shape)

# Or use the HuggingFace Parquet mirror
pip install datasets
python -c "from datasets import load_dataset; ds = load_dataset('jxu124/OpenX-Embodiment', 'fractal20220817_data', split='train', streaming=True)"

For training a VLA, most teams reuse the OpenX mix filter list from the OpenVLA repo, which drops the lowest-quality contributions and rebalances the distribution. Plug the filtered mix into the OpenVLA or Octo training scripts and you can reproduce the published checkpoints on 8xH100 in under a week.

Common use cases & model pairings

• VLA pretraining. OpenVLA, RT-2-X, Octo, and Pi-0 all pretrain on the OpenX mix before fine-tuning on a target dataset like LIBERO or a custom collection.
• Cross-embodiment transfer studies. OXE is the only dataset where you can train on 21 robots and evaluate zero-shot on the 22nd with statistical power.
• Action-space normalization research. The heterogeneity of end-effector control modes (Cartesian delta, joint velocity, gripper binary) makes OXE the default testbed for unified action tokenizers.
• Data-scaling law experiments. Because sub-datasets span four orders of magnitude in size, OXE lets you fit clean scaling curves without collecting new data.

Benchmarks & leaderboards

OXE itself is a training corpus, not a closed benchmark, so evaluations happen on downstream suites. The canonical numbers to beat are the RT-X paper's in-distribution and cross-embodiment success rates on the Google Robot and WidowX evaluation splits, and the OpenVLA LIBERO fine-tune numbers. See the Papers with Code Open X-Embodiment entry, the official RT-X project page, and the RT-X paper (arXiv:2310.08864) for the reference metrics.

Technical deep dive: the RLDS action space

One of the trickiest parts of working with Open X-Embodiment is that every contributing lab had its own convention for expressing a robot action. Some datasets record absolute joint targets, others record Cartesian end-effector deltas, some record velocity commands, and a few record torques. The OXE team solved this by standardizing on an RLDS schema in which every episode is a sequence of (observation, action, reward, discount, is_first, is_last, is_terminal) tuples, and each sub-dataset documents its own action field semantics — typically 7 dimensions for a 6-DoF arm plus binary gripper, or 14 dimensions for a bimanual rig.

Most modern VLA training recipes (OpenVLA, Octo, RT-2-X) handle this heterogeneity by learning a per-dataset normalizer and a unified 7-dimensional Cartesian delta action output, then projecting back to the native space at evaluation time. The normalization statistics ship alongside the OpenX mix, which means you do not have to recompute them yourself unless you introduce a new sub-dataset.

Observations are similarly non-uniform. Some datasets have four cameras, some have one, and image resolutions range from 128x128 to 640x480. The standard preprocessing trick is to resize to 224x224 and randomly select one camera per step during training — a choice that sacrifices some information but drastically simplifies the training pipeline and allows a single transformer to ingest any embodiment.

Known limitations

• Quality variance. Not every sub-dataset is expert teleoperation. Some contributions are autonomously-generated scripted data with mixed success. The OpenVLA filter list is the community's best attempt at a quality floor.
• Language annotation gaps. Roughly 30% of episodes lack natural language instructions; VLA pretraining typically back-fills with a placeholder token.
• Depth coverage is thin. Only a handful of sub-datasets include depth, which limits OXE's usefulness for full 3D policies.
• License fragmentation. Most contributions are Apache 2.0, but a handful are research-only. If you plan to fine-tune a commercial product on OXE, audit the per-dataset license table.

FAQ

Should I train on all of OXE or just the OpenX mix? Start with the OpenX mix — it drops the noisiest contributions and gives you a strong baseline in a fraction of the compute. Add the full corpus only if you are chasing the last 1-2 points of success rate.

How long does pretraining take? Roughly 4-7 days on 8xH100 for an OpenVLA-7B scale model on the OpenX mix, or 1-2 days for a smaller Octo-scale model.

Can I fine-tune OpenVLA on my own data without OXE at all? Yes — OpenVLA ships a pretrained checkpoint, so you only need OXE if you want to re-run pretraining from scratch.