Robot data formats compared: LeRobot, RLDS, HDF5 and MCAP

Contents

1. What every format has to hold
2. LeRobotDataset
3. RLDS
4. HDF5
5. MCAP
6. Side by side
7. Converting
8. Choosing
9. Questions buyers ask
10. Sources

The file format is the last decision in a collection and the first thing the training engineer meets. A dataset in the wrong layout costs days of conversion, and a conversion can quietly drop a stream or shift a timestamp. Four formats cover most robot-learning work. This guide describes each from its own documentation.

What every format has to hold

Whatever the container, a teleoperated episode is the same set of things: camera frames, the robot's joint and gripper states, the actions commanded, timestamps on one clock, and metadata such as the task, the operator and the outcome⁠1How to collect teleoperation data for robot learning (LeRobot, RLDS, HDF5)RoboPilots · 9 Oct 2026. Formats differ in how they pack these and in which tools read them.

LeRobotDataset

Hugging Face describes LeRobotDataset v3.0 as a standardised format for robot learning data. States, actions and timestamps are stored in Apache Parquet. Camera frames are concatenated and encoded as MP4. Metadata in JSON and Parquet records the schema, frame rates, normalisation statistics and episode segmentation⁠2LeRobotDataset v3.0Hugging Face LeRobot documentation.

Version 3.0 packs many episodes into each Parquet and MP4 file, where version 2 used one file per episode. It ships with lerobot 0.4.0 and later, with a converter from version 2.1⁠2LeRobotDataset v3.0Hugging Face LeRobot documentation. Datasets can be opened in a browser visualiser that shows the camera recordings beside state and action charts⁠2LeRobotDataset v3.0Hugging Face LeRobot documentation.

It suits teams that train with the LeRobot code or share data on the Hugging Face Hub.

RLDS

In RLDS a dataset is a collection of episodes, and each episode is a sequence of steps. A step carries flags that mark the first and last step and, optionally, an observation, an action, a reward and a discount. All steps in a dataset must have the same fields⁠3RLDS: reinforcement learning datasets (README)Google Research (GitHub).

RLDS matters for robot learning chiefly because the Open X-Embodiment collection uses it. Each of its datasets is a sequence of episodes in the RLDS episode format, loaded through TensorFlow Datasets⁠4Open X-Embodiment repositoryGoogle DeepMind (GitHub). The collection pools more than one million trajectories from 22 robot types⁠5Open X-Embodiment: robotic learning datasets and RT-X modelsarXiv (Open X-Embodiment Collaboration) · 13 Oct 2023.

It suits TensorFlow pipelines and teams that co-train with cross-embodiment data.

HDF5

HDF5 is a general container for arrays, and robot learning uses it by convention. In robomimic's layout a dataset is a single HDF5 file with a group per trajectory, and each trajectory holds arrays of states, actions, rewards and observations. Images are stored as 8-bit arrays in height, width, channel order⁠6robomimic: dataset overviewrobomimic documentation. ALOHA's recorder writes an HDF5 file per episode and ships scripts to visualise and replay it⁠7ALOHA repository (README)Zhao et al. (GitHub).

It suits teams that want a simple layout they can inspect with standard tools. Since the layout is a convention, two HDF5 datasets are not necessarily compatible.

MCAP

MCAP is an open-source container for multimodal log data. It holds multiple channels of timestamped messages in any serialisation, including Protobuf, ROS and JSON. Its maintainers state that it is the default log format in ROS 2⁠8MCAP: a container file format for multimodal log dataFoxglove.

MCAP is a logging format. It records everything a robot published, in time order, and leaves episode structure to a later step. It suits vehicles and ROS-based robots where the raw log is the record of truth.

Side by side

Converting

Conversion is routine but not free. Hugging Face's guide to porting large datasets uses DROID as its example. The dataset is 1.7 TB in RLDS form and 8.7 TB as raw data, and it has to be converted from TensorFlow records before LeRobot can read it⁠9Porting large datasets to LeRobot Dataset v3.0Hugging Face LeRobot documentation.

Vendors that deliver data name their formats in similar terms. Robotics Center lists HDF5 for ACT, ALOHA and robomimic, RLDS for Open X-Embodiment and Octo, and LeRobot with Parquet⁠10Custom robot data collection servicesRobotics Center. RoboPilots delivers LeRobot, RLDS or HDF5, and MCAP or ROS 2 bags for vehicles⁠1How to collect teleoperation data for robot learning (LeRobot, RLDS, HDF5)RoboPilots · 9 Oct 2026.

Choosing

1. Ask what the training code loads today. That is the delivery format.
2. Record raw, deliver converted. Keep the original recording so that a conversion bug can be fixed without collecting again.
3. Write the schema down. Camera names, resolutions, frame rate, action space and units. A policy trained on a dataset expects the same camera names and resolution at deployment⁠11Imitation learning on real-world robotsHugging Face LeRobot documentation.
4. Load a sample before collecting at scale. Ten episodes through the real training loader will expose most format problems.
5. Store the protocol version and outcome with every episode. Filtering later depends on it.

The end state is a dataset that loads into the training job with no manual steps, in a layout documented well enough that a new engineer can read it on their first day.

Questions buyers ask

Sources

1. How to collect teleoperation data for robot learning (LeRobot, RLDS, HDF5). RoboPilots, 9 Oct 2026. Accessed 11 Nov 2026.
2. LeRobotDataset v3.0. Hugging Face LeRobot documentation. Accessed 11 Nov 2026.
3. RLDS: reinforcement learning datasets (README). Google Research (GitHub). Accessed 11 Nov 2026.
4. Open X-Embodiment repository. Google DeepMind (GitHub). Accessed 11 Nov 2026.
5. Open X-Embodiment: robotic learning datasets and RT-X models. arXiv (Open X-Embodiment Collaboration), 13 Oct 2023. Accessed 11 Nov 2026.
6. robomimic: dataset overview. robomimic documentation. Accessed 11 Nov 2026.
7. ALOHA repository (README). Zhao et al. (GitHub). Accessed 11 Nov 2026.
8. MCAP: a container file format for multimodal log data. Foxglove. Accessed 11 Nov 2026.
9. Porting large datasets to LeRobot Dataset v3.0. Hugging Face LeRobot documentation. Accessed 11 Nov 2026.
10. Custom robot data collection services. Robotics Center. Accessed 11 Nov 2026.
11. Imitation learning on real-world robots. Hugging Face LeRobot documentation. Accessed 11 Nov 2026.

Updated 11 Nov 2026: First published.

Further reading

• Field guide · Updated quarterlyTeleoperation data collection: a working guide for 2027What a robotics team needs to decide before it records its first demonstration, with the published numbers on rigs, throughput, quality checks and formats, and a worked cost per clean episode.
• Guide · Throughput and qualityEpisodes per hour and quality checks in teleoperation data collectionTwo numbers decide what a dataset costs: how many episodes an operator records in an hour and how many of them pass. What published collections report on both.
• Guide · The rigTeleoperation rigs for data collection: leader arms, VR and handheld toolsThe device in the operator’s hands decides what the dataset contains. The published evidence on four families of rig, with what each costs and records.