Two New Datasets Want to Make Humanoid Robots Safer and More Capable. Here's What They Actually Do.

They've also built a cloud-based evaluation platform so researchers can benchmark their policies in a standardized way. Honestly, that part might be just as valuable as the dataset itself. One of the persistent frustrations in this field is that everyone tests on slightly different setups, which makes it genuinely hard to compare results across papers.

I initially thought this was just another "we collected a lot of data" paper, but after reading through it more carefully, the human-humanoid interaction component stood out to me. Most datasets treat the robot as operating in isolation. The fact that this one explicitly includes interaction with people feels like a meaningful step toward robots that can actually function alongside us.

Okay, but what's ROBOSHACKLES?

This is the more unsettling one, tbh.

RoboShackles, from a separate research group whose work is also on arXiv, starts from a genuinely difficult problem: you can't safely or ethically collect real-world data of robots injuring people or creating dangerous situations in homes. But if you want to train robots not to do those things, you need examples of exactly that.

So the team built a synthetic data pipeline. They started from real observations in the DROID dataset, then used scene understanding, hazard-aware image editing, and a video synthesis model called Wan2.7 to generate realistic-looking robotic rollouts from edited hazardous states. The result is ROBOSHACKLES, a 10,000-clip video dataset spanning six categories of harm: two direct (the robot physically injures someone) and four indirect (the robot creates a hazardous situation that could lead to injury).

Then they tested six representative Embodied Foundation Models against it.

All six produced unsafe actions in the tested scenarios. Every single one. 100% unsafe action generation rate.

You might be wondering whether that number is as alarming as it sounds. I think the honest answer is: sort of. These models weren't specifically trained to refuse dangerous actions, so it's not shocking that they fail at refusal. What the dataset is designed to do is give researchers a benchmark and training resource for building that refusal capability in. Think of it less as a report card and more as a starting line.

Still. A 100% failure rate across six different models is a number worth sitting with for a moment.

Why does the synthetic data approach matter?

This is actually the part I find most interesting about ROBOSHACKLES, even though it's easy to gloss over.

The pipeline the researchers built, starting from real observations, editing them to introduce hazards, generating temporal prompts that describe how the scene should evolve, then synthesizing video, is a scalable approach to a problem that would otherwise be basically unsolvable. You can't run a robot into a person to collect training data. You can't stage a kitchen fire. But you can construct synthetic versions of those scenarios that are realistic enough to be useful.

Whether the synthetic data is realistic enough is, honestly, an open question. The team used automatic metrics to assess task completion and visual quality, but it remains unclear how well performance on synthetic hazard scenarios translates to real-world safety. That's not a criticism of the research so much as an inherent limitation of the approach. This is based on what the paper reports, and independent validation of the dataset's real-world utility hasn't happened yet, at least not that I've found.

So what do these two datasets mean together?

Taken separately, each dataset fills a gap. Humanoid Everyday gives researchers a richer, more realistic training resource for general-purpose humanoid manipulation. ROBOSHACKLES gives them a way to start building and benchmarking safety refusal into Embodied Foundation Models.

Taken together, they point at something bigger. The field is starting to take seriously the idea that humanoid robots need to be trained not just on what to do, but on what to refuse. That's a conceptual shift that I think matters a lot, even if the specific tools here are early-stage.

The Humanoid Everyday dataset is publicly available through the project website. ROBOSHACKLES is available on Hugging Face. Both are open research resources, which is the right call.

What we don't know yet is how much these datasets will actually move the needle on real-world robot safety and capability. Dataset papers are a bit like announcing you've built a better practice track. The race hasn't happened. But you do need the track first.

I'm watching both of these closely. The ROBOSHACKLES framing, in particular, feels like it's asking a question the field needs to get much better at answering.

Sarah Williams · 25 Jun · 7 min