One of the most stressful experiences in robotics is a live demo. The robot has usually been working for days or weeks. The route has been rehearsed. The environment has been prepared. Then it’s time for the demo and you’re still on edge wondering if it will work.

I’ve participated in a lot of robotics demos over the years. Looking back, one thing I’ve noticed is that people tend to walk away believing they saw more than they actually saw.

Not because anyone is trying to be deceptive.

Mostly because demos are very good at hiding things.

Demos Hide Variance

The easiest robot demo is a video.

Videos are great because robots don’t know they’re being filmed. If something goes wrong, you stop recording and try again. If it goes wrong again, you try again again. Eventually the robot does the thing you wanted it to do and everyone goes home happy.

The finished video looks like a seamless demonstration of capability.

What the video doesn’t show is how many attempts it took to get there.

I remember filming a robot arm picking up a cube. The task seemed simple enough. We hit record.

The robot missed.

We tried again.

The robot missed again.

A few more attempts later we finally realized we had the wrong version of the software installed.

Once we fixed it, things worked much better.

If all you saw was the final video, you’d conclude that the robot picked up the cube.

That’s true.

What you wouldn’t know is whether it succeeded on the first attempt, the second attempt, or the tenth attempt.

The robot succeeded. The variance disappeared.

Demos Hide Constraints

One demo I worked on involved a home robot operating in a mock living room built on an exhibition floor.

To most people it looked like a living room.

To the engineers it looked like a carefully constructed robotics environment that happened to contain a couch.

There was more space between furniture than many people have in their homes. We also used fiducial markers on some objects because object localization wasn’t always reliable on its own.

None of this was dishonest. We were trying to answer a specific question.

Could the robot perform useful tasks in a home-like environment?

That’s a much smaller claim than “the robot works in homes.”

The robot wasn’t cheating. Neither were we.

But successful demos have a way of making carefully chosen constraints disappear from memory.

Demos Hide Problems

One of the more impressive demos I participated in was at NASA. We had Robonaut traversing a mockup of the International Space Station inside the gravity offload system and interacting with a task panel.

From the audience’s perspective, the robot appeared to be moving through the station and performing useful work on its own.

From my perspective, there was a group of anxious engineers watching every step.

Commands were retried.

Waypoints were adjusted.

Errors were cleared.

The demo successfully proved that we could perform useful work remotely.

What it didn’t prove was that the robot could operate independently without supervision.

The interesting part is that the audience never saw the recoveries. They only saw the successful outcome.

The robot encountered problems.

The system recovered.

But much of that recovery happened outside the robot itself.

You should be skeptical of conclusions drawn from demos, even when they’re impressive.

A demo can show that something is possible. What it usually can’t tell you is how often it works, how broadly it applies, or how much support was required to make it successful.

But that’s usually okay. Those aren’t the questions the demo was designed to answer.

The mistake is forgetting how much work sits between “it worked” and “it works.”