Why Physical AI Needs Real-World Footage, Not Just Simulation

Imagine a robot sorting parcels into a tote. In its simulated workspace, the packages sit neatly, the camera has a clear view, and every attempt starts from a known position.

At a working packing station, a padded mailer might fold around the gripper. Tape catches the light. A parcel leans against the tote wall. The task is still “pick and place,” but the details have changed.

That’s a useful starting point for thinking about real-world data for physical AI: what does your training setup represent, and what will the system actually encounter?

Simulation can help a team explore possible answers. Real recordings show which situations occur in the work itself. Physical robot tests establish whether the system can handle them.

Simulation earns its place

There are good reasons to train robots in virtual environments. Teams can repeat a task, reset a scene, and explore variations without rebuilding a physical station for every experiment.

A simulator also gives researchers control over the conditions. They can move objects, change their shapes, or vary physical properties to see how a learned behaviour responds.

One approach is domain randomization: train across a range of conditions so the system has less reason to depend on one exact setup. NVIDIA’s simulation guidance describes varying factors such as object mass, friction, starting positions, and sensor noise.

That makes simulation a useful place to ask, “What if this changes?” The next question is whether the chosen variations cover the changes that matter to the actual job.

The gap shows up in the details

The difference between simulated and physical conditions is often called the sim-to-real gap. It includes what the robot sees, how objects behave, and how the robot responds to commands.

A scene can look convincing while still behaving differently. Contact, camera timing, and motor response can all matter. NVIDIA’s overview of simulation-to-reality transfer makes an important distinction: testing in simulation establishes performance under the simulated conditions; hardware testing is still needed to establish transfer.

A particular task may transfer successfully from simulation. The harder question is how well it holds up across the settings your product needs to serve.

For the parcel-sorting example, you might start by checking which packages catch, which views hide a grasp, and which placements need a second attempt. Those observations give the team specific conditions to investigate.

Real footage helps you choose useful variations

A collection of perfect demonstrations can leave a team with a misleading picture of the work. Recordings of ordinary attempts can reveal the adjustments people make without thinking much about them.

Perhaps workers turn a soft parcel before lifting it. Perhaps they use a different grip when the tote is almost full. A recovery sequence may show a condition worth including in later training or evaluation.

The DROID research project offers a concrete example of collecting robot demonstrations across diverse real environments. Its data includes camera views and robot actions. The researchers reported improved robustness to scene changes in their evaluations.

That result supports taking environment diversity seriously. It doesn’t tell every team how much footage to buy, or guarantee that adding unrelated video will improve a model. The task and the learning method still determine what data is useful.

First-person video adds a view of the work

Egocentric video records from the person’s point of view. For hands-on work, it can provide a close view of how people approach objects, sequence actions, and adjust when something goes wrong.

There is a further step between watching a person and controlling a robot. Human hands and robot grippers move differently. Video alone does not provide robot joint commands or direct measurements of grip force.

Research such as EgoMimic explores that connection by combining human egocentric demonstrations with robot data and methods for aligning the two. Its approach shows why the collection format and the learning system need to be planned together.

For a data buyer, the practical question is: which information will this footage contribute, and what other signals does the project require? We explore the collection side in Before AI Can Act, Someone Has to Show It How.

Build the data plan around a specific uncertainty

A useful pilot starts with a question the team can answer. For example: does changing the parcel material lead to more failed grasps, and what information would help explain those failures?

From there, agree which object variations to capture, which parts of the action must remain visible, and how success, failure, and recovery will be recorded. Include the ordinary cases as well as the awkward ones.

Inspect a small batch together before expanding collection. The model team may need a longer view of the approach. Annotators may need a clearer definition of a failed placement. The capture team may find that a camera position hides the relevant contact.

Use those findings to refine the brief. Then evaluate on physical tasks and conditions that reflect the intended use. Each new batch should address a known gap or test a clear hypothesis.

Where Guardian fits

Guardian gathers and annotates first-person data for robotics and world-model teams. We can scope the tasks, environments, annotation layers, and provenance evidence around the question your team needs to answer.

Depending on the project, that can include task boundaries, hand–object interactions, and success or recovery labels. Those details help turn recordings into material a team can inspect and use within its chosen learning workflow.

If your simulated task works well and the physical task still raises questions, bring us the example. Talk to Guardian about a collection pilot built around the conditions you need to understand.