Before AI Can Act, Someone Has to Show It How

Watch someone pack a mug into a box. It looks simple. They find the handle, lift the mug, adjust their grip, and settle it into the packaging. If it catches on the edge, they pause and try a different angle.

Most of us barely notice those decisions. For a team building an AI system that needs to understand physical work, they’re the interesting part.

A video of the finished parcel tells you what happened. A carefully collected, thoughtfully annotated sequence can help you study how it happened: what the person touched, when the action changed, and whether the task succeeded.

That’s the work behind data gathering and annotation. It starts long before anyone draws a box around an object.

Start with the task you want to understand

“We need more video” is an understandable starting point. It leaves a lot unanswered, though.

Which task? In what setting? From whose point of view? Does the team need to recognize a mug, identify the moment a hand grasps it, or distinguish a successful placement from a failed attempt? Those are different data briefs.

For our packing example, a useful brief might cover several mug shapes, different work surfaces, and both smooth attempts and awkward ones. It should also say which camera view and sensor information the project needs, and what makes a recording usable.

The aim is to collect examples that reflect the conditions the system will encounter. Google’s guidance on framing machine-learning problems makes this practical point: data needs to be available, correct, and representative of the real world.

That gives a collection team something concrete to work toward. It also gives reviewers a reason to reject a beautiful recording that misses the action the model actually needs to see.

The useful moments are easy to miss

First-person, or egocentric, video records a task from the person’s point of view. It can bring the camera close to the hands, tools, and objects involved.

That perspective comes with practical challenges. A hand can block the object. A quick head turn can move the task out of frame. An action that feels obvious to the person doing it may be difficult for someone else to interpret.

Imagine our packer reaches for the mug and finds that its handle faces away. They turn it before lifting. If the recording starts too late, that adjustment disappears. If the hand covers the handle, a reviewer may need to mark the contact as uncertain.

A good collection brief leaves room for these moments. It defines what must be visible and which variations matter. It also makes space for failed attempts and recovery, rather than collecting only tidy demonstrations.

Annotation turns a recording into something you can work with

Annotation adds structure to what’s been recorded. Depending on the project, that might mean identifying objects, marking hand contact, dividing a task into steps, or describing an outcome.

In our example, “packing a mug” is a useful overall description. A team studying individual actions may also need to know when the reach begins, when the grasp is secure, and when the mug is released inside the box.

Those boundaries need shared definitions. Does “place” end at first contact with the packaging, or when the person lets go? What happens if they lift the mug back out? How should an annotator handle an action that is partly hidden?

Clear instructions, worked examples, and an agreed way to escalate uncertainty help people make consistent decisions. Google’s discussion of human raters also recommends comparing judgments and improving instructions when discrepancies appear.

A disagreement can be useful. It may reveal an unclear rule before that rule spreads across the rest of the dataset.

Quality needs a closer look than an hours count

Ten hours of footage can contain repeated tasks, unusable angles, or gaps in the situations a team needs to study. An hours count tells you the volume. Reviewing what’s inside tells you whether it fits the brief.

A small pilot lets collectors, annotators, and the model team inspect the same batch. Are the important moments visible? Are the labels consistent? Can the receiving team use the files?

Model evaluation needs care too. Google’s guidance on dividing datasets recommends unseen test examples and keeping duplicates out of training and test sets. With video, closely related clips can also make evaluation easier than real deployment.

The pilot should end with decisions: what to keep, what to change, and what must pass review before collection expands.

Know where the data came from

There’s another part of a useful dataset that you won’t find by watching the video: its history.

Who contributed the recording? What permission covers its use? Which device captured it? Was it trimmed or anonymized afterward? Can those changes be traced?

Agree how to handle contributor consent, site permissions, bystanders, and sensitive details before recording begins. Keep a clear record so the receiving team can understand what it has.

At Guardian, we gather and annotate first-person data for robotics and world models. We plan the task being captured and the information being labeled together.

Our annotation layers include task segmentation, hand–object interaction, and outcomes. Provenance tiers describe the evidence attached to each clip. The scope depends on what your team needs to learn.

A better first conversation

If you’re planning a data project, bring a specific task to the first discussion. Show a normal attempt. Show an awkward one. Point out the moment your current data doesn’t explain.

That’s a useful place to start: a real task, a clear learning goal, and an honest look at the missing examples.

Tell Guardian what your robot needs to learn. We can scope the collection and annotation requirements together.