From Language Models to World Models: The Next Data Race

Put a blue block in a drawer, close it, then open it again. Most people have a reasonable expectation about what they will find. The block has been hidden from view, but it has not simply vanished.

That small expectation contains a lot: a memory of the scene, an idea of how drawers move, and an assumption about what can change while something is out of sight. Building systems that can make useful predictions about physical events is one reason world models attract so much attention.

For teams gathering training data, this changes the brief. The interesting unit is often a sequence: what was there, what happened, and what followed.

What changes when a model predicts the world?

Language models learn patterns in tokens, the pieces used to represent language. Google’s introduction to LLMs describes their ability to predict a token or a sequence of tokens from context. Those capabilities can support explanations, instructions and many other tasks.

A world model learns to predict aspects of an environment as it evolves. In systems built for action, those predictions also depend on what an agent does. The useful question becomes: given this situation and this action, what might happen next?

There is no single design behind that name. Some systems generate future video. Others predict compact internal states that help an agent choose actions. Language, images and action information can also be combined, so the boundary between model categories is not a clean dividing line.

Dreamer is an established example of the second approach. The 2025 Nature paper describes an agent that learns a model of its environment and improves its behaviour through imagined scenarios. The earlier Google Research explanation shows how predictions can operate in compact model states rather than directly generating every future image.

The research points to several kinds of data

Google DeepMind’s 2025 Genie 3 announcement describes interactive environments generated in response to user actions. It also identifies limits, including a constrained action space and challenges with interactions between independent agents. It is a useful research example of interactive world generation; its announcement does not establish that a generated scene can accurately reproduce a particular workplace.

NVIDIA’s original Cosmos paper takes another angle: a platform with video curation, pretrained world foundation models and examples of adapting them for downstream uses. The presence of a curation pipeline alongside the models is relevant to data teams. Preparing the material is part of the system.

These examples should prompt a specific conversation before collection begins. Is the team learning visual dynamics, building an interactive environment, or improving action selection? Each goal changes what a useful recording must contain.

Keep the sequence that explains the change

In our drawer example, an image of the closed drawer says little about its contents. The earlier view supplies information that the later view hides. Cutting away that earlier moment changes the prediction problem.

Collection plans should define how much context to retain before an action and how long to continue afterward. A hand reaching for the handle, the drawer moving and the final state may all matter. A polished montage can remove exactly the continuity a team needs to study.

Timing matters too. When video is paired with robot commands or other sensors, the receiving team needs to know which observation belongs with which action. Missing frames or misaligned timestamps can make a sequence difficult to interpret.

The required signals depend on the model. A video-prediction project may begin with images and descriptions. An action-conditioned project may need recorded controls and state information. Human first-person video does not automatically provide a robot’s joint commands or measured contact forces.

Variation needs a purpose

Recording the same drawer opening repeatedly can establish a baseline. It does not tell the team much about a stiff runner, a different handle, an object blocking the motion or someone interrupting the task.

Choose variations around what the model is expected to predict. Change the starting state, the objects, the environment or the action in a deliberate way. Keep enough metadata to distinguish those conditions later.

Different outcomes are useful when they answer a learning question. A drawer that stops halfway could represent an obstruction, a deliberate pause or a person losing their grip. The footage may not reveal which explanation is correct. Record what is observable and preserve uncertainty.

This is where annotation earns its place. Task boundaries, object interactions, language descriptions and visible outcomes can help a team retrieve and interpret the sequences it needs. The label definitions should follow the intended learning use; every project does not need every possible layer.

Test the prediction your application depends on

A generated clip can look convincing while making a mistake that matters to an application. For the drawer, the team might care about whether the block remains present after being hidden, whether the drawer follows a plausible path, or whether the predicted outcome changes appropriately when the action changes.

Turn those concerns into evaluation tasks. Agree what will be predicted, over what interval, and how the result will be assessed. Use examples held apart from training, with deployment-relevant differences the team wants to test.

Generated data and simulation can expand the situations available for development. Their usefulness still needs to be checked against the target task and observations from the environment the system must handle. Keep a record of which material was captured, simulated or generated, and how it was transformed.

The next data race starts with a better brief

For a particular team, the missing resource may be quite narrow: longer continuous episodes, a rare interaction, synchronised action signals, or examples from an unfamiliar setting. A larger library will help only if it addresses the gap.

Before asking for more hours, bring the collection partner four decisions:

• What should the model predict, and how far ahead?
• Which starting conditions, actions and outcomes need coverage?
• Which views, signals and annotation layers will the learning method use?
• What sample must pass review before the programme expands?

Guardian gathers egocentric data and provides annotation, alongside remote teleoperation for robot demonstrations. Share the physical task and the prediction your team needs to improve. We can scope the collection and labels around the evidence that will make the next experiment useful.