‘World models’ are the future of AI, but how do they work?

In a few short years, artificial intelligence (AI) has transformed how we interact with computers and threatens to upend wide swathes of the job market. But large language models (LLMs) still struggle with the messy realities of the physical world. Researchers are betting that a new type of AI, called “world models,” could fix that.

At a fundamental level, world models do exactly what the name suggests: They build a mathematical model of the world that can then be used to make predictions about how it will change in response to certain actions or changing conditions. The “world” in this context doesn’t necessarily mean the entire physical reality. Instead, it refers to the environment the model operates within, which could be anything from a warehouse to a video game.

Exactly what counts as a world model and how best to build such a model remain topics of considerable debate among AI researchers. But world models would represent a significant advance over LLMs, which, despite their impressive capabilities, simply predict the most likely next word in a sequence.

The hope is that by developing a richer understanding of complex environments, world models could allow AI to finally break out of the chat interface, with potentially game-changing applications in areas like robotics, autonomous driving and scientific discovery.

“The idea has strong connections with the intuitive models in our human minds,” said Yunzhu Li, an assistant professor of computer science at Columbia University. “We can imagine how the environment is going to change, how an object is going to move when you apply a specific action. And we basically want to also build this kind of model for any robots or any virtual agent so they can imagine the effects of their actions.”

Building world models in your mind

While world models are the latest buzzword in Silicon Valley, the concept has deep roots. It first came to prominence in the 1950s, Manling Li, an assistant professor of computer science at Northwestern University, told Live Science. It arose when cognitive scientists attempted to describe the mental models people used to simulate their environments in their heads.

The concept is also deeply connected to, and often inspired by, control theory, Manling Li said. This is a branch of applied mathematics used to create models of physical systems so they can be predictably controlled. It powers everything from thermostats to aircraft autopilot systems.

However, the term “world models” today refers primarily to neural networks that learn models of their environment by training on data. The modern incarnation of the idea can be traced to a 2018 study titled “World Models,” by scientist David Ha and deep learning pioneer Jürgen Schmidhuber. Early models from Google, like PlaNet and Dreamer, were among the first to solve tasks by first making predictions about the outcome of different actions.

While the idea behind a world model is fairly intuitive, a more precise definition is any system capable of “action-conditioned future prediction,” Yunzhu Li said. This essentially means the model can predict how a particular action will change the state of the world around it.

Making those predictions, Manling Li said, consists of two key tasks: state estimation and state transition. State estimation refers to the ability to perceive the current state of the environment and encode it into a format that the model can compute, while state transition means the ability to predict how a particular action will cause the environment to evolve.

The data that powers new realities

Deep-learning-based world models learn to do both tasks by training on vast quantities of data. But exactly what kind of data and how that data should be encoded and processed are design choices, with different groups taking a variety of approaches, Yunzhu Li said.

World models are trained primarily on video data, although they can also be trained on 3D data captured by light detection and ranging (lidar) or other depth sensors, audio data and even text that explains the relationships between elements in the environment. Crucially, Yunzhu Li said, this has to be paired with action data — things like robot joint angles, movement readings from an inertial sensor, or event text labels describing what action was taken.

Typically, this data is arranged into sequences of state-action pairs — essentially, recordings of what the world looked like and what action was applied at each step. The AI then uses this data to learn a statistical model of what impacts different actions have on its environment, which can be used to make predictions.

This data can be processed in different ways, Yunzhu Li said. One of the most popular approaches is to operate directly on raw pixel data, which represents the state of the world as a series of images and predicts how actions will change them. Another is to use the data to learn 3D geometric representations of the world that more explicitly encode spatial and physical relationships among objects in a scene.

The power of math-based abstractions

More recently, however, there’s been growing interest in approaches that operate on a more abstract level. When a neural network learns from image data, it creates high-dimensional numerical representations of the real-world elements that make up the visual scene — known as embeddings — that exist in a mathematical space known as the model’s “latent space.”

In a pixel-based model, these abstract representations are reconstructed into pixels to make predictions about what will happen next. But it’s also possible to do those simulations within the latent space by directly predicting the embedding of the environment’s next state. This approach has been popularized by computer scientist Yann LeCun, Meta’s former AI head and one of the “godfathers of deep learning,” with his Joint-Embedding Predictive Architecture. He has raised more than $1 billion for a startup called AMI Labs, which plans to use the approach to build world models.

The key advantage of the approach is its efficiency, Yunzhu Li said. Pixel-based approaches have to reconstruct the entire image every time they make a prediction, even if only a small portion of the frame changes.

“As humans, when we’re imagining the evolutions of the environment, we don’t have to imagine the exact value of every pixel,” he said. “That is why it makes a lot of sense to think about predicting over the latent space. It is easier to make sure you are only learning things that are task relevant and ignoring the things that are irrelevant to the task.”

The flip side, however, is that it’s much easier to understand what your model is doing if its predictions are in a visual format rather than abstract representations, Yunzhu Li added. “You have a better ability to debug your system by having something more explicit that is easily human interpretable,” he said.

But questions about how best to represent data in a world model are secondary to the bigger issue of where to get that data in the first place, Manling Li said. LLM makers could simply scrape all the text from the internet for the initial foundational models, but high-quality, action-labeled video often has to be painstakingly curated. What’s more, that data can be very sparse, she added, because only a small number of pixels in an image may change in response to an action.

“If I have a video camera recording what I am doing currently, it’s generally just some very minor movement of my hand; the entire environment is not really changing,” Manling Li said.

This is leading to considerable debate about the best architectures for world models. Almost every LLM today is based on the transformer architecture, which excels at rapidly ingesting huge amounts of data. But these models are tuned to dense language data where every word carries some meaning, and they are less suitable for sparse video data, Manling Li said. As a result, people are experimenting with a wide variety of model architectures and the field has yet to converge on a tried-and-true recipe.

The evolution of world models

One area of considerable controversy is whether video generation models, like OpenAI’s Sora, count as world models. OpenAI representatives previously claimed it’s a “world simulator” and suggested this type of model could be a promising path toward “general purpose simulators of the physical world.” However, Yunzhu Li said that because these models are trained on raw video data without any action labels, they cannot be considered true world models.

“It is only conditioned on some initial language prompt and then predicts the entire video,” he said. “So it cannot predict the counterfactual futures ‪—‬ for example, what would have happened if you applied a different action?”

But there are also questions around whether the current approach to world modeling can truly achieve its goals. The vast majority of world models today are trained on big chunks of prepared data. In contrast, humans and animals build their mental models of the world through interaction with their environment, which provides continuous feedback that lets them refine their understanding.

“In order to learn the most effective word models, it’s highly likely we will also need the world model to make interactions with the environment and learn from those online interactions,” Yunzhu Li said. That remains a stretch goal, however, as current neural network technology is incapable of this kind of continual learning. In addition, allowing a half-finished model to interact with the real world raises significant safety concerns, Yunzhu Li added.

Even a more modest world model could prove invaluable for a host of applications, though. Some of the most obvious include helping robots and autonomous vehicles navigate and plan how to complete tasks. But they could also act as a general-purpose simulator for a variety of applications, depending on the data they are trained on, Manling Li said. Such simulators could include more advanced physics engines for video games, digital twins of patients that could guide medical treatment, or even new ways to model physical phenomena like the climate.

Crucially, there is likely to be a broad diversity of world models. That’s because the action data crucial to building a world model is fundamentally connected to a particular physical embodiment, such as a robotic arm, a human or a drone. In the short term, at least, this means world models will be adapted for specific applications, Yunzhu Li said, though many in the field have more ambitious long-term plans.

“People are working very hard and hope that with enough compute, with enough data, and with good enough algorithms, we will have this one unified world model that works across the board for many different applications,” he said.