Thoughts · · By Ben Paget
The AI godfather is betting against the entire AI industry
Yann LeCun, one of the "godfathers" of AI, thinks the whole industry is betting on the wrong horse, and he has raised about $1 billion to prove it.
In September, he did a bit of maths in front of a packed lecture hall at Sciences Po (in French; the translation is mine).
Take all the public text on the internet that today's large language models are trained on. That's about 20 trillion words, or roughly 10¹⁴ bytes once it's cut into tokens. It would take any of us around half a million years to read.
Now take a four-year-old. By that age, a child has been awake for about 16,000 hours. Their optic nerves have about 2 million fibres (a million per eye), each carrying roughly a byte per second to the brain. Multiply it out and you get... about 10¹⁴ bytes.
Same number. A four-year-old has taken in as much data as the biggest LLM ever trained on, through their eyes alone 🤯
It's back-of-the-envelope maths, and people do argue with it: video is much more redundant than text. LeCun's answer is that redundancy is exactly what you need to learn from raw observation. Either way, the conclusion holds: « Il y a toute une partie de la réalité qui n'est pas présente dans le texte. » A whole part of reality just isn't in the text.
Text is the easy part
I use LLMs every day, and I build things with them that I couldn't have built on my own a couple of years ago. So this isn't an anti-LLM post.
But I do think we've become a bit obsessed. Most of the money and the roadmaps now assume the answer is a bigger language model. At an AI symposium in New York in October, LeCun said Silicon Valley "seems completely hypnotized by LLMs".
We have models that pass the bar exam, write code and prove theorems. We don't have a robot that can clear the table and fill the dishwasher, which most 10-year-olds manage the first time you ask. A teenager learns to drive in about 20 hours of practice; we have billions of hours of recorded driving, and still no car that drives itself everywhere. According to LeCun, no AI today understands the physical world as well as an alley cat 🐈
He explains: "Text is easy". Predicting the next word means picking from a list of maybe 100,000 options. The real world is continuous and noisy, and most of it can't be predicted.
So what's a world model?
It's something you do all the time without noticing. Push a glass at its base and it slides. Push it near the top and it tips over. Push the table just as hard and nothing happens. You learned that as a baby (gravity clicks at around nine months), first by watching, then by poking things.
(enlarge)A world model is the machine version of that intuition: given the state of the world now and an action you're thinking about, predict what happens next. Once a system can do that, it can plan, by trying out sequences of actions in its head before doing any of them.
The hard part is training it. The LLM recipe (predict the next word) doesn't transfer to video. LeCun's example: film the room, pan the camera to the right, stop, and ask a model to continue the video. It can guess the camera keeps turning. It has no way of knowing what every person on the other side of the room looks like. Make a model predict every pixel and, in his words, "you basically kill it."
This is what JEPA tries to fix (Joint Embedding Predictive Architecture, which LeCun proposed in 2022). Instead of predicting pixels, the model learns an abstract representation of what it sees and makes its predictions in that space. It keeps what's predictable and drops the rest. His analogy is physics: you can predict how much the pressure rises in a sealed box of air when you heat it (PV = nRT) without tracking a single molecule.
(enlarge)Where it's at
In the last few weeks, LeCun and Randall Balestriero published a paper (opens in a new tab) in which a 109M-parameter text JEPA lined up with image models better than every language model they tested, up to 72B parameters. Another, H-JEPA (opens in a new tab), plans at several levels at once (get to the airport; under that, hail a taxi; under that, walk out the door...) and took success on one navigation benchmark from 18% to 73%.
Meanwhile, Google DeepMind showed Genie 4 (opens in a new tab), and AMD agreed to buy Fei-Fei Li's World Labs for about $8.2 billion. (Careful though: "world model" doesn't mean the same thing everywhere. Genie generates the world you see; JEPA deliberately doesn't.)
It's also a European story. LeCun's company, AMI Labs, is headquartered in Paris and raised about $1 billion in its seed round. His pitch to the students was that Europe may have lost the LLM race, but not the next one.
None of this is settled. AMI Labs has no product yet. Under a Reddit thread about LeCun's latest "scaling LLMs won't get us there" claim, one commenter wrote that his talks convinced them to do their research on JEPA, "which is the most miserable experience. It just doesn't learn." (opens in a new tab) And an engineer who ran 600 experiments (opens in a new tab) on a JEPA world model found that a standard benchmark was easier than it looked: success dropped from 68% to 44% on fresh tasks.
It's early. The last time LeCun spent decades on an unfashionable idea, it was neural networks.
Watch the Sciences Po talk (opens in a new tab) (the optic nerve bit starts at 38:29) and the shorter English version at the Evident AI Symposium (opens in a new tab).
LLMs learned about the world by reading about it. The next ones might learn by watching it.







