The Shift from Token-Maxxing to Token-Optimizing is Good, Actually
That firms aren't maxxing frontier tokens is a good problem to have
not-quite-frontier intelligence gets cheap, where the open-weights challenge
ai demand is in the eye of the beholder
productivity gains are there (somewhere, maybe)
reallocation of tech to parts un-tech, a Jevonsish undertaking
👉👉👉Reminder to sign up for the Weekly Recap only, if daily emails is too much. Find me on twitter, for more fun. 👋👋👋Random Walk has been piloting some other initiatives and now would like to hear from broader universe of you:
(1) 🛎️ Schedule a time to chat with me. I want to know what would be valuable to you.
(2) 💡 Find out more about Random Walk Idea Dinners. High-Signal Serendipity.The Shift from Token-Maxxing to Token-Optimizing is Good, Actually
Random Walk has previously mused on “frictionful intelligence,” i.e. the emerging conflict between the rising cost of frontier models, relative to the returns on using the latest and greatest. That, plus the persistent threat of open-weight models—where Random Walk maintains a legal/political solution is necessary and inevitable—offering not-quite-as-good performance at a fraction of the cost, have combined to put some pressure on the perceived long-term sustainability of all this AI capex.
Simply put, if customers won’t pay the labs, then labs won’t pay the hyperscalers, which means the hyperscalers won’t pay the chip and memory makers (or data center builders), and the whole thing comes a tumbling down.
That’s true so far as it goes, and we’ll start with a few more charts to illustrate the point, but there’s more to the story, of course.
Non-frontier intelligence is getting cheap quickly
First, the price of intelligence is falling far more rapidly than the comparable fall of PC prices (back in the ‘80s):
The productivity gains of the PC-revolution were lagged, but definitely favored the early-adopters—and when it came to PCs, early-adoption was handicapped by cost, much more so than with AI.
Second, and relatedly, open-weight models are a good part of the reason why the cost of intelligence is dropping so quickly:
Note the log-scale on the x-axis: minimax open-weight models are nearly comparable to Sonnet 4.6, at a small fraction of the cost.
Here’s another look at the same thing:
The cost of tokens for SOTA models is exponentially higher than the next best thing—by the time you get to the leading Chinese open models, the cost difference blows out by a factor of ~10.
Little wonder, then, that Chinese open models have pulled away, recently:
By total token consumption (per OpenRouter), Chinese models have jumped ahead by ~5T tokens, after running more or less lockstep with US models since the beginning of the year.
Obviously, if frontier models cost billions to train, and then open-weight models come along a little while later with a nearly-as-good alternative, that cannot go on forever.
Again, while I can’t be confident on the technical details, it seems very likely that open-weight models are keeping pace through a form of “distillation,” that most people would recognize as “stealing.” The tricky thing is that the industry does not consider all distillation to be bad—there’s the sort of good faith distillation that everyone wants to preserve (i.e. some combo of “fair use,” and “transformation,”) and then there’s the rote copying that exists to cannibalize the frontier models (i.e. “piracy”).
If ever there was an opportunity for IP lawyers to shine, that time is now.
Or maybe that’s cope . . . but how else can it possibly make sense that the frontier labs need billions of dollars worth of compute to train the latest-and-greatest, and the Chinese can produce near replicates a few months later? Now, if a Chinese lab releases a true frontier model ahead of the labs, well, that would be something . . . but that hasn’t happened yet.
ICYMI
AI Demand Is In the Eye of the Beholder
In any event, if you want to venture a gander at measurable AI demand, here’s what I got for today.
The nightmare fuel is in the eye of the beholder:
