On The Too-Well-Known But Seldom Acknowledged Limitations of LLMs ➹

I really didn’t have time to comment on The Illusion of Thinking paper before a bunch of knee-jerk reactions started coming in from the “LLM religion” side of the industry, so I decided to let it blow over even though I was really tempted to comment on the more idiotic takes that pointed to the paper as a way for Apple to minimize the importance of LLMs and thus (as the conspiracy theory goes) distract people from having dropped the ball.

There was already enough idiocy online, and feeding both anti-Apple and pro-LLM trolls would benefit nobody.

But this piece does paint a good summary of where we are at, and looking back, it’s interesting to see that many people still assume probabilistic token generation is tantamount to thought (let alone logic) to the point where a single simple, completely deterministic experiment has raised so many hackles.

The key point for me is not the exact test Apple uses–it is, rather that even with a step-by-step solution laid out, the models “overthink” (i.e., there’s a probability cascade that misses the mark) and stumble in a way that’s as revealing as it is disappointing.

I’ve already given up on the AI bubble bursting–there’s too much riding on the premise that the current tech, even as fallible and unwieldy as it is starting to prove itself, can translate into (somewhat elusive) business value and even political clout, so I now expect GPU investments to follow along pretty much the same track as, say, nuclear missiles even as the software model design side keeps exhausting the possibilities of brute force compute.

But I don’t see a lot of reason to complain that Apple has just pointed out that throwing dice and using them for divination isn’t reasoning–it’s always been kind of obvious.