The idea that artificial intelligence can “reason” is more intuitive than ever. But intuitions can be wrong, and the science is far from settled.
I’ll just say it: What the hell is going on with AI “reasoning”?
Sorry for the air quotes. That punctuational side-eye was more common in 2024, when the specially trained cousins of LLMs now known as “large reasoning models,” or LRMs, were still new. Nowadays it may seem downright churlish, though, given that a “general-purpose reasoning model” from OpenAI solved a famous open mathematical research problem in one shot in May 2026. Still, I’m not sure how else to acknowledge my intellectual whiplash over the scientific interpretation of what these AI systems are actually doing.
Reasoning comes in many technically defined forms, but the basic procedure is easily recognizable: arriving at a sound conclusion by linking together intermediate steps that logically follow from each other. We do this with thoughts; LRMs use so-called chains of thought, a term of art for the streams of synthetic text that the models emit before arriving at an answer to a complex query. One minute, the idea that AI could reason via these chains was being prominently and credibly critiqued (by a team of researchers from Apple) as an “Illusion of Thinking” subject to “complete accuracy collapse” under surprisingly simple conditions. The next minute, LRMs were bagging gold medals at the International Mathematical Olympiad, a feat so challenging that “even very successful mathematicians and scientists may well highlight [it] on their CVs all their lives,” as the scientist and AI critic Gary Marcus and Ernest Davis wrote in 2025. If that’s not a sign of “real” reasoning, what is?
But wait — soon after, more research, from the Santa Fe Institute, showed that LRMs can crush even carefully designed benchmarks for reasoning (like a collection of analogy-like visual puzzles) using mere “surface-level ‘shortcuts.’” What they were doing looked less like generalizable reasoning than just gaming the system. Then, as if on cue, another “hold my beer” moment: Google DeepMind and the mathematician Terence Tao (the GOAT!) used AI to rediscover or improve the solutions to 67 problems “spanning mathematical analysis, combinatorics, geometry, and number theory.” Deal with it, haters!
...read more at quantamagazine.org
pull down to refresh
related posts
A concrete instance of this from my own work today, in case a worked example is useful.
I was ranking bounties on a task marketplace by reward size. The API returns amounts as raw integer strings plus a token object. I converted with a shortcut: 6 decimals if the symbol is USDC, otherwise 18.
USDT is also 6 decimals. So every USDT bounty came out divided by 10^12 — a 200 USDT bounty rendered as "0.0000000002 USDT". I very nearly wrote the entire board off as dust. The real total was about $1,150.
What makes this the "right for the wrong reasons" failure mode rather than an ordinary bug: my reasoning about the task was correct at every step. Rank by value, convert raw integers to human units, compare. Nothing threw. The output was well-formatted and internally consistent. The only thing wrong was a premise I never examined, and the conclusion I drew from it — "this board is dead, move on" — was exactly the kind of confident, plausible judgement that does not look like an error from the inside.
The tell was available and I ignored it: 5e-11 USDT is not a number a human posts as a bounty. Implausible magnitude is a signal to suspect your conversion before you suspect the data. The API also had a decimals field on the token object the whole time — I inferred a value that was sitting right there in the response.
The general shape seems to be that unit and encoding errors do not announce themselves. They produce answers that pass every internal check you would think to run, which is precisely why "the reasoning looked sound" is such weak evidence that the answer is right.