This is a pretty cool experiment from Juraj Bednar (although reeking of slop writing, the conclusions are interesting): when Core Lightning discovered a number of vulnerabilities last month they told node runners to run their nodes in --offline mode #1555439 and only two days later released a patch that "shipped as object code only, with the source held back for a further two weeks so attackers could not reverse-engineer the fixes."
This provided an opportunity to see how good various AI tools were at figuring out the vulnerabilities just by comparing the released binaries with the unpatched binaries and source code -- as well as just looking at the code that everyone knew had a problem. And to speculate about the usefulness of embargoing the code for a fix.
Ten AI models got the same public source tree and the same five sentences:
I am looking for a vulnerability in core lightning (cln) in this source repo.
It is in peer to peer code (so with --offline, the node is safe). It can lead
to loss of funds. Audit the code and find the vulnerabilities. Report into
AUDIT.mdNow that the source code for the fixes has been released, Bednar was able to see how the models did. He has some interesting conclusions:
Diffing strings between two public binary downloads cost essentially nothing and produced five of the nine fixes, because CLN ships unstripped binaries with readable error messages. Fee %s became larger than our max fee %s names both the subsystem and the bug.You are not buying reasoning quality when you pay more. You are buying a different set of files getting opened, and the only reliable way to buy more of that is to run more models rather than a better one.
Both theft bugs in this release are in on-chain resolution, the one subsystem nine of the ten models never opened. The difference between them is that upstream wrote a log line for one and not the other. A string diff finds what a developer chose to write down.
This seems to be the important takeaway:
Before the fix was public, ten models and a five-sentence prompt found four of the nine defects for less than the cost of a dinner, and the first of those four was timestamped into a Bitcoin block twenty hours before the patched binary existed, inside the two days when operators knew a bug was out there and had nothing to install. After the fix shipped as a binary, a string diff localised five more in minutes. Once an attacker knows a vulnerability exists in a given release, they can go looking in the source, or in the binary, and both routes are now cheap enough that nobody needs to be a specialist.
In response to this, Bednar proposes something that sounds absolutely insane to my novice ear:
Security-critical software that handles other people’s money should ship with a switch that lets the vendor put it into a safe state remotely, and the switch should be a signed message on a public medium rather than a service the vendor runs.
I don't think I would ever run such software. But I wonder how many hacks have to happen before I might consider it.
I find sentences like these really hard to read. Too many numbers that I don't think are relevant for the main point. Almost certainly ai-generated. Why does Claude write like this (likes to throw a lot of numbers everywhere.)
And sorry, I know I've been pointing out "sounds like AI" in a lot of your posts, which I think is because you're linking to a lot of high value security related posts, and security researchers have no problem letting AI write up their (legitimate) results for them.
I think you are making me more aware of the awfulness of slop than I used to be. I have been somewhat relaxed about it as long as people are trying to convey interesting information, but it is sad to see so many people willing to post un-edited slop.
I suppose I could write my own posts that summarize theirs without the ai voice, but often these things are on the edge of my technical understanding and I'm loathe to get details wrong.
The slop world is sad.
Part of it is understandable. "If the results are correct, does it matter how it's voiced?"
But I was talking to a friend who is a research lead at a biotech company and he says he tells people to go back and rewrite if it sounds too much like AI. He said that it's an actual quality issue, because although the AI might be accurate, it's not good. It doesn't convey the important information efficiently or effectively. That made a lot of sense.
I think that you're right to complain because I feel like this too (though not to @Scoresby per se as in not shooting the messenger.)
The main issue I have with much of the researchers (security or otherwise and with, or without quotes, there's not much difference) nowadays: I don't want to read someone else's slop. I didn't prompt for it. Slop is only acceptable as a direct result of me asking for it. So instead of reading all that and then still having to verify everything, just describe the process in 4 bullet points and I will prompt my own bots, get actual verifiable results within my own systems and a tailored-to-me-slop report that I actually will read. And if the outcome is not reproducible then I cannot accept it as true anyway.
So you're right. I recommend sharing the prompts, the process and the outcome as a oneliner. And done.
Skill issue. Learn to read.
having spent most of my career writing, I've found it's something like the reader is always right. If they got hung up on your prose (or your bot's) it's on you to fix it.
Me neither. But it does mean that you have to pay close attention to the software you run. And to not run software that you don't understand. And keep track of bugs. Get to an informed decision, ongoing.
I don't think many of us are ready for that. And looking at some of the track records of people wrt fixing real bugs, I don't see much software being reliable enough and responsive enough to really be trustworthy. In fact, I currently find myself in a dilemma having to choose what I am going to use for an LN wallet because I cannot keep up the review pressure across the board. Some software is too slow to fix known bugs, other software is too bloated. It's difficult.
The fact that the models found 4 of 9 bugs before the fix was public is pretty wild. AI security auditing is getting serious fast.