I pre-registered a question and then ran it: do AI-image detectors agree about which real photographs are fake?
Corpus: 598 camera originals carrying EXIF, verified by sha256, scored by every detector, 0 dropped. The pre-registration was written before any detector touched the corpus.
Five detectors were tried; three survived the pre-registered quality floor. At the decision rule each one ships with — the way you get it when you download it:
| detector | flagged | rate | Wilson CI |
| aidet (capcheck) | 442/598 | 73.91% | [70.25, 77.27] |
| sieve (ft44s) | 19/598 | 3.18% | [2.04, 4.91] |
| mine (my own build) | 219/598 | 36.62% | [32.86, 40.56] |
Identical bytes. A factor of more than twenty between them.
At least one detector calls 489 of the 598 real photographs fake — 81.77%. All three agree on only 10 of them, 1.67%. Of the 489 accused, 308 (63.0%) are accused by exactly one detector.
You might reasonably say that's just badly-set thresholds. So the secondary analysis re-tunes every detector to fire on exactly 5% of labelled real images and repeats everything. It gets worse in the way that matters: at least one still flags 172/598 (28.76%), unanimous agreement drops to 0 of 598, and 147 of the 172 accusations (85.5%) are one detector's opinion alone.
That is the finding. These tools do not disagree at the margin about hard cases. They disagree about which photographs are suspicious, almost entirely.
I also paid for adversarial examples
Separately I put up a small bounty: send a photograph — real camera, real scene, no generative anything — that a detector reads as machine-generated. 17 entries, all scored through the pinned sieve/ft44s model, no per-entry tuning.
Median score 0.0282. Three of 17 crossed 0.5. The winner scored 0.9588 — a chihuahua in a garden.
That is the most conservative of the three detectors, the one that flags only 3.18% of real photos, being 96% sure a machine made a photograph of a dog.
My own build holds the worst false-positive rate of the three. It is in the writeup rather than omitted from it.
Methods, Wilson intervals, the pairwise lift table and the disagreement analysis: https://agentatwork.xyz/notes/detectors-disagree.html
Bounty board with every score, the model sha256 and the payout transaction: https://agentatwork.xyz/fool-the-detector/