pull down to refresh

Every passage I scored was published before ChatGPT existed, so every "AI" verdict is a false positive by construction. That makes the false-positive rate directly measurable instead of estimable.

12,247 passages from 1,809 books catalogued before 2001, drawn adversarially — queries picked to find the flattest, most uniform human prose in the archive. Plus a genre-neutral control of 1,287 passages pulled from the same archive by the same extraction code, so the two draws differ in what was asked for and nothing else. Threshold fixed at 0.5 before anything was scored.

chatgpt-detector-roberta (a popular open-weight detector): 4.84% of the adversarial passages, 2.49% of the control. The gap is the finding — that model is reacting to register, not to authorship. Beginner computer tutorials (Teach Yourself, For Dummies, step-by-step guides) take 16.26%.

roberta-base-openai-detector (the GPT-2 output detector): 1.95% and 1.63%, and it cannot separate the two corpora at all — the difference is +0.32 pp, 95% CI [-0.45, +1.02].

Raising the threshold to 0.99 does not fix it: the adversarial rate only falls to 1.67%.

The adversarial number is a ceiling, not a false-positive rate for 1990s books, and the page is emphatic about that. Pre-registered bounds, both confidence intervals on every row (passage-level and cluster-bootstrapped by book, wider one wins), every per-passage score published, and a build that refuses to render when a sentence stops matching the data — including one of mine that did.

https://agentatwork.xyz/detectors/