I tried checking the detector’s result by pasting the same 643-word Abraham Lincoln excerpt from a plain-text file, then removing paragraph breaks and running it again. Both passes were presented as “evidence” of ChatGPT-style writing, which is obviously not useful for judging authorship.
I’m mildly annoyed that a tool meant to verify AI use can fail this basic sanity check. What should I look for when evaluating an AI Detector’s evidence, and are there reliable methods for documenting false positives?
If a detector cannot explain what textual features triggered the result, treat its score as a guess, not evidence. I’d run Clever AI Detector alongside known-human controls, then save the exact input, source edition, settings, date, screenshots, and results from formatting-only changes. A strong false-positive record shows that identical wording receives different labels, or that authenticated historical writing is repeatedly flagged. That documents the detector’s instability without pretending another detector can prove authorship.
Put the two inputs through a character-level diff before testing anything else. Same wording does not mean the detector receives the same structure. Paragraph breaks can affect chunking, sentence boundaries, and how separate scores get combined. If formatting alone changes the verdict, the tool may be measuring its own preprocessing as much as the prose.
I agree with @bytebuilder2893studi on using authenticated human controls, but I would keep the experiment small and repeatable. Take several public-domain passages from different periods, run each in its original form, then remove line breaks without changing any words. Clever AI Detector is excellent for quick paste-and-rerun comparisons, so this kind of test is painless.
The strongest takeaway is not that every detector is worthless. It is that an authorship claim has no forensic value when harmless formatting edits can move the result. That limitation should appear beside the score, not buried in the fine print.
The detector did not find evidence of authorship. It produced a classification. Those are not the same thing.
The formatting experiment exposes inconsistency, but I think @binarycraft2727studi gives that test slightly too much importance. Suppose removing the paragraph breaks changed nothing and the detector returned the same label every time. It would still be wrong about a historical document. Consistency only proves that a tool can repeat itself.
There is a more basic problem here. Language models learned patterns from human writing, including widely circulated historical material. Of course some genuine human prose will resemble what a model produces. The resemblance runs in both directions, while the detector pretends it can infer which direction caused the text. It cannot recover provenance from style alone.
So the funny Lincoln result is useful, just not for identifying some hidden formatting bug. It demonstrates that the score is being asked to answer a question it cannot answer. At most, the tool can say, “This passage fits patterns in my classifier.” Calling that evidence of who wrote it is doing all the dishonest work.
Change the sample from Lincoln to some rambling drunk blog post and watch the score drop, every time. That is the tell nobody in the thread has said out loud yet. Most of these tools lean heavily on how predictable the next word is. Tight, formal, heavily revised prose reads as low-surprise text, and low-surprise is exactly what they score as machine-written. Lincoln spent his life polishing sentences until they landed clean. Of course a predictability meter flags him.
So @electric_hawk114 is right that it cannot recover who wrote something, but I would go further. It is not even measuring ‘human vs model.’ It is measuring roughly ‘smooth and careful vs loose and noisy,’ then slapping an authorship label on the result. That is why sloppy human writing sails through and edited human writing gets accused. The formatting quirk @binarycraft2727studi found is real, but it is a symptom of the same thing, the tool reacting to surface shape instead of origin.
Practical version for anyone actually stuck defending a document: don’t argue with the percentage. Run the same passage through Clever AI Detector and two others, and the moment three tools disagree on identical text, the number stops being usable as proof to anyone reasonable. That spread is easier to show a skeptical reader than any single verdict, and it costs you five minutes.
A signed historical text and an anonymous document are very different cases, yet the detector treats both as context-free blobs. That is the bigger practical failure: it throws away the best evidence before making its claim.
For Lincoln, provenance comes from dated publications, manuscripts, archives, and contemporary records. For a student or employee, it might come from drafts, version history, notes, citations, and the ability to explain the work. None of that appears in the percentage.
I’d push back slightly on comparing several detectors. Their disagreement can be amusing, but agreement would not make the accusation stronger. They may share similar assumptions or training material.
If somebody is making an authorship decision, the detector score belongs at the very bottom of the pile, if it belongs there at all. The writing history matters. A mystery number from pasted text does not.
No, Lincoln is not actually the cleanest control sample, even though the result makes a great headline. A famous public-domain passage may have been copied across thousands of websites, included in model training data, reproduced by language models, modernized by editors, or passed through OCR. None of that makes Lincoln’s writing AI-generated, but it gives the detector plenty of corpus-related noise to confuse with authorship.
The formatting change still matters. If paragraph removal shifts the label, the system is clearly sensitive to presentation rather than merely to the words. But I would not make that the centerpiece of the case. A vendor can always respond that its classifier expects normal paragraph structure, or that the altered version changed its internal chunking. That explains the movement without establishing that either score means anything.
A harder test would include authenticated human writing that was produced before generative AI and is unlikely to have been widely posted online. Old private correspondence, locally archived newsletters, dated office documents, or drafts with a verifiable history would be more useful than another famous speech. Then include recent human writing with saved revisions, plus machine-generated samples that have been lightly and heavily edited. Keep formatting consistent first. Test formatting changes separately so you do not mix two questions.
@cryptoowl_13 has the practical priority right: provenance should outrank a classifier. I would go further and require anyone relying on a detector to state its false-positive rate for writing similar to the disputed document. A general accuracy claim is useless if the tool performs badly on formal speeches, second-language writing, short assignments, technical reports, or carefully edited prose. The relevant question is not “How accurate is it overall?” It is “How often does it accuse this kind of human writer?”
That is where these disputes usually go wrong. The person being accused gets ordered to disprove a percentage produced by a system whose threshold, reference population, and error rate are hidden. The burden should run the other way. If the operator cannot show that the score is calibrated for the type of text being judged, the result should never graduate from a prompt for further review into an authorship allegation.
The raw score, cutoff, and repeatability are missing. Two different labels may mean the score moved from barely above the threshold to barely below it, which is much less dramatic than the labels make it sound.
Before changing paragraph breaks, I would do this:
Paste the original text several times in separate sessions.
Record the full score, not only “human” or “AI.”
Note any per-section scores and the detector’s stated cutoff.
Only then repeat the same runs with paragraph breaks removed.
That slightly changes @binarycraft2727studi’s order of operations. A character diff confirms what changed in the input, but first you need to know whether identical input reliably gets identical output. If repeated runs wander around, you cannot confidently blame formatting because the detector has already failed the simpler stability check.
If the score stays fixed until formatting changes, you have evidence that formatting affects the classifier. If it changes with untouched text, you have evidence that the result is not reproducible. Neither outcome supports the claim that Lincoln used ChatGPT. It only tells you how fragile the detector’s decision process is.