Guide
AI detector false positives: why base rates matter
When an AI detector flags a document, the obvious question is whether the flag is right. The less obvious question, and the one institutions most often skip, is how many flags in a whole batch are likely to be wrong. This guide explains false positive rates, shows with simple arithmetic why a low rate can still produce many wrongly flagged people, and sets out how to reduce the harm in a review process.
What does a false positive rate mean?
A detector's false positive rate is the share of human-written documents it wrongly labels as AI-generated. Its false negative rate is the share of AI-generated documents it lets through as human. The two pull against each other: a detector set to catch more AI text will usually flag more human text as well. See the glossary entry on false positive rate for a short definition.
No widely used detector claims a rate of zero. Turnitin states that its false positive rate "is not zero" (Turnitin, 2023). Its current report does not show an exact percentage or highlights when the AI score is below 20%, showing an asterisk instead, because in that range "results are less reliable and false positives are more likely" (Turnitin Guides).
A published false positive rate also only describes the text it was measured on. A rate measured on long, unedited student essays says little about short emails, technical reports or writing by non-native English speakers.
Why do base rates matter?
The base rate is how common AI-written text actually is in the batch being checked. It matters because false positives come from the human-written documents, and in most batches those are the majority. The fewer AI-written documents a batch contains, the larger the share of flags that land on people who did nothing wrong.
The worked example below uses round, made-up numbers to show the arithmetic. The figures are illustrative only. They are not the measured performance of any detector. Assume a detector that catches 90% of AI-written documents and wrongly flags 1% of human-written ones, used on 10,000 submissions.
| Scenario (illustrative) | AI-written documents | Correct flags | Human documents wrongly flagged | Share of all flags that are wrong |
|---|---|---|---|---|
| No AI use at all | 0 of 10,000 | 0 | 100 of 10,000 | 100% |
| Low AI use | 200 of 10,000 | 180 | 98 of 9,800 | About 35% |
| Moderate AI use | 1,000 of 10,000 | 900 | 90 of 9,000 | About 9% |
The detector is identical in all three rows. What changes is the base rate. When AI use is rare, roughly one flag in three in this example falls on a human writer, and if no one used AI every flag is wrong. A detector's accuracy figure cannot tell a reviewer how likely a particular flag is to be correct without knowing the base rate as well.
The rule this example shows
The share of flags that are wrong depends on the detector's false positive rate and on how common AI writing is in the batch. A flag is weakest exactly where AI use is rarest.
Who gets flagged more often?
False positives are not spread evenly. Research points to a few groups and kinds of text that carry more risk:
- Non-native English writers. Seven widely used detectors misclassified more than half of 91 human-written TOEFL essays as AI-generated, an average false positive rate of 61.22%, while classifying essays by US eighth-grade students with near-perfect accuracy. 89 of the 91 essays were flagged by at least one detector (Liang et al., 2023). The authors note that their samples were small and that most detectors tested used GPT-2 as a base model.
- Plain, predictable writing. The same study found that the essays flagged by all seven detectors had lower perplexity, meaning more predictable word choices, and that simplifying the word choices in native writers' essays raised the average misclassification rate from 5.19% to 56.65% (Liang et al., 2023).
- Short text. Turnitin requires at least 300 words of prose before it will produce an AI writing report (Turnitin Guides). Many detectors accept far shorter inputs, where there is less evidence to judge.
Independent testing of 14 detection tools concluded that they were "neither accurate nor reliable" (Weber-Wulff et al., 2023). For why each detection method can fail, see AFIP's research on how AI text detection works.
How can institutions reduce the harm?
- Estimate your base rate. Before using a detector across a group, think about how common AI use is likely to be. Where it is rare, expect a large share of flags to be wrong.
- Know the tool on your kind of text. Ask the vendor how the detector performs on text like yours: its length, language and subject. A rate measured on other text does not transfer.
- Treat every flag as a lead. A flag is grade 1 on the AFIP evidence-weight scale. It justifies a closer look, not a finding.
- Check with a different method. Run a second detector, for example the free Walter Writes AI detector, and look at drafts, version history and the author's knowledge of the content.
- Protect the groups at higher risk. Take extra care with short texts and with writing by non-native English speakers, where the research above found errors cluster.
- Record and allow appeal. Write down the evidence for each decision and give the author a chance to respond. AFIP's methodology sets out what it never concludes from a score alone.
For related guidance, see how managers should handle AI use in workplace reports, how stylometry compares a text with an author's known writing in stylometry and authorship analysis, or browse all AFIP verification guides.
Frequently asked questions
What is the false positive rate of AI detectors?
There is no single figure. Each detector has its own rate, and it changes with the length, language and style of the text being checked. Vendors such as Turnitin state their false positive rate is not zero, and an academic study found an average false positive rate of 61.22% on essays by non-native English writers across seven detectors. Ask how a tool performs on text like yours before relying on it.
Can AI detectors falsely detect AI?
Yes. A false positive is human writing wrongly flagged as AI-generated. Every detector produces some, and research has found they are more common on short text, predictable writing and writing by non-native English speakers. When AI use in a group is rare, a large share of all flags can be false positives.
What should I do if my human writing is flagged as AI?
Stay calm and gather evidence of how you wrote it: drafts, notes, sources and the version history of your document. Ask which tool was used and what score it gave, and ask for the flag to be weighed with that evidence. A second check with a different detector, such as the free Walter Writes AI detector, can show whether the result holds up, but process evidence usually settles the question.
Why does a low false positive rate still matter?
Because it applies to every human-written document checked. At an illustrative 1% rate, checking 10,000 human-written documents would wrongly flag about 100 people. When few documents in a batch are actually AI-written, those wrong flags can make up a large share of all flags.