By Oliver Bennett, MSc in Work and Organizational Psychology
In short
There is no reliable way to determine from text alone whether a response was written by a person or a language model, and classifiers that attempt it produce false positives at rates that make them unsafe for decisions about individuals. The error falls unevenly, with second-language writers among the most often misclassified. The workable response is to design tasks whose answers a model cannot supply, rather than to adjudicate authorship after the fact.
Every assessment conversation now contains the same question, usually phrased as a feature request: can you tell us whether a candidate used AI to write this?
The honest answer is no — not from the text, not reliably, and not in a way that survives a candidate asking how you know. That answer tends to disappoint, and it is worth more than the alternative, because the alternative is a confident accusation you cannot evidence.
What a detector is actually doing
AI text classifiers work on statistical properties of writing: how predictable each word is given the ones before it, how uniform sentence length and structure are, how vocabulary is distributed.
These are real differences in aggregate. They are also properties that careful human writing shares. A candidate who writes in clear, well-structured, moderately formal prose — the register a professional assessment invites — produces text that looks, on exactly these measures, like model output.
The overlap is not a defect that more training data removes. It is the consequence of measuring surface regularity when the thing you want to know is who typed it, and the two are only loosely related.
The arithmetic nobody runs
Take a detector that is right 90% of the time in both directions, which is better than most published evaluations report. Suppose 5% of your candidates used a model.
Out of 1,000 candidates:
| Used a model (50) | Did not (950) | |
|---|---|---|
| Flagged | 45 | 95 |
| Not flagged | 5 | 855 |
140 candidates are flagged, and 95 of them — about two thirds — did nothing wrong.
This is not a failure of the detector. It is population arithmetic: the honest group is nineteen times larger, so even a small error rate applied to it produces more false accusations than the true positives from the small dishonest group. Pushing accuracy to 95% leaves roughly half the accused innocent. The structure of the problem, not the quality of the tool, is what makes it unusable for decisions about individuals.
The errors fall unevenly
If the false positives were random, this would be merely bad. They are not.
Text flagged most often shares a profile: shorter sentences, more common vocabulary, more predictable constructions, fewer idioms. That description also fits a great deal of competent writing by people working in a second language — and it fits candidates who write plainly because the brief asked for a clear customer reply rather than an essay.
So a detector deployed at scale does not distribute its errors evenly across your applicant pool. It concentrates them on groups already disadvantaged in written assessment, and it does so invisibly, because an accusation generated by a tool rarely arrives with its error rate attached.
An unevidenced accusation is worse than no check
Consider what happens downstream. A candidate is flagged. Nobody can show them what the evidence was, because the evidence is a probability score derived from properties of their prose. They cannot rebut it — there is no version of "I wrote this myself" that moves a classifier. And in most processes they are never told, so the rejection simply arrives.
That is the shape of an unappealable decision made on evidence that cannot be examined. It is the exact failure mode a structured assessment process exists to avoid, reintroduced by a tool bought to protect integrity.
A process with no detector at all makes fewer of these errors, because it makes no accusations it cannot support.
What actually works: change the question
The productive move is upstream. If a model can produce a good answer to your task, the task was measuring something a model can do — and that was true before anyone cheated.
Tie the task to a specific case. Not "write a reply to a frustrated customer" but a reply to this customer, whose account history, previous ticket and contract terms are in the brief, where the right answer depends on a constraint only this scenario contains. A general model has no access to specifics, and a candidate who pastes a generic reply produces something visibly unmoored from the case.
Assess the reasoning, not only the artefact. Ask what was considered and discarded, which rule was applied and why, what assumption was made where the brief was silent. These are answerable by someone who did the work and awkward for someone who received it.
Use a live follow-up at the stage that matters. A ten-minute conversation about a submitted response separates the two situations quickly and without accusation. "Walk me through why you handled the refund that way" is not an integrity check; it is a good interview question that happens to be impossible to prepare for by proxy.
Decide the policy and say it. Whether AI is permitted is your call and either answer is defensible. What is not defensible is leaving it unstated and then penalising a candidate for guessing wrong.
Where signals still earn their place
None of this means behavioural signals are worthless. A large paste into an answer field, or question text reappearing in a response, points a reviewer at something worth reading.
But they point; they do not conclude. Each has innocent explanations — drafting in a text editor, quoting the brief back deliberately — and each is a reason for a person to read the work, which is the step that can actually reach a judgement.
That is the whole of the honest position. The work is the evidence. A signal says where to look. A classifier that says who wrote it is answering a question it cannot see.
- AI detection
- AI-generated text
- assessment integrity
- false positives
- work-sample design
Key takeaways
- Detection is a classification problem on a base rate that makes it unworkable: even a good classifier accuses mostly innocent people when few candidates are cheating.
- Misclassification is not random. Writers with simpler sentence structure — including many second-language candidates — are flagged disproportionately.
- An accusation you cannot evidence is worse than no check at all, because it is unappealable and it lands on the candidate who cannot argue back.
- The design response works: a task tied to a specific brief, with reasoning assessed alongside the output, degrades gracefully whether or not a model was involved.