Cheating detection in skills assessments: how to read a signal

A signal is an observation, not a finding

What a monitoring control actually records is narrow and mechanical: at 14:22:07 the exam surface lost focus; at 14:23:41 nine hundred characters entered an answer field in one event; between the third and fourth task there were four minutes with no input. Each of those is true, timestamped and uncontested. None of them is a statement about the person. The move from “this happened” to “this candidate cheated” is an inference, and it is made by a human being who can be right or wrong — never by the sensor, which has no access to the only fact that matters.

Keeping that distinction visible in the language you use is not pedantry; it is what stops a process drifting. Once a dashboard column is called “cheating risk” instead of “events recorded”, reviewers start reading a judgement that no measurement supports, and the number begins to feel like something to act on rather than something to check. The federal Testing Integrity Symposium convened by the National Center for Education Statistics reached the same conclusion about far more sophisticated methods than a focus listener: computer-based detection may reduce false positives and produce stronger evidence, but it is still inferential and “still proves nothing. It takes [an] investigation to do that.”

So the useful question about any signal is not “how accurate is it?” but “what would I have to add to it before it could support a decision?” That question has a real answer for each family of signal, and the answer is almost always the candidate's work.

Three questions before you read the flag itself

The first is whether the monitoring was disclosed and deliberately chosen. Candidates are told what is monitored before an assessment begins, and integrity controls are selected per assessment rather than switched on everywhere by default — which means a flag from a control you decided this assessment needed carries different weight from one produced incidentally. If you cannot say why this control was on for this assessment, you are not yet in a position to hold its output against anyone.

The second is what innocent looks like. Before opening a flagged session, write down the honest explanation you would expect to see and how you would recognise it. A notification stealing focus for two seconds mid-task. A candidate pasting the notes they drafted in the box above. A four-minute gap that is a person thinking, or getting a glass of water, or reading the scenario again. If you cannot describe the innocent version in advance, you will find the guilty one, because that is the only one you brought with you.

The third is whether the signals are independent. A dropped connection can fire a focus change, a fullscreen exit and a screen-share interruption within the same second, and a reviewer counting three flags will feel three times as certain as one who understands they are three views of one event. Correlated sensors inflate confidence far faster than they add information — read the timeline, not the total.

• Was this control chosen for this assessment, and disclosed to the candidate before it began?
• What does the innocent version of this flag look like, described before you open the session?
• Are these separate events, or one event seen by several sensors?
• Does the flagged moment line up with anything unusual in the work itself?

Why most flags on an honest cohort are wrong

This is arithmetic about any detector, not a property of a particular product. Suppose the great majority of a cohort is honest, which is the normal case. A control that fires on even a small fraction of honest sessions is taking that fraction of a very large group, while the guilty sessions it catches are a share of a very small one. The honest flags outnumber the guilty ones — not because the control is badly built, but because the population is lopsided. Making the control more sensitive makes this worse, not better.

The consequence is practical. A flag is worth roughly what the follow-up work is worth, so the value of a detection system lies in how efficiently it directs attention, not in how many alerts it produces. A control that fires often and is never read is worse than no control, because it manufactures the appearance of rigour while training reviewers to click through.

It also explains why an unreviewed flag should never touch an outcome. When most flags on an honest cohort are innocent, an automatic consequence attached to a flag is, in the aggregate, a policy of rejecting honest candidates. That is the whole argument for keeping a person in the loop, and it survives any improvement in the sensor.

What each family of signal can and cannot support

Different signals have genuinely different evidential weight, and treating them as one undifferentiated pile of “flags” is how weak signals get used as if they were strong ones. The table below is the short version of what each family will and will not carry.

Two rows deserve emphasis. Identity checks are a human comparison: a named reviewer compares the photo taken at check-in with the document the candidate presented, and their name and the date are recorded with the result. There is no automatic face matching and no biometric face template is stored — so the evidence a reviewer produces is a person's judgement, attributable and open to review, which is exactly what it should be. And behavioural signals such as typing rhythm are advisory at best; assistive input, an IME and a genuinely fast typist all produce patterns that look unusual, so a candidate using accommodations should never be weighted more heavily because of them.

From flag to decision: a reading procedure

Read the work first. A response that engages the particular angry customer in the prompt, in a voice consistent across every task, is hard to fake regardless of what the event log says — and a polished three-paragraph answer that never touches the scenario's specifics exposes itself with no sensor involved. Most flags dissolve inside a minute of this kind of reading, which is the point of having the work and the timeline side by side.

For flags that do not dissolve, escalate like an investigator rather than a judge. Weigh the pattern instead of the incident, bring in a second reviewer for anything that could change an outcome, and where real doubt remains prefer a follow-up — a short live exercise, a conversation — over a silent rejection. The International Test Commission's security guidelines put the underlying principle simply: test takers suspected or accused of test fraud have the right to due process, and a score should be invalidated where it is shown to be inaccurate, not where a sensor found it suspicious.

Record the reasoning either way, in the same decision file as the scores. In the United States, the Uniform Guidelines on Employee Selection Procedures treat any procedure used as a basis for an employment decision as a selection procedure — so a flag that influences an outcome is part of your process and inherits the documentation expectations that come with that. Obligations differ by jurisdiction, none of this is legal advice, and no product claim of compliance is being made here; the practice of writing down what was flagged, what the reviewer examined and why they concluded as they did is defensible everywhere regardless.

• Open the response before the event log; judge whether the work is the candidate's on its own terms.
• Place the flag on the timeline and check whether it coincides with anything odd in the output.
• Get a second reviewer on anything that might change the outcome.
• Where doubt survives, invite a short live demonstration rather than rejecting quietly.
• Write down what was flagged, what was examined and what was concluded — including when the conclusion was “nothing”.

Four things a signal can never do

It cannot reject a candidate. No integrity signal produces an outcome on its own; every consequential reading is a named person's, recorded as theirs. A platform that auto-rejects on a threshold has not automated a judgement, it has hidden one.

It cannot follow a candidate. A flag belongs to one assessment and one hiring decision. There is no cross-client candidate reputation and there never will be — a person who was reviewed and cleared, or reviewed and rejected, arrives at the next employer with nothing attached.

It cannot substitute for disclosure, and it cannot be a compliance claim. Monitoring is disclosed before an assessment begins, both because candidates who know the rules can follow them and because guessing at invisible rules produces exactly the strange behaviour that pollutes your signals. And no combination of controls makes a process lawful in a given jurisdiction; that assessment is yours to make, with advice, against the obligations that apply to you.

What a signal can do is worth keeping in view. It gives a reviewer a place to look, a moment to compare, and a record that the looking happened. Used that way it protects the assessment and the candidate at the same time — which is more than a verdict engine has ever managed.

Sources & further reading

• Testing Integrity Symposium: Issues and Recommendations for Best Practice (2013) — U.S. Department of Education, National Center for Education Statistics
• International Test Commission (2014), ITC Guidelines on the Security of Tests, Examinations, and Other Assessments
• Uniform Guidelines on Employee Selection Procedures, 29 CFR Part 1607 — eCFR