What online candidate assessment actually buys you

What moving assessment online actually changes

Online assessment is a delivery decision before it is a measurement decision. What you are trying to observe — whether this person can triage a queue of support tickets, read a contract closely, or reason with a table of numbers — does not change because the task arrives in a browser. What changes is administration: who receives the task, in what form, under what timing, with what help available, and what is left behind when it is over.

That distinction matters, because most of the value people attribute to "going online" actually comes from standardising administration and scoring. Paper sessions and in-person exercises could always do that. They rarely did, because holding conditions constant across eleven interviewers and four offices takes discipline nobody had time for. The browser makes that discipline cheap. It does not supply it.

So the honest way to evaluate the move is to name what it buys — identical conditions, structured scores, a durable record, and reach — and then be equally clear about what it does not buy, which is the quality of the instrument itself.

The same task, under the same conditions

The first gain is the plainest. Every candidate gets the same prompt, the same time allowance, the same reference material, and the same instructions in the same order. Nobody gets the version of the exercise that a tired interviewer explained badly at 4pm on a Friday, and nobody gets the extra ten minutes that a chatty session happened to allow.

This matters more than it sounds, because unstructured administration quietly moves the thing you are measuring. If one candidate is asked "walk me through a hard customer situation" and another is asked "here is an angry customer email, draft the reply", you have not scored two people on one skill. You have scored two people on two different tasks and then compared the numbers as though you had. Standard delivery closes that gap before scoring ever starts, which is why administration guidance sits alongside validity in the professional testing standards rather than being treated as an operational footnote.

One honest caveat: conditions are standardised, not identical. A candidate working from a shared kitchen table on a five-year-old laptop is not in the same room as one with a quiet office and a second monitor. Online delivery removes the variance you introduced; it does not remove the variance the world introduced. Design for that — generous timing, no dependency on a large screen, no penalty for a dropped connection — rather than pretending it is not there.

Structured scores instead of remembered impressions

The second gain is what happens after the candidate finishes. An in-person exercise usually produces a memory and a few notes. An online one produces an artefact — the actual reply they wrote, the actual table they filled in — which can be scored against a rubric that existed before anyone applied.

That ordering is the whole point. When the criteria are written first, the scorer is answering a defined question ("did the response acknowledge the customer's specific complaint before offering a remedy?") rather than forming an overall impression and then reverse-engineering reasons for it. It also makes disagreement visible and useful: when two evaluators score the same work differently, you can see exactly which criterion they split on and settle it, instead of averaging two impressions that were never about the same thing.

Where several evaluators score enough shared candidates, agreement between them can be computed rather than assumed, and a criterion that nobody scores the same way twice is a criterion that needs rewriting. AI has a place here, but a narrow one: it can summarise a long response or point a reviewer at the part of a transcript worth reading. It does not produce a status, and it should never be the thing that decides.

• Write the rubric before the first candidate sees the task, not after the first ten.
• Score criterion by criterion, not overall impression first.
• Keep the evidence next to the score, so a reader can check the judgement.
• Treat evaluator disagreement as information about the rubric, not noise.

A record that outlives the decision

Six months after a hire, the useful question is rarely "what was the score?" It is "what did we actually see, and were we right about it?" A file that holds the candidate's real output, the criteria applied, who applied them and when, answers that. A spreadsheet cell holding 7/10 does not.

The same file is what you need when a decision is questioned — by the candidate, by a manager who disagrees, or by anyone reviewing how the process ran. Being able to show the task, the rubric and the work is a different position from being able to show a number and a recollection. It is also the only way to improve the instrument: you cannot tell which questions were doing work and which were noise unless you kept the answers.

This cuts both ways. A durable record of candidate work is personal data, and how long you may keep it, what you must tell candidates, and what they may ask you to delete depend on where you and they are. Set a retention period deliberately, say what you collect before the assessment starts, and do not keep material simply because storage is cheap. Obligations vary by jurisdiction and none of this is legal advice.

Reach beyond who can travel

The fourth gain is access. An assessment that requires a Tuesday morning in a specific building silently filters for people who can take a weekday off, afford the travel, and arrange care for whoever depends on them. Removing that requirement widens the pool without lowering any standard, which is a rare combination.

It also makes accommodation easier to grant. Extended time, an interface that works with a screen reader, or a task that does not depend on speed of typing are configuration decisions online, where in a room they are an awkward conversation someone has to initiate in front of others. Offer accommodations in the invitation rather than waiting to be asked; most candidates will not ask.

But online delivery has its own exclusions, and it is dishonest to count only the ones it fixes. A candidate with no reliable connection, no quiet space, or no device beyond a phone is now the one being filtered out. Keep an alternative route available and make it easy to request, and be wary of any control — continuous camera capture, for instance — that quietly turns "you need a laptop" into "you need a laptop and a room where nobody walks in".

What online delivery does not buy you

A bad test online is a bad test delivered faster, to more people, with more confidence attached to the output. Scale is not neutral: an item that measures nothing in particular measured nothing in particular before, but it did so forty times a year. Now it does so four hundred times, and the numbers it produces look authoritative enough to be sorted, filtered and defended.

Nor does the channel supply job-relatedness. The reason to prefer a work sample is that it asks the candidate to do something the job actually contains, and that argument is made by the task's design — the chain from what the role requires, to what the task asks, to what the rubric rewards. Delivering an irrelevant task online does not make it relevant. Regulators looking at a selection procedure ask what it measures and whether that is needed for the job; the software it ran on is not part of the answer.

And it does not buy integrity by itself. Monitoring signals — a tab switch, a large paste, a lost fullscreen — are context for a human reader, not verdicts, and every control you add is paid for by every honest candidate in friction and suspicion. Match the controls to what is at stake. A thirty-minute screening exercise does not carry the stakes of a certification exam, and treating it as though it does costs you good applicants for no gain.

A short test of whether yours is worth running

Most of the failure modes above are visible before you send a single invitation. The questions are unglamorous and they take an afternoon to answer honestly.

If you cannot answer them, the problem is not the delivery method, and moving the exercise online will not fix it — it will only make the same mistake reproducible.

• Can you name, for each task, the part of the job it stands in for?
• Was the rubric written before the first candidate saw the task?
• If two evaluators disagree, can you see which criterion they split on?
• Six months from now, can someone re-read what the candidate actually produced?
• Would a candidate who asked "how was I assessed?" get a specific answer?
• Are your integrity controls proportional to what this stage decides?

Sources & further reading

• ITC/ATP Guidelines for Technology-Based Assessment — International Test Commission
• Standards for Educational and Psychological Testing (AERA, APA, NCME) — Joint Committee
• Employment Tests and Selection Procedures — U.S. Equal Employment Opportunity Commission
• Sackett, Zhang, Berry & Lievens (2022), Revisiting Meta-Analytic Estimates of Validity in Personnel Selection — Journal of Applied Psychology