Test timing and cognitive load: what a timer actually measures

A timer is a measurement decision, not a setting

In most assessment builders the time limit sits in the same panel as question shuffling and navigation, which makes it look like a preference. It is not. Changing the limit changes what the score means, in the same way that changing the rubric would. Two assessments with identical questions and different limits are two different measurements, and only one of them can be the one you meant to build.

The effect is easiest to see in two candidates who finish with the same score for opposite reasons. One worked steadily, answered eight of ten items well and never saw the last two. The other skimmed everything, answered ten items adequately and had four minutes left. If the role rewards considered judgement, the first candidate is the stronger hire and the timer has just told you the reverse. Nothing in the score reports this; only the per-item timing does.

There is a simple test for whether a limit is doing legitimate work. Write down the sentence that justifies the number. “The role handles a live queue, so working under mild pace pressure is part of the skill” is a justification. “To stop people looking things up” is not — that is an integrity control wearing a timer's clothes, and it belongs in the integrity design, where it can be chosen in proportion to the stakes rather than paid for by every honest candidate's score.

Speed and skill are different constructs

Speededness is the measurement term for what happens when a time limit stops a substantial number of test takers from fully considering the items. When speed was not part of what the assessment was designed to measure, the variance the limit introduces is construct-irrelevant: it moves scores around for reasons unrelated to the skill. ETS's review of the literature describes how this shows up structurally — rushed items near the end of a test can cluster into a separate speededness factor, and that factor can produce differential item functioning between groups who work at different paces for reasons that have nothing to do with ability.

The practical symptom is rapid guessing. Response times collapse in the final stretch, accuracy on the last items drops well below the rest of the assessment regardless of how hard those items are, and the drop is sharper for candidates who read carefully. If your item analytics show a question performing badly only when it sits late in the form, the question is probably fine and the limit is not.

None of this argues against timing. A support triage exercise where every scenario is answerable in unlimited time measures something the job never asks for. The argument is only that pace should be in the assessment because you put it there deliberately, and that you should be able to see, afterwards, whether it did what you intended.

• Completion rate on the last two items, compared with the first two — a large gap is speededness, not difficulty.
• Median time per item across the form: a steep fall at the end is rushing, not fluency.
• Score differences that appear only under the limit and disappear in an untimed pilot.
• Items whose difficulty and discrimination change when they are moved earlier in the form.

Load the task, not the interface

A candidate has finite working memory during an assessment, and every demand on it competes with the one you care about. The scenario, the constraints and the judgement call are the load you intend. Scrolling back up to re-read a customer's message, wondering whether an answer saved, discovering a rule halfway through, working out how to get back to a page — that is load the interface added, and it is paid for out of the same budget. Under a tight limit the interface tax is charged twice: once in attention and once in minutes.

The most common version of this is charging candidates for reading the rules. If instructions appear on the first page of a timed assessment, careful readers are penalised for being careful. Separating them fixes it outright: in SkillCort, instructions are their own step and the clock starts when the candidate presses start, so reading the brief never eats the time allowed for the work. It is a small mechanical change that removes a real confound.

The rest is unglamorous and worth an afternoon. Put the scenario next to the answer field rather than above a scroll. State the answer format, the file limits and whether pages can be revisited before the candidate needs to know. Say plainly whether responses save automatically. Every one of these questions, left unanswered, becomes a minute spent on your interface instead of on the work you are trying to see.

• The scenario stays readable while the candidate is answering — no scrolling away from the thing being answered.
• Rules that affect how a candidate works (revisiting pages, file types, selection limits) are stated before the timer starts.
• Autosave behaviour is visible, so nobody spends attention on whether their work is safe.
• The countdown is available but not shouting — a permanent blinking clock adds pressure that is not part of any construct.
• Anything the candidate must learn about the tool belongs in the untimed instructions step, not in the timed task.

Accommodations belong in the design, not in an exception queue

The US Department of Justice's guidance on testing accommodations under the ADA puts the principle plainly: accommodations exist so that results reflect a candidate's aptitude rather than their disability, and extended time is the most common one. Obligations differ by jurisdiction and this is not legal advice — but the design consequence is the same everywhere. If a limit is tight enough that removing it changes who passes, it is tight enough that an accommodation decision has to be routine rather than improvised.

The accessibility standards make the same point from the interface side. WCAG 2.2 Success Criterion 2.2.1, Timing Adjustable, is a Level A requirement that users be able to turn off, adjust or extend a time limit — with an exception where the limit is essential and extending it would invalidate the activity. Assessments often qualify for that exception, which is precisely why the justification sentence from the first section matters. If you cannot say why the limit is essential, you have failed a measurement test and an accessibility test with one omission.

Mechanically this should be boring. In SkillCort, extended time is a per-candidate multiplier between 1x and 3x attached to the invitation, and it applies to every timer in that assessment rather than to one section someone remembered. Grant it when the invitation is sent, record who approved it, and keep it out of the evaluators' view so it cannot leak into scoring — a candidate who needed extra time is not a candidate who scored worse.

• Decide the accommodation route before the assessment goes live, not when the first request arrives.
• Apply the multiplier to the whole assessment, so no single section quietly stays at the standard limit.
• Record who approved the accommodation and when, in the same file as the rest of the decision evidence.
• Keep accommodation status out of the scoring view; evaluators judge the work, not the arrangements.

How to decide whether a section should be timed at all

The decision is per section, not per assessment, because a single assessment usually mixes constructs. A knowledge check, a written work sample and a triage exercise ask for different things, and giving all three the same treatment is how a well-designed assessment ends up measuring one candidate's typing speed.

Start from the question the section exists to answer, then ask whether pace is part of that question in the real job. Where the honest answer is no, you can still bound the sitting — for scheduling, for fatigue, for containment — but the bound should be generous enough that nobody capable of the work is cut off by it. Where the answer is yes, time it, disclose it in the instructions, and say why, because a candidate who knows pace is being assessed will work differently and that difference is the thing you want to see.

• If removing the limit would change who passes, and pace is not part of the job, the limit is measuring the wrong thing.
• If you are timing to prevent lookup, choose an integrity control instead and set it in proportion to the stakes.
• If the section is a rehearsal for pressure the role really applies, timing it is the honest choice — say so out loud.

Setting a limit you can defend

Pilot the section with people who already do the job, and use their times as a floor rather than a target. Take the slowest competent finisher, not the median, and add headroom — you are trying to exclude nobody who can do the work, and the pilot group is more familiar with the material than any candidate will be. If pace is genuinely part of the construct, tighten deliberately from that floor and write down how far you tightened and why.

Prefer the largest workable unit. A bound on the whole sitting distorts less than a section limit, and a section limit distorts far less than a per-item countdown, which turns every question into a small speed test whether or not you meant it to be. Reserve item-level timing for the rare case where the item itself is about reacting quickly.

Then check the limit against real candidates once you have them, and keep the reasoning next to the number. When a hiring manager asks why the exercise was forty-five minutes, or a candidate asks the same thing, “the role works a live queue, and forty-five minutes is the slowest of five internal pilots plus a third again” is an answer that survives being read back. “Forty-five felt about right” is the same decision with the evidence removed.

Sources & further reading

• Cintron (2021), Methods for Measuring Speededness: Chronology, Classification, and Ensuing Research and Development — ETS Research Report RR-21-22
• ADA Requirements: Testing Accommodations — U.S. Department of Justice, Civil Rights Division
• Understanding SC 2.2.1: Timing Adjustable (WCAG 2.2) — W3C Web Accessibility Initiative