By Daniel Whitmore, MSc in Industrial and Organizational Psychology
In short
Choosing a pre-employment assessment platform is a decision about evidence, not features. Six questions separate vendors: can you see the evidence behind a score, who owns the hiring decision when a model is involved, what happens to candidate data and for how long, can you export a record that survives being questioned, how are integrity controls chosen and disclosed, and what can the vendor actually show rather than assert. Ask them of every vendor on the shortlist, and prefer an answer you can see on a screen over one on a slide.
Buy the decision, not the feature list
A feature comparison is a poor instrument for this purchase, because every platform in the category has the same features written down. Question banks, timers, anti-cheating, dashboards, exports — the words match across vendors while the substance varies enormously. What does not match is what each platform can produce when a decision is challenged.
So run the evaluation backwards. Picture the moment a rejected candidate asks why, or a hiring manager overrules a shortlist, or legal asks what the process was for a role that has just drawn a complaint. Whatever you need then — the candidate's work, the rubric applied to it, the name of the person who applied it, the date — is the requirement. The rest is convenience.
One framing helps throughout: the obligation is yours, not the vendor's. US guidance on employment tests and selection procedures is addressed to employers, and buying a tool does not transfer responsibility for how a selection procedure is used. Obligations differ by jurisdiction and none of this is legal advice, but the practical consequence is constant — you will be the one explaining, so buy the thing that lets you explain.
1. Can you see the evidence behind a score?
Ask to click a score in the demo and keep clicking until you reach the candidate's actual response. A defensible platform lets you land on what the person wrote, chose, uploaded or said, next to the criterion it was scored against. If the trail stops at a number, a band, or a radar chart, then the score is the only evidence that exists and it cannot be re-examined by anyone who was not there.
The same question applies to the items themselves. Where did this task come from, who wrote it, and what makes it relevant to this role? The professional testing standards most vendors gesture at put documentation of exactly that chain at the centre of a validity argument. A vendor who cannot show how a task connects to a competency is asking you to take the connection on faith.
Then ask what happens when two evaluators disagree. Whether disagreement is visible at all, whether the earlier score is retained when someone revises it, and whether you can tell which of the two ended up in the total, tells you more about the platform's seriousness than any accuracy claim on the website.
2. Who owns the decision when a model is in the loop?
Almost every platform now has AI somewhere in it, and the useful question is not whether it is there but what it is permitted to set. Ask directly: can the system move a candidate's status without a person doing it? Can it reject, advance, or shortlist? If the answer is yes, or if the answer is a long one, you are buying a tool that makes decisions you will nonetheless have to defend as your own.
The defensible arrangement is narrow. A model can summarise a long response, transcribe an interview, draft a first read, or point out where two evaluators diverged — all of which saves real time on a panel scoring forty people. It should not produce a status, and the record should show which parts of the file a model wrote and which a named human recorded.
Ask what is logged about it: which model, which version, on what date, for each generated summary — a summary produced by a model you can no longer identify is an artefact you cannot stand behind. Some jurisdictions now attach specific duties here. New York City's Local Law 144 requires a bias audit, a published summary of its results, and notice to candidates before an automated employment decision tool is used. Whether that reaches you depends on where you hire, and it is worth asking rather than assuming.
3. What happens to candidate data?
Start with the boring, checkable facts: where is the data stored, who can reach it, and how long is it kept. Retention is the one people skip and regret, because the default is usually “forever”. Ask whether you control the retention window, whether deletion actually removes recordings and uploaded files rather than hiding them from a list, and what a candidate's deletion request triggers in practice.
Then ask about the uses you were not told about. Is candidate data used to train models? Is any candidate signal shared, ranked or compared across the vendor's other clients? The second one deserves a flat refusal wherever you find it — a hidden reputation score that follows a person between employers is not a feature, and no candidate has consented to it.
If the platform records anything — camera, screen, microphone — the questions sharpen, because recording and biometric rules vary by country and state and can carry consent requirements of their own. Ask what is captured, what the candidate is told before it starts, whether an identity photo produces a stored biometric template or is simply compared by a named reviewer, and what the retention rule is for media specifically.
- Where is candidate data stored, and which sub-processors touch it?
- What is the retention window, who sets it, and does deletion cover recordings and uploads?
- Is candidate data used for model training, and can you opt out?
- Is any candidate signal shared or compared across the vendor's other customers?
- For any recording: what is captured, what is disclosed beforehand, and is a biometric template stored?
4. Can you export a record that survives being questioned?
Ask for an export during the evaluation, not after the contract. Not a screenshot of a dashboard — an actual file, from an actual assessment, that you can open without the vendor's software. Then read it as the person who has to answer for the decision: the candidate's responses, the criteria, the scores, who gave them, when.
Two failures show up immediately with this test. The first is an export that contains conclusions but not evidence — a PDF of scores with no trace of what was scored. The second is an export that only works while you are a customer: if your record of a decision lives inside a subscription, then the day you churn is the day your defensibility expires.
It is worth asking what the platform records that you did not ask it to. Who opened a candidate's file, who changed a score and what it was before, who exported the data — an audit trail is unglamorous and it is precisely what you need when the question arrives late.
5. How are integrity controls chosen, and who is told?
Integrity features are where demos get theatrical, so ask about proportionality first. A ninety-minute regulated certification and a thirty-minute skills screen do not warrant the same controls, and a platform that offers exactly one setting has made a decision on your behalf. Ask which controls can be turned off, and what the assessment looks like to a candidate with each of them on.
Then separate what is recorded from what blocks. A signal that quietly notes a tab switch differs in kind from a control that ends someone's session, and the second belongs only where the stakes justify it. Ask what happens on a false positive — whether a flag can end an attempt automatically, or whether it surfaces for a human to read in context, which is the only version that survives contact with a candidate on a slow train.
Finally, ask what the candidate sees. Controls that are disclosed before the assessment starts, in plain language, are defensible; controls a candidate discovers mid-task are a complaint waiting to happen. And ask what accommodation looks like — extended time, alternative formats, a documented exception — because a platform with no answer there will produce integrity flags that are really accessibility failures.
6. What can the vendor show — including us?
Assessment marketing leans on words that sound like evidence: validated, normed, benchmarked, predictive. Each is a specific technical claim with an answer behind it that a serious vendor can produce — validated against what criterion, on which population, in what year, by whom, and can you read the study? Be equally direct about what a claim does not cover: a test that behaves well for one population is not thereby valid for another role, and reliability is not validity.
This is where the list costs us something, which is the point of publishing it. Our own answer to the validity question is deliberately narrow: we describe content validity — the chain from role blueprint to competency to task to rubric, expert-reviewed — and report reliability where there is enough scoring data to compute it. We do not claim that a SkillCort assessment predicts later job performance. A model here produces summaries and never a status, a named evaluator records every score, identity photos are compared by a person rather than matched by an algorithm and no biometric template is stored, and there is no cross-client candidate reputation of any kind.
Whether our exports, retention controls and integrity settings meet your bar is not something a blog post can settle. Ask for them in a demo, open the file yourself, and judge it the way you would judge anyone else's — and if an answer we give you is a slide rather than a screen, apply the same scepticism there. The most useful question in the whole evaluation is also the smallest: can you show me?
- buying an assessment platform
- vendor evaluation
- candidate data
- assessment integrity
- AI in hiring
Key takeaways
- You are buying the ability to explain a decision months later — evaluate every platform against that moment, not against a feature grid.
- Click through a score in the demo until you reach the candidate's actual response; if the trail stops at a number, the number is the only evidence there is.
- Ask whether the system can change a candidate's status on its own. A model should summarise and flag; a named human should decide and be recorded doing it.
- Get a real export during the evaluation and read it as the person who has to defend the decision — an export of conclusions without evidence is not a record.
- Treat validated, normed, benchmarked and predictive as claims that require a study, a population and a date — including when the vendor making them is us.