Psychometric properties explained: reliability, validity, and the difference that matters

Two properties, two different questions

Reliability asks whether a measurement holds still. If the same candidate sat an equivalent version of your assessment next week, or if a second evaluator scored the same piece of work against the same criterion, how close would the two results be? Notice what that question does not contain: any reference to correctness. A reliable measure is a repeatable one, and repeatability says nothing about whether you are measuring the right thing.

Validity asks the other question. Does the score capture the thing the decision needs — the ability to do this job, this well? Where reliability is about precision, validity is about aim. The distinction is easiest to hold onto with a target in mind: reliability is how tightly your shots group, validity is whether the group sits on the bullseye. Tight grouping in the wrong corner is a real result, and a useless one.

The two are connected in one direction only, and it is worth knowing which. A measure that cannot repeat itself cannot be measuring anything stable, so unreliability puts a ceiling on validity. But the reverse does not hold: consistency buys you nothing on its own. Reliability is a necessary condition, never a sufficient one, and the whole of this article is really an elaboration of that sentence.

Reliability: would the same candidate get the same result again?

Reliability is not a single statistic but a family of them, distinguished by what source of inconsistency they are looking at. Three matter in hiring, and it is worth being able to name which one a number refers to, because they are not interchangeable.

Each of those describes a different threat. Internal consistency asks whether a set of items hangs together as a measure of one thing. Inter-rater agreement asks whether the score belongs to the candidate or to the person who happened to grade them — the most common and most correctable source of noise in human-scored work. Test–retest asks whether the result survives the week. A number quoted without saying which of these it is has told you almost nothing.

It is also worth separating this from the level below it. How individual questions behave — how many candidates get one right, and whether the strong candidates are the ones getting it right — is item analysis, and it is covered in depth in our post on item quality signals. Reliability is a property of the whole score rather than of any one question, and the two levels can disagree: a section can be full of individually healthy items and still be unreliable, because the items are each fine and are not measuring the same thing.

• Internal consistency — do the items within a section behave as though they measure one construct? Reported as Cronbach's alpha or McDonald's omega.
• Inter-rater reliability — do two evaluators scoring the same candidates on the same criterion agree? Usually reported as an intraclass correlation coefficient.
• Test–retest and parallel forms — would the same person score similarly on a second occasion, or on an equivalent version of the assessment?

Reliable and wrong: the bathroom scale problem

A bathroom scale that reads two kilos heavy is a perfectly reliable instrument. Step on it five times and it gives you the same figure five times; hand it to a colleague and it behaves identically. It is also wrong every single time, and no amount of repetition will reveal the error, because the error is in the calibration rather than in the variation.

The hiring version is a timed multiple-choice quiz on product trivia. Its reliability is likely to be excellent: the answer key is unambiguous so two graders agree completely, and the same candidate would score much the same next week. Those clean numbers make the assessment look rigorous in a report. What they cannot tell you is that the quiz measures recall of a manual, while the job it screens for is about judgment under an unhappy customer's time pressure. High reliability made a mis-aimed measure look dependable.

This is why reliability is reported alongside validity and never instead of it. The reliability coefficient is much the easier number to produce — it needs only your own response data, no outcomes and no argument — which is precisely why it is the one that ends up quoted on its own.

Validity is an argument, not a number

The most useful shift in how validity is treated is that it is no longer thought of as a set of separate 'types' you collect one of. The Standards for Educational and Psychological Testing, published jointly by AERA, APA and NCME, treat validity as a single unified judgment about whether the interpretation you place on a score is justified for the use you are putting it to. What varies is not the kind of validity but the kind of evidence supporting the argument — and the same test can be well supported for one use and unsupported for another.

Content-related evidence is the most inspectable kind, and the least glamorous. It asks whether the tasks represent the domain the role actually contains. For a work sample this is traceable rather than statistical: a role blueprint names the competencies, each competency is served by a task, each task is scored against named rubric criteria, and anyone can follow that chain backwards from a score to the part of the job it came from. It requires no outcome data, which is why it is available on day one, and it is the evidence SkillCort describes.

Criterion-related evidence is the kind people usually mean when they reach for the word: relating assessment scores to some later outcome, such as a performance rating. It requires outcome data most organisations do not have when they start, and observed correlations understate the underlying relationship — the candidates you hire are a restricted slice of those who applied, and the outcome measure has measurement error of its own. The Uniform Guidelines are explicit that any statistical adjustment for these effects should be described and explained, and that uncorrected coefficients should be shown alongside. Where SkillCort records outcomes, it shows the observed, uncorrected correlation and labels it preliminary; it does not claim that an assessment forecasts later job performance.

'Enough data' is the part everyone skips

Every coefficient on this page is an estimate computed from a sample, and below some sample size it is noise wearing a decimal point. That is not a caveat at the bottom of the page — it is the difference between a number you can act on and a number that will reverse itself with the next ten candidates. Internal consistency needs a reasonable count of candidates who completed the same items. Test–retest needs people who sat it twice, which most hiring processes never produce.

Inter-rater agreement has the strictest and most frequently ignored requirement: it needs at least two evaluators who scored the same candidates on the same criterion. One evaluator produces no inter-rater number at all, however many candidates they have graded, because there is no second opinion to disagree with. A panel where each evaluator handled a separate group of candidates has the same problem in a less obvious form — plenty of scores, no overlap, nothing to correlate.

The honest handling of that situation is to report nothing. A tool that always shows a coefficient is telling you less than one that sometimes shows 'not enough data', because the first has quietly decided that a number computed on four people deserves the same font as one computed on four hundred. It is also worth remembering that the interpretive bands attached to these statistics — the familiar poor/moderate/good/excellent labels, such as those in Koo and Li's guidance for intraclass correlations — are reporting conventions rather than thresholds of truth, and the bands for one coefficient do not transfer to another.

What to ask, and what an honest answer sounds like

You do not need to compute any of this yourself to interrogate it. Four questions separate a vendor with evidence from a vendor with adjectives, and the useful thing about them is that the good answers are specific and the bad ones are reassuring.

The answer you should want to hear at least once is 'we cannot compute that yet, and here is what would let us'. A supplier who can tell you which coefficient is missing, and why, understands their own measurement. A supplier whose every number is available and every number is good is describing a brochure, not an assessment.

• Which coefficient is that, computed on how many people, and on whom — your sample or ours?
• What kind of evidence supports validity for this use: content-related, criterion-related, or an assertion?
• What does the interface show when there is not enough data to compute a figure?
• If two of our evaluators score the same candidate, can you show me how far apart they were?

Sources & further reading

• Standards for Educational and Psychological Testing (AERA, APA, NCME) — Joint Committee
• Uniform Guidelines on Employee Selection Procedures, 29 CFR Part 1607 — eCFR
• Koo & Li (2016), A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research — Journal of Chiropractic Medicine