By Daniel Whitmore, MSc in Industrial and Organizational Psychology
In short
An interview produces comparable evidence when the questions come from the role definition rather than from each candidate's CV, when every candidate is asked the same core set in the same order, and when each answer is scored against a rating scale written before anyone was interviewed. Interviewers take notes during the conversation and rate the candidate at the close of their own interview, independently, rather than ranking the field from memory at the end of the round. The interview's job is depth, reasoning and motivation in conversation; proving the person can do the work is the work sample's job, and mixing the two is what makes a hiring panel argue about impressions.
Decide what the interview is for before you design it
An interview is a conversation under time pressure with a person who wants the job. It is good at some things — how someone reasons out loud, what they chose to do when a project went wrong, what they actually want from the next two years, whether they can explain a technical decision to someone who does not share their background. It is poor at others, particularly at establishing whether the person can produce the work, because talking about a skill and having it are different capacities and the interview only ever sees the first.
The mistake is not using interviews. It is asking one interview to do every job at once: verify competence, assess motivation, sell the role, and rank the field. A conversation carrying four assignments does none of them cleanly, and the notes it produces are impressions that no two interviewers weigh the same way.
So write down, in one sentence, what this interview is deciding. "Whether this candidate's judgment holds up when the situation is ambiguous" is a decidable question. "Whether they are any good" is not. We have argued the division of labour between the two methods at length in the guide on work samples versus interviews; this post assumes it and deals with the interview half.
Build the questions from the role, not from the CV
Start from the role definition — the competencies the job actually requires, derived from an analysis of the work rather than from the last person who held the title. The US Office of Personnel Management's structured interview guide puts this first for a reason: its whole procedure begins with a job analysis identifying "the requirements of the job and the competencies necessary to perform them", and it asks that every question be reflective of competencies derived from that analysis, realistic of the responsibilities of the job, open-ended, and free of jargon.
Reading questions off the CV inverts this. It produces a different interview for each candidate, weighted towards whoever wrote the most interesting document, and it quietly rewards a familiar career shape. It also wastes the conversation on facts you could have confirmed in writing. Keep two or three CV-specific follow-ups if you want them, but keep them outside the scored core set and label them as such.
Two question formats do most of the work. Behavioural questions ask what the candidate actually did in a past situation — the premise being that past behaviour under similar circumstances is the better guide. Situational questions put a realistic dilemma from the role in front of them and ask what they would do. Pick one competency per question, write three or four questions per competency, and stop; a scored core set of six to eight questions is usually the limit of what an hour supports.
- One competency per question — a question that measures two things scores as neither.
- Open-ended, with planned probes for candidates who answer too briefly.
- Realistic: a situation that has genuinely occurred in this role in the last year.
- Free of internal jargon and acronyms an outsider could not decode.
- Answerable from a candidate's own experience, not from having worked at a competitor.
Ask every candidate the same core set
The comparison only exists if the stimulus is the same. OPM's contrast between structured and unstructured interviews is blunt about it: in a structured interview all candidates are asked the same questions in the same order, all candidates are evaluated using a common rating scale, and interviewers are in agreement on acceptable answers. In an unstructured one, none of those hold. Meta-analytic work on employment interviews going back to Huffcutt and Arthur's 1994 review has consistently found that how much structure an interview has is what separates the ones that carry useful signal from the ones that mostly record rapport.
In practice this means the same wording, the same order, the same planned probes, and the same amount of time. Variation should be a conscious accommodation — a candidate who needs longer, an accessibility adjustment — recorded as such, rather than an artefact of the interviewer being tired by the fourth conversation of the day.
It also means resisting the two habits that dissolve structure without anyone noticing: helping a candidate you like by rephrasing until the answer improves, and cutting a question short because you have "already got a sense" of the person. Both are generous impulses. Both delete the comparison you convened the panel to make.
Score against criteria as you go, not from memory afterwards
Write the rating scale before the first interview, and anchor it. A bare 1-to-5 is an opinion with a number attached; a scale becomes usable when each level carries a behavioural example or a representative answer that a rater can match against what they just heard. OPM's guide builds exactly this — a proficiency scale supplemented with behavioural examples for each question — and the reason it works is that it forces the panel to agree what "good" sounds like while nobody is in the room to be liked or disliked.
During the interview, take notes on what was said and done rather than on how it felt. Then rate at the close of that candidate's own interview: OPM's instruction is that immediately after the candidate leaves the room the interviewer reviews their notes and rates, with each rating supported by the actual examples given and an explanation of why they merit that score. That is the practical meaning of scoring during rather than after — the evidence is fixed while it is still accurate, not reconstructed at the end of a round when the last candidate is vivid and the first has faded.
In a panel, each member rates independently before anyone speaks. Only then do you compare notes, explore the basis for the discrepancies, and reach a consensus rating that gets recorded. The disagreements are the useful part: two people who scored the same answer 2 and 4 usually heard different things, and finding out which is how a panel calibrates itself. Averaging first, discussing never, throws that away.
Leave the proof of skill to the work sample
The reason this post keeps narrowing the interview's remit is that the other half of the evidence is better collected elsewhere. A work sample puts the candidate in front of a slice of the actual job — the ticket, the dataset, the objection, the broken spreadsheet — and produces an artefact you can read and score against a rubric. Contemporary meta-analytic reviews of selection methods, including Sackett and colleagues' 2022 re-estimation, continue to place structured interviews and job-relevant samples of work among the more informative methods available, and they are informative about different things.
So sequence them. The work sample establishes whether the person can do the core task; the interview then starts from that artefact — "walk me through why you refused the refund here" — instead of trying to establish competence from scratch in conversation. The interviewer's scarce hour goes to reasoning, judgment under follow-up, and motivation, which is what a conversation is genuinely good at.
This also removes the single most common failure of a hiring debrief, where one person's impression of a candidate's confidence is set against another's impression of their depth, with no shared object between them. When there is an artefact and a scored core set on the table, the argument becomes about evidence, and arguments about evidence resolve.
Keep the record the decision rests on
Whatever the panel concludes, the record should let someone who was not there reconstruct why. That means the questions asked, the notes taken, the score per criterion with the answer that justified it, the consensus rating, and who was in the room. OPM's guide asks for the same set — interview questions, scores and notes for each candidate, the guides and rating scales used, and the training the interviewers received — because a rating that cannot be traced to something the candidate said is not defensible six months later, whether the person asking is a hiring manager, an unsuccessful candidate, or a regulator.
Where an assistant summarises a transcript, keep the roles unambiguous. A model can compress a long answer, surface the passage that supports a criterion, or flag that a question went unanswered. It does not produce a rating and it does not produce a status; every advance-or-decline call belongs to a named person, and the record should show which person made it.
One legal note, stated plainly. Interviews are selection procedures, and they carry the same job-relatedness expectations as any other assessment — the framework in the US Uniform Guidelines and the EEOC's guidance on tests and selection procedures, with equivalents elsewhere. Obligations vary by jurisdiction, none of this is legal advice, and no claim of compliance is being made. What is true regardless is that a structured interview with a written scale and a traceable record is easier to explain than a conversation nobody wrote down.
- structured interviews
- interview design
- candidate evaluation
- work samples
- hiring panel
Key takeaways
- Write the questions from the role definition and the competencies it requires, not from each candidate's CV — otherwise you run a different interview per candidate.
- Ask the same core set, in the same order, with the same planned probes; keep CV-specific follow-ups outside the scored part.
- Anchor the rating scale with behavioural examples before the first interview, and rate at the close of each candidate's own interview from notes on what was said.
- In a panel, rate independently first, then explore the disagreements — that discussion is where calibration actually happens.
- Let a work sample prove the person can do the work, and spend the interview on reasoning, judgment and motivation instead.
Sources & further reading
- U.S. Office of Personnel Management (2008), Structured Interviews: A Practical Guide
- Huffcutt & Arthur (1994), Hunter and Hunter (1984) Revisited: Interview Validity for Entry-Level Jobs — Journal of Applied Psychology
- Sackett, Zhang, Berry & Lievens (2022), Revisiting Meta-Analytic Estimates of Validity in Personnel Selection — Journal of Applied Psychology
- U.S. EEOC — Employment Tests and Selection Procedures