Analytical skills interview questions — and the thing they cannot show you

What an interview question is actually sampling

Be clear about the measurement before choosing the questions. An interview does not put a table in front of anyone. It samples the candidate's account of their own thinking: what they can retrieve about work they did months ago, how they frame it for a stranger, and how fluently they narrate it under mild social pressure. That is genuinely useful — explaining reasoning to a colleague who was not there is a real part of most analytical jobs — but it is not the same as doing the analysis.

The main confound is fluency. A candidate who talks well can describe an analysis they largely did not do, and a candidate who did careful work can under-sell it in a room full of strangers. Both errors are systematic rather than random, which means asking more questions of the same kind does not average them out. It just gives you more of the same measurement.

None of this is an argument against interviews. It is an argument for structure. The U.S. Office of Personnel Management's guidance for federal hiring, and decades of meta-analytic work behind it, point the same way: asking every candidate the same questions in the same order and scoring them against defined criteria produces more dependable results than an open conversation. Conway, Jako and Goodman found interview reliability rising with structure, and Huffcutt and Arthur found the same pattern for validity. Structure is the cheapest improvement available here, and most teams have not made it.

Six questions, and what each one is for

A question earns its place by targeting a specific move, not by sounding searching. Each of these gives the candidate something concrete to hold, and each maps to one of the behaviours you would score: reading material accurately, noticing what is missing, committing to an interpretation, and stating limits. Pick three or four for a single interview — enough to probe properly, rather than eight skimmed.

Notice what is not on the list. “How do you approach a problem?” and “Would you say you are data-driven?” cannot be answered badly, so they separate nobody; they collect a self-description and hand you back the confidence the candidate arrived with. A question that produces signal always contains something specific — a number, a constraint, a conflict — that the candidate has to do something with in front of you.

• “Walk me through the last analysis you did that changed someone's mind. What was the number, and what did they do differently?” — targets consequence and specificity. Claims that were never really made collapse when asked what actually changed.
• “Tell me about a time your conclusion was only partly supported by the data. What did you do about the part it did not support?” — targets stating limits, the move most candidates skip entirely.
• “A weekly report you own shows sign-ups up thirty per cent and revenue flat. You have ten minutes before the meeting. What do you check first, and why that first?” — targets ordering hypotheses under a real constraint.
• “Describe a time the data you were handed turned out to be wrong. How did you find out?” — targets noticing what is missing; the answer reveals whether they check anything by default.
• “Someone asks you for a number you cannot get. What do you give them instead?” — targets defensible approximation and honest uncertainty, which is most of the job in a young data function.
• “What is a metric your last team tracked that you thought was misleading? Why did it survive?” — targets scepticism plus organisational realism, without inviting a rant.

The follow-up is the question

The first answer is usually prepared. The follow-up is not, and it is where the useful signal lives. Three probes do most of the work: “what would have changed your mind?”, “what did you check that turned out not to matter?”, and “what would you have needed to be sure?” Each asks for a detail that only exists if the work happened.

A rehearsed story survives a summary comfortably. It rarely survives being asked for the denominator. If a candidate says conversion improved, ask from what to what, over how long, measured how, compared with which baseline — not to catch them out, but because those are the numbers a person who did the analysis still remembers roughly and a person who watched it happen does not.

Probe everyone identically, and say at the start that you will. Announcing “I am going to ask a lot of follow-up questions, that is the format, it is not a sign anything is wrong” costs ten seconds and removes an anxiety confound you would otherwise be scoring. Write the probes into the interview guide so the third candidate of the day gets the same treatment as the first.

What a strong answer sounds like, and a weak one

Score the same four behaviours in every answer, and write down what each looks like before the first interview rather than after the third. The table below is a serviceable starting rubric — the wording matters less than the fact that both interviewers read the same wording.

The trap in this table is that the weak column can be delivered beautifully. Confidence, structure and a good speaking voice all sit on the left-hand side just as easily as the right. What separates the columns is not polish but whether figures, sources, alternatives and limits are present at all.

Score it, or you did not really run an interview

An interview without recorded scores is a conversation that generates a preference. Give each behaviour a small scale with anchors written in the language of your questions, and have every interviewer score independently before anyone speaks. Discussion comes after the numbers are in, because the first confident opinion voiced in a debrief anchors the room and the quieter dissent never arrives.

Calibrate the panel the same way you would calibrate rubric scoring anywhere else: have two or three interviewers score the same recorded or written answers, compare where they diverge, and rewrite the anchor that caused the disagreement. Where enough scored answers accumulate, agreement between interviewers can be computed and reported rather than assumed. Where they have not, say so — “we share a rubric” is an honest claim; “our interview is reliable” is not, until it has been measured.

Be careful about what you conclude from a good interview score. It is evidence that this candidate reasons well about their own past work in conversation. It is not, on its own, evidence about how they will handle unfamiliar material on a Tuesday, and a hiring file that treats it as though it were will over-read it later.

The stronger version is not a question at all

Everything above improves an interview. None of it removes the confound, because the candidate is still narrating. The alternative is short and unglamorous: give every candidate the same twenty to forty minutes, the same fictional company, a table with a footnote that changes what a column means, a second source that disagrees with the first, and a stakeholder who wants a recommendation. Now you are reading the analysis rather than a description of it.

Three things change immediately. Everyone works on identical material, so the comparison is real. The output is an artefact that outlives the meeting — a hiring manager who was not in the room can read the actual reasoning six weeks later. And the awkward cases resolve themselves: the quiet candidate who produced the sharpest memo, and the polished one whose numbers do not reconcile, are both visible in a way no conversation reliably makes them. OPM lists work samples and simulations as a distinct assessment method for essentially this reason.

The honest cost is that a work sample has to be designed and scored, whereas an interview needs a room and a diary. That is a real difference, and it is why the sequence matters more than the choice. Run the short work sample first, as the evidence. Then use the interview for what only an interview can do: probe the specific thing this candidate's submission left open. The interview gets better too, because you are both looking at the same page instead of at a memory.

Sources & further reading

• U.S. Office of Personnel Management, Assessment & Selection — Structured Interviews
• U.S. Office of Personnel Management, Assessment & Selection — Work Samples and Simulations
• Conway, Jako & Goodman (1995), A Meta-Analysis of Interrater and Internal Consistency Reliability of Selection Interviews — Journal of Applied Psychology
• Huffcutt & Arthur (1994), Hunter and Hunter (1984) Revisited: Interview Validity for Entry-Level Jobs — Journal of Applied Psychology