Why test analytical skills instead of inferring them

“Analytical” is a compliment until you define it

The word does a great deal of work in job adverts and almost none in a hiring decision. Two interviewers can both write “strong analytical thinker” on the same candidate and mean different things: one saw somebody comfortable with numbers, the other heard a sharp question about the brief. Neither has recorded anything a third person can check next week. Before you can measure the thing you have to say what it is in terms of behaviour that leaves a trace.

A workable definition is operational rather than flattering. Analytical skill is the ability to take a body of information you did not choose — a table, a report, a thread of conflicting figures — and produce a conclusion someone else can act on and challenge. That names an input, a process, and an output, and all three can be observed. What lands on the page is the evidence; the impression in the room is not.

Notice what the definition leaves out. It is not intelligence, not “being good at maths”, and not enjoying spreadsheets. Those things may travel alongside the skill, but they are not the skill, and a process that measures them will feel rigorous while measuring the wrong construct — which is the failure mode that costs the most, because it produces confident decisions rather than uncertain ones.

What a degree and a job title actually record

Credentials are cheap to read, and that is the whole of their appeal. A degree records that a person completed a programme, usually some years ago, assessed against goals a university set for its own purposes. A previous job title records that an employer with its own internal conventions applied a label to a role whose contents you cannot see. Both are real facts. Neither is a measurement of the work in front of you.

The proxy also degrades unevenly, which is the part that quietly costs you candidates. Two people carrying the title “Analyst II” may have spent three years refreshing a weekly report and three years rebuilding a forecasting model. And screening on credentials filters partly on access to education: career changers, people who learned on the job, and candidates educated in systems your screeners do not recognise all lose signal they may in fact possess.

The frameworks used to describe work make the same separation. O*NET treats “Analyzing Data or Information” as a work activity in its own right — something a job requires and a person performs — rather than an attribute you read off a CV. Large adult-skills programmes such as PIAAC likewise measure literacy, numeracy and problem solving directly across the adult population instead of inferring them from qualifications held. If statistical agencies find it necessary to measure the skill rather than assume it from a certificate, a hiring team reasonably can too.

The four moves that make up analytical work

On the job the skill decomposes into a small number of moves that leave visible traces. Naming them is what converts “strong analytical thinker” into something two evaluators can independently agree on, and it is the step most processes skip.

The four are not evenly distributed across candidates, and averaging them into one impression destroys the most useful information you have. Plenty of people read a table carefully, describe it faithfully, and stop — accurate and inert. Others arrive at a confident story the data does not support: fast, articulate, wrong, and considerably more dangerous in a role where somebody acts on the recommendation. A single score of “analytical: 4 out of 5” cannot tell those two apart. Four separate scores can, and they also point at what to do next, since weak reading is often trainable in a few weeks while a habit of overclaiming on thin evidence is a judgement problem.

Splitting the construct also tells you what to hand the candidate. Each move needs material to act on: for someone to notice a missing baseline there must be a missing baseline; for someone to choose between readings there must be more than one plausible reading. This is the practical reason abstract puzzles rarely reach the interesting part of the skill — they usually have one answer, nothing missing, and nothing to argue about.

• Read accurately: take the right figure from the right column, and read what the header actually says rather than what you expected it to say.
• Notice what is missing: the absent baseline, the definition that changed in March, the month with no data, the total that does not reconcile.
• Choose an interpretation and defend it: commit to a reading of the evidence and say why the plausible alternatives are weaker.
• State the conclusion with its limits: say what follows, what does not follow, and what you would need in order to be sure.

Job-shaped material beats the clever puzzle

Abstract reasoning items — sequences, matrices, odd-one-out — are attractive because they are quick to write and trivial to mark. The trouble is what they sample. An item with one hidden rule measures pattern recognition and, increasingly, familiarity with the item format, and candidates who have practised that format arrive with an advantage that has nothing to do with the job. You end up ranking people partly on preparation for a genre of test.

The alternative is not a harder puzzle. It is material that resembles the work: a real-shaped table with a footnote that changes what a column means, two sources that disagree, a stakeholder who has already decided what they want the answer to be. Somebody who has never seen your product can still do that work if the context is readable in two or three minutes. What varies between candidates is not domain knowledge but what they do with the material.

Job-shaped content is also easier to justify when someone asks why you used it. The US Uniform Guidelines on Employee Selection Procedures describe content validity in these terms — a selection procedure whose content is a representative sample of the work — and the AERA, APA and NCME Standards set out the broader expectation that a claim about what a test measures should be supported by evidence. Obligations differ by jurisdiction, and none of this is legal advice, but the direction is consistent: the closer the material sits to the job, the shorter the argument you have to make for it.

Scoring it so two people reach the same number

A conclusion on its own is a poor score, because a candidate can arrive at a defensible answer by a route you would not want repeated, and a thoughtful candidate can be marked down for landing somewhere the key did not anticipate. Score each of the four moves separately, against written level descriptions with a concrete anchor for each level — what a 2 looks like, what a 4 looks like, in the words of this task rather than in the abstract.

Then calibrate, because a rubric written by one person is a rubric understood by one person. Have every evaluator score the same two or three responses before the real ones, compare where the numbers diverge, and rewrite the level that caused the disagreement. Where enough scored responses exist, agreement between raters can be computed and reported rather than assumed; where they do not, the honest position is that you have a shared rubric and not yet a measured level of agreement.

This is the chain SkillCort is built to keep visible: a role blueprint names the competency, the task samples work that belongs to it, the rubric defines what each level looks like, and the candidate's actual output sits underneath the score. Content is expert-reviewed and the limits are stated rather than dressed up. AI drafts the summary of a long response so a panel can read forty of them; it never writes a status, and no automated step decides an outcome.

What the measurement does not tell you — and saying so

A thirty-minute analytical exercise tells you how this person handled this material under these conditions. It does not tell you whether they will still reason well in a domain whose vocabulary they do not yet have, whether they will raise the problem with a stakeholder who does not want to hear it, or how they will grow over a year. Those are separate questions with their own evidence, and a result that is quietly asked to answer them will be over-read.

Fairness lives mostly in the conditions. Make the time limit as long as the work needs rather than as short as you can bear; publish it in advance so nobody is surprised; provide extended time where a candidate needs an accommodation; and make sure a table is legible to a screen reader before you decide that reading tables is the skill. Where integrity signals are recorded, keep them proportional to what is at stake and route them to a person, never to an automatic verdict.

Stating the limits is also what makes a result usable later. Three months after the decision, a file that says what was measured, on what material, by whom, and what was explicitly not measured is still evidence. An impression recorded as “strong analytical thinker” has, by then, become a memory of a good conversation.

Sources & further reading

• O*NET OnLine, Work Activities — Analyzing Data or Information (U.S. Department of Labor)
• Uniform Guidelines on Employee Selection Procedures, 29 CFR Part 1607 — eCFR
• Standards for Educational and Psychological Testing (AERA, APA, NCME) — Joint Committee
• Programme for the International Assessment of Adult Competencies (PIAAC) — U.S. National Center for Education Statistics