Psychometrics moves into LLM evals: flag the bad items, cut 80% of the questions, and detect distillation kinship
Stop Evaluating Models Like It's the 50s - Alejandro Vidal, Mindmakers · Alejandro Vidal
24 min total·Actually worth watching closely: ~17 min·3 must-watch clips
- 0:02 – 2:50Watch
Why counting right answers is wrong
Opens with the accuracy of a random selection of models on a real benchmark, using real data from epoch.ai, and points out that this way of scoring assumes every question is equally important. Then introduces item response theory: treat each item as an individual variable with a curve mapping model ability to the probability of a right answer, where difficulty is the point at which that curve crosses 50%.
Treat a question as a measuring instrument with its own difficulty and character, not as an equally weighted scoring box.
That S-curve and the 50% crossing marked on it are the foundation for everything that follows, and the speaker keeps pointing at the chart saying "here" — on audio alone you lose track of where on the curve the difficulty parameter actually sits.▶ Jump to 0:02Speaker · Alejandro Vidal - 2:50 – 7:08Skim
Discrimination: good items, dead items, and negative ones
Beyond difficulty comes a second parameter — the slope of the curve, i.e. how sharply an item separates strong models from weak ones. Steep items carry a lot of information, flat ones have almost nothing to do with intelligence, and a third kind has a negative slope, where better models are the ones getting it wrong. Wraps up on a 337-item example.
Items get evaluated too: an item with a negative slope isn't a hard item, it's a bad item.
This runs mostly on a row of curves contrasting the steep, flat and inverted shapes — the difference is obvious at a glance, so you can scan the charts and listen for the conclusion rather than following sentence by sentence.▶ Jump to 2:50Speaker · Alejandro Vidal - 7:08 – 10:20Watch
Ability comes with an interval, and bad items surface on their own
Presents each model's ability estimate together with its likelihood interval, so a score finally carries confidence and its own frame of reference. Then demonstrates using significantly-below-zero discrimination to circle suspect items automatically, with another LLM doing a cheap recheck.
Close scores don't mean close ability — Gemini and Claude sit almost one standard deviation apart, because Gemini is answering the harder questions.
What matters here is the ability plot with its error bars: whether two models' intervals overlap is the test of whether the gap is real, and that visual doesn't survive being put into words.▶ Jump to 7:08Speaker · Alejandro Vidal - 10:20 – 14:20Watch
484 items down to 97, ranking unchanged
Picking items from the highest discrimination down compresses a 484-item benchmark to about 97, still holding a 0.99 correlation with the original ranking, and saving a lot of time and tokens. A random-selection run is shown as the control, and GPQA gets named as the counter-example of a well-designed benchmark, where every item is highly discriminative and the information doesn't overlap.
"More questions means more accuracy" is an illusion; items with overlapping curves are just repeating the same information, and the optimal size can be computed.
The ranking curves for the selected subset and the random subset sit side by side, with both the gap and the overlap visible on the chart — the single most practical comparison in the talk.▶ Jump to 10:20Speaker · Alejandro Vidal - 14:20 – 17:30Watch
Residuals arrive: debugging production with answer consistency
With a curve per item, you can compute the difference between each model's expected and actual performance on each question. An item the model should get right 86% of the time and gets wrong is flagged immediately. And if one model is inconsistent across a whole region, the problem usually isn't that the model is dumb — it's the deployment.
Evaluation residuals double as an ops alarm: broad anomalies usually mean the inference platform isn't really running the model, or the quantization is wrong.
A consistency matrix with one colored cell per item, arranged by difficulty — where the red clusters is the conclusion itself; away from that chart, "inconsistent" is just an abstraction.▶ Jump to 14:20Speaker · Alejandro Vidal - 17:30 – 21:15Watch
Fingerprint sets against leaks, and item bias analysis
Proposes an adaptive scheme: everyone shares one anchor set so results stay comparable, while each organization also gets its own set of extremely complicated items. Months later, if one organization's new model is implausibly good on just its own set, you can infer the benchmark went into training. Then borrows differential item functioning from psychology, fitting separate curves for open-weight and closed-weight models to find items that systematically favor one side.
You don't need to catch leakage in the act — the residual anomaly on the items only that party has seen lets you infer it.
The bar chart contrasting the two groups' average residuals is wildly lopsided, and the two non-overlapping fitted curves on a biased item are the same kind of thing: obvious on sight, invisible by ear.▶ Jump to 17:30Speaker · Alejandro Vidal - 21:15 – 23:34Listen
DNA fingerprints for distillation, and what comes next
Uses a correlation matrix over residual vectors plus a simple projection to show model kinship: models from the same lab cluster, versions of the same model sit adjacent, and a DeepSeek distillation correlating about 0.38 with Qwen exposes the lineage outright. Closes on future directions — multidimensional and hierarchical models to reveal different skill levels for different kinds of task, and merging different benchmarks to improve the estimates from each.
Answering patterns are a model's fingerprint: you can identify unauthorized distillation and model lineage without the other party's cooperation.
The fingerprint scatter plot already appeared in the previous section; here the speaker mostly walks through specific model examples and looks ahead to further research, so listening is enough — there's no new key visual at the end.▶ Jump to 21:15Speaker · Alejandro Vidal