Can the benchmark prove its claim?

Can the benchmark prove its own claim?

Somebody hands you two benchmark scores and a conclusion. This works out whether the benchmark was ever capable of telling those two scores apart. Most of the time it was not, and the honest answer is "this cannot be resolved" rather than "there is no difference". Start with a preset, then read How to read this underneath.

Baseline, % The score you are comparing against. Usually the unquantized model.

Comparison, % The new one. Order does not matter; the test is two-sided.

Items How many questions or problems the benchmark contains. Not the number of models, and not the number of runs. HumanEval is 164. MMLU-Pro is 12,032.

They disagree on 15% An assumption, not data: on what share of items do the two models give different answers? Nobody publishes this. It decides the honest verdict.

countsThe scores as whole items. This is the first thing to look at.: ·
differenceHow many items separate them. Percentages hide how small this is.: ·
95% interval, baselineThe range the true score plausibly sits in.: ·
95% interval, comparisonSame, for the other model.: ·
items the unpaired test needsHow large the benchmark would have to be to resolve a gap this size.: ·
unpaired p, a ceilingSmall means probably real. Large does NOT mean "no difference".: ·
paired p, if they disagree that oftenThe test you should be running, under the assumption in the slider.: ·

How to read this

Step 1. Turn the percentage back into items

A benchmark score is a count divided by the number of questions. "57.0%" sounds continuous and precise. On a 164-problem benchmark it is 93 problems, and the next achievable score up is 94, which is 57.32%. There is nothing in between.

So a "0.7 point drop" on that benchmark is one problem. One problem is not a finding. Do this conversion before you form any opinion, because the percentage is designed to make small differences look substantial.

Step 2. What the p-value actually says

The p-value answers one narrow question: if the two models were truly identical, how often would chance alone produce a gap at least this large?

p = 0.89 means: about 89% of the time. That is completely unremarkable, so the data gives you no reason to believe the models differ.

The trap is reading a large p as proof they are the same. It is not. It means the instrument could not tell. A bathroom scale that reads to the nearest kilogram cannot detect a 200 gram change, and its silence is not evidence that you weigh the same.

Small p means the difference is probably real. Large p means you learned nothing. Those are not opposites.

Step 3. Why overlapping error bars prove nothing

The two intervals above are where each true score plausibly lives. People eyeball whether they overlap and call it a day. That rule is asymmetric, and only one half of it works.

If the intervals do not overlap, the difference is significant. That direction is sound. If they do overlap, you have learned nothing at all: two intervals can overlap by nearly a third of their width and still differ at p = 0.05. The overlap rule is equivalent to testing at about p = 0.006, roughly a tenth as permissive as the test you meant to run.

Read the p-value. Use the picture to communicate, never to decide.

Step 4. The unpaired test is probably the wrong test

The p-value above assumes the two models were measured on two independent samples. They were not. They answered the same questions. That is paired data, and the correct test is McNemar's.

McNemar ignores every item both models got right and every item both got wrong. It looks only at the ones they disagreed on: b, where the baseline won, and c, where the comparison won. The statistic is (b - c) / sqrt(b + c).

Here is the problem. A published table of percentages gives you b - c, because that is the visible gap. It never gives you b + c. And b + c is what decides the answer.

That is what the slider is for. Assume a disagreement rate and watch the verdict move. A model and its own quantization agree on most items, so realistic rates are low, and at low rates gaps that looked like noise stop looking like noise. Set the slider near 50% and the paired result collapses back onto the unpaired one, because that is the case where the two models are effectively unrelated.

Practical consequence: the unpaired p is a ceiling on the evidence. It is safe to say "this gap is small". It is not safe to say "this gap is not significant".

Step 5. What to do with the answer

If the gap is a handful of items, stop. It is not a result, however confidently it was printed, and no amount of restating it will make it one.

If you need to know, run your own evaluation on your own task, and check the size first. The row above tells you how many items it would take. A 200-case internal suite cannot see a one-point change. It can see a ten-point one.

And when you publish an evaluation yourself, publish the discordant counts. They are the one number that makes a comparison checkable, they cost nothing to record, and almost nobody includes them.

The trap that caused this calculator

An earlier version of this project quoted HumanEval as falling from 57.0% to 56.3% under INT4. Both numbers are real. Both sit in the correct row of the source table, for the correct model. Both are the wrong column.

Llama-3.1-70B-Instruct	MMLU-Pro	Arena-Hard	HumanEval
BF16	48.1	57.0	79.7
W4A16-INT	47.2	56.3	80.5

Arena-Hard sits immediately left of HumanEval, so reading one column short returns 57.0 and 56.3 intact. The real HumanEval figures are 79.7 and 80.5, which reverses the finding: the quantized model scored slightly higher.

This is worse than an invented number. An invented number looks unfamiliar. A number from the neighbouring column is a genuine measurement of a genuine model, it survives every sanity check you would think to run, and it still gives you a backwards conclusion. No calculator can catch it. Read the cell against its own column header, from the paper rather than from a summary of the paper.