The cheaper judge missed my own bar: 6.6% against 5%
Three judges grade my benchmark. Last month they cost $146 of my $195 model API bill. Newer versions of the same models list at lower prices, so I tried to swap them.
A judge gives each free-text answer a score from 0 to 10, and the three scores are averaged. Change a judge and every score I ever published means something slightly different. So I wrote the pass marks before the first call.
A swap goes ahead only if, against the old judges:
→ the scores correlate at 0.90 or better → the average shifts by at most 0.03 (on a 0 to 1 scale) → at most 5% of tasks cross the pass line of 0.6 → cost per task falls by at least 30%
Round 1. I replaced all three judges, in four combinations, from flagship prices down to the cheapest tier. 101 answers, each graded again by every candidate.
| Combination | Correlation | Average shift | Tasks crossing the line | Cost per task |
|---|---|---|---|---|
| Newer, flagship price | 0.73 | −0.11 | 16.5% | −16% |
| One flagship swapped for mid-tier | 0.75 | −0.12 | 18.7% | −49% |
| Two cheaper | 0.79 | −0.11 | 15.8% | −64% |
| All three cheapest tier | 0.84 | −0.08 | 13.9% | −86% |
None passed. Three things I didn't expect:
→ The old judges don't fully agree with themselves. Re-running them on the same answers at temperature 0 gives 0.92 for the average, and one of the three agrees with its own first run at only 0.54. That is the realistic ceiling for any replacement → Newer judges are harsher: all four combinations score about a tenth lower on the 0 to 1 scale. A swap would lower every row in the table → List price lied. Per call, one new judge cost 76% less than its predecessor. Another cost 33% more even though its list price is 20% lower. A third cost 75% more. What decides it is how many tokens the judge spends thinking
Round 2. One slot was close and cheap: the first judge, newer version. I wrote round 2 down before running it: 27 runs I hadn't looked at, 960 graded answers, 23 model configurations.
| Rule | Limit | Result |
|---|---|---|
| Correlation with the old average | ≥ 0.90 | 0.976 |
| Average shift | ≤ 0.03 | −0.015 |
| Rank of the 23 configurations | ≥ 0.95 | 0.983 |
| Biggest shift of one configuration | ≤ 0.05 | 0.044 |
| Cost per task | −30% or better | −36% |
| Tasks crossing the pass line | ≤ 5% | 6.6% (63 of 960) |
The misses go one way: 46 tasks fell below the line and 17 rose. Across the 960 answers the pass rate moves from 66.2% to 63.1%.
My rule says not adopted, so the judges stay. I could argue that 6.6% is close enough, and I could also argue it after seeing any other number, which is why I wrote the line first.
The whole check cost about $5 in API calls.
What I'm changing: a spending cap on the benchmark's key, because October 8 and 9 took about $100 of that bill and nothing stopped them. If I move a judge later, it will be a labelled second version with the old rows kept as the first, not a silent swap.
The judges stay as they are.