2026-09-03 · Gain axis · Held

The reviewers track task difficulty, not whether the answer is right

The same reviewer layer judges almost everything as pass on MBPP+ (0.305 error rate) and almost everything as fail on LiveCodeBench hard tasks (0.46 error rate). It follows a global property, not this particular answer.

reviewermechanism analysiscross-bank

#Two task banks, the same reviewers

	MBPP+	LCB v1 (medium + hard)
Tasks	179	91
OFF-arm failure rate	31.8%	48.4%
Actual error rate of first drafts	0.305	0.460
Reviewers' raw FAIL claims	Few (almost always PASS)	234 / 261 = 89.7% (almost always FAIL)
Reviewer layer − zero-intelligence baseline	−2.99 pp	+0.77 pp

#A surprise during the audit

While checking the FAIL claims, we found a large number of "discarded FAIL claims":


Raw FAIL-claim rate in round346	1.2% – 25.8%
Of those, discarded as `outside_input_contract`	73.6% – 100%

The root cause was an indentation bug

Recomputing the 99 discarded cases:

What it actually was	Count	Share
Real counterexamples swallowed by the bug	38	38.4%
Genuinely invalid	20	20.2%
Genuinely out of scope	41	41.4%

Nearly four in ten of the "errors the reviewers caught" were real; an indentation bug just swallowed them. This one is kept in methods.json as a classic measurement trap.

Pooling the three runs, on the 49 tasks that really do fail:


Total tasks	275
True negatives (the TN = 0 part)	0
Votes cast by the 3 reviewer agents	147
Of which said "pass"	145 (98.6%)
Tasks that ever got a fail vote	2 / 49 (and both times outvoted by the majority)

#Consistent results across three runs

The R438 analysis and two follow-up runs:

run	Reviewer accuracy	Zero-intelligence baseline	Gap
R438 (179 tasks)	0.7552	0.7522	+0.29 pp
R440N (179 tasks, gemma-only)	+4.19 pp (grounded)	—	95% [0.00, +8.58]
R440T (91 tasks, LCB hard)	—	—	+0.77 pp, 95% [−19.16, +11.11]

#How this conclusion led to a design change

From "the reviewers aren't good enough" to "the reviewers aren't the referee"

But the actual shape of this conclusion is different: the reviewers aren't too weak; the question they are asked is different from the question they are able to answer. They can judge "roughly how hard is this task," not "is this answer right."

So the right change is to change the type of referee, not the strength of the referee:

Old design	New design
Three AI reviewers + sampling audit	The customer's own executable acceptance tests
"Does this look right?"	"Does it pass your tests?"
The referee is a model	The referee is data

How the gate replaced it

#A counter-test

R440P ran a very clean check: using zero-API offline replay, it reran all 895 candidates (179 tasks × 5 each).

	`g_r441`	`g_r356`
Candidates	895	735
"Passed visible but failed hidden"	0	0
Early-stop gate pass rate	78.21%	82.99%
Calls per task	1.34	1.13

#Honest boundary

#Next

The full numbers of the committee losing to majority vote: committee-loses-to-majority
The full rules of the execution referee: conformance-gate
Why reviewer quality is the only tight constraint on the defense axis: reviewer-accuracy-binds

Our AI review committee lost to "generate five times, take the majority"
Same 5 calls, same set of tasks: a committee of three AI reviewers plus one revision lost to "generate five times and take a majority vote." The point estimate of the difference was −1.98 percentage points — and after we fixed an unrelated bug, it became +2.97.
The only truly tight constraint in the whole defense system: reviewer accuracy
We swept the stake, probation length, ecosystem size, attacker share, and the correlation of reviewer errors — all of them are slack. The only cell where attacking starts to pay is when reviewer accuracy drops to 0.
Detection probability is one product, not three dimensions
We originally claimed that "raising the audit rate cannot rescue cases with a blind spot." Measured: at blind spot 0.5, raising the audit rate from 0.1 to 1.0 cut the attacker's total hits by 82%, p = 5.5e-6. The three factors are terms in the same product.
The harder the task, the more the acceptance gate is worth
The early-stop gate delivers 7.8 percentage points more on MBPP+, and 20.9 percentage points more on hard LiveCodeBench problems. And the "leak rate" drops from 44 to 11 — refusing delivery is worth the most on hard tasks.

Rule	Pass rate	Calls
Execution gate (deliver the first that passes)	78.21%	1.34
Majority vote over five	73.74%	5.00