2026-09-03 · Gain axis · Refuted

Only after our own experiments excluded two explanations did the mechanism have room to change

"The model is too strong" and "the tasks are too easy" were two explanations proposed by humans. Two experiments excluded both, and only then was the reviewer mechanism forced into a redesign — and the direction was to change the referee, not the reviewer model.

hypothesis exclusionLiveCodeBenchself-correction

This page cites 2 verdicts: Refuted 2click one to jump to its claim card

RefutedA weaker model would reveal the gain (ceiling hypothesis H-A)
RefutedA harder task bank would reveal the gain (difficulty hypothesis H-B)

#The situation

Our mechanism lost under an equal budget

ON vs OFF5: 75.25% vs 77.23%, a difference of −1.98 pp, p = 0.7905.

#H-A: the ceiling hypothesis (proposed by a human on 2026-09-01)

qwen3.6-35b is too strong. So every model gets it right, and the mechanism has no chance to make a difference. Switch to a weaker model and the gain will show.

#Experiment (R440N / E1)


Design	Single model gemma-4-12b, 179 tasks
Audit	Independent audit recomputed row by row, fully consistent with R516
Audit receipts	rows.jsonl 525 lines, sha `440d973c`; calls 1,986, sha `3d24b73d`

Arm	n	Passed	Failure rate
OFF	179	122	31.8% (Wilson [25.5, 39.0])
ON	167 (+12 void = 6.7%)	122	26.9%
OFF5	179	132	26.3%

Comparison	Difference	Statistics
ON vs OFF5 (equal budget)	−0.60 pp	b=11 c=12, p = 1.0000
ON vs OFF	+5.99 pp	b=17 c=7, p = 0.0639
OFF5 vs OFF	+5.59 pp	b=12 c=2, p = 0.0129
Reviewer grounded lift	+4.19 pp	95% [0.00, +8.58]

H-A excluded.

#H-B: the difficulty hypothesis (proposed by a human on 2026-09-01)

MBPP+ is too easy for modern models. Switch to date-stamped contest problems (LiveCodeBench) and the gain will show.

#Experiment (R440T / E3)


Design	91 medium + hard tasks from LCB v1, gemma-only, seed `g-r442-lcb`

Arm	n	Passed	Failure rate	Delivered	Leaked	Calls per task
OFF	91	47	48.4% [38.4, 58.5]	91	44	1.00
ON	87 (4 void = 4.4%)	58	33.3%	69 (79.3% coverage)	11	5.09
OFF5	91	56	38.5%	91	35	5.00

Comparison	Difference	Statistics
ON vs OFF5 (equal budget)	+5.75 pp	b=15 c=10, p = 0.4244
ON vs OFF	+16.09 pp	b=18 c=4, p = 0.0043
OFF5 vs OFF	+9.89 pp	b=19 c=10, p = 0.1360

H-B excluded.

#Another finding from this experiment: the reviewer layer is worse on hard tasks

	MBPP+	LCB hard tasks
Reviewers' raw FAIL claims	Few (almost always PASS)	234 / 261 = 89.7% (almost always FAIL)
Actual error rate of first drafts	0.305	0.460
Raw layer − zero-intelligence baseline	−2.99 pp	+0.77 pp

#Summary of three clean runs

Both possible explanations were excluded, so what remains to explain it is the mechanism itself. Only then was there a reason to change the design — otherwise we would have kept fixing something that "was really just a too-strong model."

#What it became

2026-09-03, R440P

	Old design	New design
Referee	Three AI reviewers	The customer's own executable acceptance tests
Question	"Does this look right?"	"Does it pass your tests?"
Nature of the referee	A model	Data
Call cost	3 review calls + 1 revision call	1.5 calls (early stopping)

#Honest boundary

#Next

How the gate replaced the committee: conformance-gate
How difficulty changes the gate's value: harder-tasks-more-worth
The full numbers of the committee's loss: committee-loses-to-majority

Our AI review committee lost to "generate five times, take the majority"
Same 5 calls, same set of tasks: a committee of three AI reviewers plus one revision lost to "generate five times and take a majority vote." The point estimate of the difference was −1.98 percentage points — and after we fixed an unrelated bug, it became +2.97.
The reviewers track task difficulty, not whether the answer is right
The same reviewer layer judges almost everything as pass on MBPP+ (0.305 error rate) and almost everything as fail on LiveCodeBench hard tasks (0.46 error rate). It follows a global property, not this particular answer.
The birth of the judging gate: execution instead of opinion, receipts instead of votes
We deleted the AI review committee and replaced it with "run the customer's acceptance tests, deliver the first candidate that passes, refuse delivery if none passes." On three task banks it delivered 4.58 to 19.17 percentage points more, at only 1.5 calls per task.
The harder the task, the more the acceptance gate is worth
The early-stop gate delivers 7.8 percentage points more on MBPP+, and 20.9 percentage points more on hard LiveCodeBench problems. And the "leak rate" drops from 44 to 11 — refusing delivery is worth the most on hard tasks.