2026-08-30 · Gain axis · Refuted

Our AI review committee lost to "generate five times, take the majority"

Same 5 calls, same set of tasks: a committee of three AI reviewers plus one revision lost to "generate five times and take a majority vote." The point estimate of the difference was −1.98 percentage points — and after we fixed an unrelated bug, it became +2.97.

equal budgetreviewerrefutedMcNemar

This page cites 3 verdicts: Held 1 · Overstated 1 · No discernible difference 1click one to jump to its claim card

No discernible differenceThe three arms differ discernibly in delivery quality
HeldWhy it can't win: reviewer votes are near-constant, and revision makes no net correction
OverstatedThe decisive run is the first clean complete dataset

On 2026-08-30 we ran the most critical experiment of the whole project. The result: the core mechanism we designed lost.

#Background: the committee design

Our architecture originally looked like this:

1. Pick a worker by reputation routing
2. Generate an answer
3. Three peer reviewers (AI) each give a judgment
4. Deterministic sampling audit (sandbox rerun)
5. If it fails, one round of revision

The problem: the referee of this whole thing is an AI. And our control was the most brainless kind: generate the same task five times and take a majority vote over the five.

Under the same call budget (5 calls each), which one has the higher delivery rate?

#Design


Task bank	MBPP+ (EvalPlus v0.2.0)
Tasks	179 tasks, 6 blocks
Three arms	OFF (single sample) / ON (full mechanism) / OFF5 (five tries, majority)
Criteria frozen	2026-08-24 and 2026-08-28
Statistics	Paired McNemar

#Results

Criterion	Verdict
① The gauge has signal	✓ 26.44% (Wilson [20.4, 33.4])
② The three arms differ discernibly	Not measured
③ ON beats OFF5	✓ The answer is no

Paired results (n = 101, no arm void):

Measure	ON	OFF5	Diff	b	c	p
Original	75.25%	77.23%	−1.98 pp	6	8	0.7905
Typing-whitelist bug fixed	84.16%	81.19%	+2.97 pp	5	2	0.4531

#Why it lost: mechanism analysis

Metric	Value
Reviewer vote accuracy	0.7552
Zero-intelligence baseline "judge everything as pass"	0.7522
Gap	+0.29 percentage points
Single-round revision "had no chance to fix"	93 / 113
Hidden pass rate after blindly adopting the revision	82.3% → 78.8% (−3.5 pp)
Real improvement from revision	1 / 113

#How the "review committee" design got deleted

2026-09-03, R440P. The project document puts it this way:

Execution replaces opinion; receipts replace votes.

With the review committee and single-round revision deleted: each task saves 3 review calls + 1 revision call. What remains is the customer's own executable acceptance gate.

All the numbers for the gate line

#The data-quality claim withdrawn at the same time

"The decisive run is the first clean complete dataset" → changed to "no known bug contamination, paired subset n = 101."

Arm	infra_void
ON	66 / 179 = 36.9%
OFF5	30 / 179
OFF	5 / 179

In summary.json, equal_budget_comparison_valid = false, and R437 itself logged a VOID-GATE-WARNING line. The n=101 conclusion of the paired analysis is not directly affected by voids (it only uses tasks where no arm is void), but the word "clean" cannot be used.

#An earlier signal we ignored

The G series (exploratory) before 2026-08-30 already had the answer:

Comparison	n	ON	OFF5	p
Equal budget (371 tasks)	347	82.42%	83.57%	0.5572
Unequal budget: OFF5 vs OFF	349	83.09%	78.51%	0.0090

The second row is the key

And "ON vs OFF5 under an equal budget" shows no difference. So the relationship among the three is:

OFF5 > OFF  (works, p=0.009)
ON ≈ OFF5   (tied under equal budget, p=0.557)
ON ≈ OFF    (diff p=0.0576)

Our mechanism was no better than "generate a few more times." And it costs more than OFF5 (4 review + revision calls vs 4 more generations).

#Honest boundary

#Next

Two hypotheses proposed and then excluded: hypotheses-excluded
Why the reviewer layer is nearly constant: reviewer-tracks-difficulty
The gate, after the reviewers were deleted: conformance-gate

The reviewers track task difficulty, not whether the answer is right
The same reviewer layer judges almost everything as pass on MBPP+ (0.305 error rate) and almost everything as fail on LiveCodeBench hard tasks (0.46 error rate). It follows a global property, not this particular answer.
Only after our own experiments excluded two explanations did the mechanism have room to change
"The model is too strong" and "the tasks are too easy" were two explanations proposed by humans. Two experiments excluded both, and only then was the reviewer mechanism forced into a redesign — and the direction was to change the referee, not the reviewer model.
The birth of the judging gate: execution instead of opinion, receipts instead of votes
We deleted the AI review committee and replaced it with "run the customer's acceptance tests, deliver the first candidate that passes, refuse delivery if none passes." On three task banks it delivered 4.58 to 19.17 percentage points more, at only 1.5 calls per task.
The "equal budget" trap: our ranking flipped completely
The criterion was written as defected <= BUDGET, an upper bound that always holds, so it could not catch "this arm was never bound at all." As a result, whitewash misbehaved 2.5 times as much and we read it as "the strategy is cheaper." After switching to == and verifying, the ranking flipped.