2026-09-05 · Gain axis · Held

The harder the task, the more the acceptance gate is worth

The early-stop gate delivers 7.8 percentage points more on MBPP+, and 20.9 percentage points more on hard LiveCodeBench problems. And the "leak rate" drops from 44 to 11 — refusing delivery is worth the most on hard tasks.

LiveCodeBenchgatedifficultygain

#A comparison along three dimensions

	MBPP+	LCB v1 (hard tasks)
Tasks	179	91
OFF (single draw) failure rate	31.8%	48.4%
Gate vs single draw	+7.82 pp	+20.88 pp
b / c (rescued by early stop / right on the single draw)	14 / 0	19 / 0
Candidate-pool ceiling	82.68%	75.82%
Single-draw success rate	78.2%	45.05%

#Leak rate: refusing delivery is worth the most on hard tasks

Arm	MBPP+ leaks	LCB hard-task leaks
OFF (single draw)	—	44 / 91
OFF5 (five-way vote)	—	35
CONFORM (gate)	—	11

	MBPP+	LCB hard tasks
Gate vs five-way vote	+4.47 pp	+4.40 pp
p	0.0963	0.424
Gate vs single draw	+7.82 pp	+20.88 pp
p	0.0001	< 0.0001

This is because on hard tasks the model is more likely to confidently hand over something bad. The gate's mechanism is "run your tests, deliver only on a pass" — on hard tasks it can still judge "this doesn't pass," while a voting mechanism can only be wrong when all five are wrong.

#The 41 tasks where delivery was refused

A post-hoc audit of every task where the gate refused delivery, checking whether all five drafts really were wrong:

Batch	Refused	All five wrong
r447	7	7 / 7
r449b	8	8 / 8
r449c	10	10 / 10
r461	15	15 / 15

#Convergence across three task banks

R445 converged across three task-bank sets (gate vs single draw):

Task bank	n	Δ	Statistics	Calls per task
MBPP+ (pooled, 371)	371	+4.58 pp	95% CI [+0.57, +8.04]	1.51
LCB v2 (hard tasks)	120	+19.17 pp	b=31 c=8, p = 0.0003	1.71
LCB v3	189	+7.94 pp	b=25 c=10, p = 0.0167	1.55

R445's 8 predictions made in advance: 7 hit, 1 missed (the miss was that the interval half-width was larger than predicted).

#The mechanism behind this finding

② The information value of refusal. On hard tasks the model's false delivery rate is higher (44 / 91 = 48.3%), so being "able to judge that it doesn't pass" is worth more.

#What this means for the claim "our mechanism works"

There is a tension here

The H-B hypothesis was "MBPP+ is too easy for modern models; switch to a hard task bank and the gain will show." The object of that hypothesis was the committee design.

Result: after switching to the hard task bank, the committee's gain still did not appear (ON vs OFF5 on hard LCB tasks, p = 0.4244).

#Honest boundaries

#Next

The gate's full rules: conformance-gate
The two hypotheses that were ruled out: hypotheses-excluded
Why the reviewer layer is worse on hard tasks: reviewer-tracks-difficulty

The birth of the judging gate: execution instead of opinion, receipts instead of votes
We deleted the AI review committee and replaced it with "run the customer's acceptance tests, deliver the first candidate that passes, refuse delivery if none passes." On three task banks it delivered 4.58 to 19.17 percentage points more, at only 1.5 calls per task.
Only after our own experiments excluded two explanations did the mechanism have room to change
"The model is too strong" and "the tasks are too easy" were two explanations proposed by humans. Two experiments excluded both, and only then was the reviewer mechanism forced into a redesign — and the direction was to change the referee, not the reviewer model.
The reviewers track task difficulty, not whether the answer is right
The same reviewer layer judges almost everything as pass on MBPP+ (0.305 error rate) and almost everything as fail on LiveCodeBench hard tasks (0.46 error rate). It follows a global property, not this particular answer.
"The hidden tests are lossless" is empty by construction
A criterion R440P set in advance was refuted nine days later: in LCB, 91/91 hidden test sets are supersets of the visible tests, so "would pass hidden but fails visible" cannot happen by construction. This criterion had zero statistical power.