The birth of the judging gate: execution instead of opinion, receipts instead of votes
We deleted the AI review committee and replaced it with "run the customer's acceptance tests, deliver the first candidate that passes, refuse delivery if none passes." On three task banks it delivered 4.58 to 19.17 percentage points more, at only 1.5 calls per task.
#The rule (frozen word for word)
- Generate candidates in order.
- Run the visible acceptance tests on every candidate in a sandbox.
- Let the first candidate that passes through, and stop generating.
- At most five. If none passes, refuse delivery.
- Refused delivery counts as a failure, and the denominator is all tasks.
#Why "stop at the first pass" works
Because the ceiling of any selection rule is the share of tasks where at least one candidate in the pool is correct. Majority vote can't escape this ceiling either.
Two saved batches of five-candidate data:
| Task set | Candidate-pool ceiling | Tasks with all five wrong |
|---|---|---|
g_r441 | 82.68% | 31 / 179 |
g_r356 | 85.03% | 22 / 147 |
#Convergence across three task banks
| Task bank | n | Difference (vs one single sample) | Statistic | Calls per task |
|---|---|---|---|---|
| MBPP+ (pooled 371) | 371 | +4.58 pp | 95% CI [+0.57, +8.04] | 1.51 |
| LCB v2 (hard tasks) | 120 | +19.17 pp | b=31 c=8, p = 0.0003 | 1.71 |
| LCB v3 | 189 | +7.94 pp | b=25 c=10, p = 0.0167 | 1.55 |
The harder, the more it's worth
#Refused tasks are not bad luck
We audited every task the gate refused after the fact, to see whether all five drafts really were wrong:
| Batch | Tasks refused | All five wrong |
|---|---|---|
| r447 | 7 | 7 / 7 |
| r449b | 8 | 8 / 8 |
| r449c | 10 | 10 / 10 |
| r461 | 15 | 15 / 15 |
#EQ5: the cleanest comparison
This is the best-designed experiment on the whole gain axis — the same 5 candidates, the same 5 calls; one delivers the first that passes, the other takes a majority vote of the five. The candidates are fully aligned, and so is the budget.
| Stage / task bank | Gate success | Majority-vote success | Δ | Uncorrected p |
|---|---|---|---|---|
| r446 MBPP+ | 280 / 371 | 265 / 371 | +4.04 | .0135 |
| r448 MBPP+ (new seed) | 286 / 371 | 273 / 371 | +3.50 | .0241 |
| r449b LCB v2 | 85 / 120 | 75 / 120 | +8.33 | .0414 |
| r449c LCB v3 | 157 / 189 | 149 / 189 | +4.23 | .0963 Same sign, unresolved |
#Directly checking the candidate-pool ceiling
R440P checked it once with a zero-API offline replay (recomputing the saved five candidates):
g_r441 | g_r356 | |
|---|---|---|
| Candidates | 895 | 735 |
| Early-stop pass rate | 78.21%, 1.34 calls | 82.99%, 1.13 calls |
| What OFF5 actually shipped | 73.74%, 5.00 calls | 77.55%, 5.00 calls |
| Difference | +4.47 pp, p=0.0963 | +5.44 pp, p=0.0574 |
| Early stop vs single sample | +7.82 pp, b=14 c=0, p=0.0001 | +6.80 pp, b=10 c=0, p=0.0020 |
#The gate itself has been attacked
R451/R452 on 2026-09-06: as long as the acceptance suite is a piece of "trusted code," an attacker can break the gauge.
| Stage | Result |
|---|---|
| Suite bound into the commit, plain suite | 371 / 371 all refused (normal) |
| Stateful gauge variant | Delivery rate dropped 75.20 pp, while the gauge reported ok |
| Fix: SuiteSpec (the suite becomes data) | exec smuggling 368 / 371 → 0 / 371 |
| Residual: generic_gauged | +2.72 pp |
| After-the-fact upper bound: weak_oracle | +4.35 pp |
#The gate's premise
We put this sentence word for word into every document that reports effectiveness numbers. Deleting that block deletes the condition under which those numbers hold.
A related structural finding: in the task banks we used, the hidden tests were always a superset of the visible tests (V ⊆ GT), so the sentence "visible filtering is lossless" is true by construction and has no statistical power. v-gt-is-not-lossless
#Honest boundaries
#Next
- The full discussion of the premise: v-gt-is-not-lossless
- How task difficulty changes the value of this rule: harder-tasks-more-worth
- The gate's full specification: Acceptance gate CONFORM
