2026-09-12 · Gain axis · Held

The only thing that held stably across five replications: the loop beats a single draw by 17 to 29 percentage points

All 15 blocks pass their own Holm correction, with differences from +17.5 to +29.2 percentage points, across five same-task replications and four mutually exclusive task sets. This is the hardest result in the whole project.

preregistrationloopgainheld

This page cites 2 verdicts: Held 2click one to jump to its claim card

HeldThe feedback loop delivers more than taking a single sample
HeldThe bulk of the gain is gate + resampling; the loop is a small increment

#The difference in what is compared is the whole point

Comparison	Result	Verdict
Loop vs a single draw (OFF)	+17.5 ~ +29.2 pp, 15/15 pass Holm	Held
Gate vs a single draw (OFF)	+14.17 ~ +19.97 pp	Held
Loop vs resampling with a fresh attempt (CONFORM)	+0.83 ~ +5.83 pp, 0/5 pass	Same sign, unresolved
Loop vs voting (OFF5, exploratory)	−1.74 ~ −15.83 pp	Same sign, unresolved

harness.loop_beats_single_shot_replicatedharness-2026-09-08HeldMeasurement (**five** same-task replications × three harness arms + four mutually exclusive task sets)

Spending five calls on a feedback loop (H-MIX/H-PI/H-OC) delivers more than taking a single sample and accepting it (OFF) — this one holds up across task banks and across replications

Supporting figures

R460R_vs_OFF_by_rep: {"HMIX":[{"rep":1,"num":93,"off_num":69,"den":120,"delta_pp":20,"b":29,"c":5},{"rep":2,"num":92,"off_num":65,"den":120,"delta_pp":22.5,"b":34,"c":7},{"rep":3,"num":91,"off_num":69,"den":120,"delta_pp":18.33,"b":28,"c":6},{"rep":4,"num":88,"off_num":62,"den":120,"delta_pp":21.67,"b":33,"c":7},{"rep":5,"num":88,"off_num":61,"den":118,"delta_pp":22.88,"b":34,"c":7,"denominator":"complete-case 118"}],"HPI":[{"rep":1,"delta_pp":19.17,"b":26,"c":3,"den":120},{"rep":2,"delta_pp":25,"b":35,"c":5,"den":120},{"rep":3,"delta_pp":18.33,"b":29,"c":7,"den":120},{"rep":4,"delta_pp":29.17,"b":39,"c":4,"den":120},{"rep":5,"delta_pp":19.33,"b":31,"c":8,"den":119}],"HOC":[{"rep":1,"delta_pp":19.17,"b":28,"c":5,"den":120},{"rep":2,"delta_pp":19.17,"b":30,"c":7,"den":120},{"rep":3,"delta_pp":17.5,"b":28,"c":7,"den":120},{"rep":4,"delta_pp":26.67,"b":38,"c":6,"den":120},{"rep":5,"delta_pp":24.37,"b":31,"c":2,"den":119}],"holm":"all fifteen cells significant (family of 6 tests within each replication)"}
R529_HMIX_vs_OFF_by_set: [{"set":"lcb3_medium","num":126,"off_num":115,"den":135,"delta_pp":8.15,"b":17,"c":6},{"set":"lcb3_hard","num":43,"off_num":38,"den":54,"delta_pp":9.26,"b":7,"c":2},{"set":"humanevalplus","num":148,"off_num":129,"den":156,"delta_pp":12.18,"b":24,"c":5},{"set":"mbppplus","num":299,"off_num":277,"den":371,"delta_pp":5.93,"b":31,"c":9}]
R529_HMIX_vs_OFF_pooled: {"num":616,"off_num":559,"den":716,"delta_pp":7.96,"b":79,"c":22,"holm_p_adj":2.02e-8}
Number of real sources: 3
Denominator warning: HumanEval+ is 156/164 (8 tasks excluded by the sandbox envelope); the three comparison denominators of R460R r5 each differ (118/119/119, 7 infra_void rows); do not backfill them uniformly to 120
Backend load is not homogeneous: The co-tenancy rates of the five replications and R529 are 8.5/71.3/2.1/0/0%; the effect size in this claim is too large for co-tenancy to explain (magnitude ceiling ≤1.7pp/arm), but it must still be stated

Source record: runs/g_r460r{1,2,3,4,5}_harness_lcb2_*/rows.jsonl (five × 120 tasks; r5 has 713 rows); runs/g_r529_*/rows.jsonl (four sets, 716 tasks)

Read the full note →

#The 15 blocks in detail

	Result
Number of blocks	5 replications × 3 H arms = 15
Blocks passing their own Holm correction	15 / 15
Δ range	+17.5 ~ +29.2 pp

#Gate vs a single draw

CONFORM − OFF, five runs:

	r1	r2	r3	r4	r5
CONFORM	86	87	88	87	85
OFF	69	65	66	60	58
Δ	+14.17	+18.33	+17.50	+19.17	+18.97
Uncorrected p	0.0015	—	—	—	—

#Attribution: the bulk of the gain is gate + resampling

harness.gain_comes_from_executable_gate_and_resampleharness-2026-09-08HeldAttribution (**audit judgment**, not a preregistered primary endpoint; CONFORM−OFF is an exploratory quantity outside the family)

The bulk of the gain is "the executable acceptance gate + resampling"; the feedback loop is only a small increment stacked on top of it

Supporting figures

R460R_CONFORM_vs_OFF_by_rep: [{"rep":1,"conform_num":86,"off_num":69,"den":120,"delta_pp":14.17,"b":22,"c":5,"p_uncorrected":0.0015},{"rep":2,"conform_num":87,"off_num":65,"den":120,"delta_pp":18.33,"b":28,"c":6,"p_uncorrected":0.0002},{"rep":3,"conform_num":90,"off_num":69,"den":120,"delta_pp":17.5,"b":26,"c":5,"p_uncorrected":0.00019},{"rep":4,"conform_num":85,"off_num":62,"den":120,"delta_pp":19.17,"b":30,"c":7,"p_uncorrected":0.00019},{"rep":5,"conform_num":82,"off_num":60,"den":116,"delta_pp":18.97,"b":28,"c":6,"p_uncorrected":0.0002,"denominator":"complete-case 116 (7 infra_void rows voided)"}]
R529_CONFORM_vs_OFF_by_set: [{"set":"lcb3_medium","conform_num":125,"off_num":115,"den":135,"delta_pp":7.41,"b":14,"c":4,"p_uncorrected":0.031},{"set":"lcb3_hard","conform_num":41,"off_num":38,"den":54,"delta_pp":5.56,"b":3,"c":0,"p_uncorrected":0.25},{"set":"humanevalplus","conform_num":147,"off_num":129,"den":156,"delta_pp":11.54,"b":21,"c":3,"p_uncorrected":0.00028},{"set":"mbppplus","conform_num":295,"off_num":277,"den":371,"delta_pp":4.85,"b":26,"c":8,"p_uncorrected":0.0029}]
R529_CONFORM_vs_OFF_pooled: {"conform_num":608,"off_num":559,"den":716,"delta_pp":6.84,"b":64,"c":15,"p_uncorrected":2.3e-8}
How much more the loop adds: {"R460R":[5.83,4.17,0.83,2.5,4.31],"R529":[0.74,3.7,0.64,1.08]}
All p values uncorrected: The p_raw of CONFORM−OFF is < 0.002 in all five replications, but CONFORM−OFF is not in any preregistered Holm family ⇒ it may not be read as "the gate's effect is significant"
Holm for the four sets, added after the fact (descriptive only): {"humanevalplus":0.0011,"mbppplus":0.0088,"lcb3_medium":0.062,"lcb3_hard":0.25}
Not causal: CONFORM and H-MIX are two separately run arms, not two stages; "gate +14pp, loop +6pp" is a narrative obtained by subtraction, not separated components

Source record: runs/g_r460r{1,2,3,4,5}_harness_lcb2_*/rows.jsonl; runs/g_r529_*/rows.jsonl

Read the full note →

Stacking the five runs' data together:

	Difference
Three H arms vs OFF	+17.5 ~ +29.2 pp
CONFORM vs OFF	+14.17 ~ +19.97 pp
H-MIX vs CONFORM	+0.83 ~ +5.83 pp (0/5 pass)

#Where the gain comes from: back to R460P's offline replay

Using a zero-API recomputation to take the candidate pool apart:

	`g_r441`	`g_r356`
Candidate-pool ceiling (at least one correct)	82.68%	85.03%
Rate reached by the early-stop gate	78.21%	82.99%
Early stop vs single draw	+7.82 pp, b=14 c=0	+6.80 pp, b=10 c=0
Early stop vs five-way vote	+4.47 pp, p=0.0963	+5.44 pp, p=0.0574

#Cost: this claim comes with a condition

The full token ledger

R460 tokens per task:

Arm	Rate	tokens/task	tokens/correct
OFF	58.33%	2,913	4,994
CONFORM	70.83%	5,805	8,195
OFF5	65.00%	15,004	23,084
H-PI	75.83%	9,448	12,459
H-OC	80.00%	7,432	9,290
H-MIX	84.17%	5,677	6,745

#The boundaries of this claim

#Next

The day the headline claim was downgraded: five-replications
Cost: token-cost-ledger
Whether the gain comes from the loop or the prompt: loop-not-prompt
How the gate makes "selection" possible: conformance-gate

After five replications: the headline claim lost to its own stability rule
R460 measured the revision loop at 13.33 percentage points above resampling with a fresh attempt, Holm p = .0112. After five same-task replications, all five were not significant, 0/5 passed correction, and the status was downgraded to "Same sign, unresolved."
This is a success-rate gain, not a token-efficiency gain
Recomputed call by call, 197 cells, zero differences. H-MIX uses 2.23–2.78× the tokens of a single draw, and in four of the five replications it was more expensive per correct delivery than the gate. "The same ceiling" is not "the same actual cost."
The gain comes from the loop, not from the first-round prompt
We pulled out the complete first-round responses and re-scored them offline: the loop contributes +19.17 / +17.50 percentage points, the first-round prompt only +6.67 / +8.33. 21 tasks were rescued by the loop.
The birth of the judging gate: execution instead of opinion, receipts instead of votes
We deleted the AI review committee and replaced it with "run the customer's acceptance tests, deliver the first candidate that passes, refuse delivery if none passes." On three task banks it delivered 4.58 to 19.17 percentage points more, at only 1.5 calls per task.
Only after our own experiments excluded two explanations did the mechanism have room to change
"The model is too strong" and "the tasks are too easy" were two explanations proposed by humans. Two experiments excluded both, and only then was the reviewer mechanism forced into a redesign — and the direction was to change the referee, not the reviewer model.