After five replications: the headline claim lost to its own stability rule
R460 measured the revision loop at 13.33 percentage points above resampling with a fresh attempt, Holm p = .0112. After five same-task replications, all five were not significant, 0/5 passed correction, and the status was downgraded to "Same sign, unresolved."
This page cites 3 verdicts: Held 2 · Same sign, unresolved 1click one to jump to its claim card
#The order of events
2026-09-07: R460's preregistration fixed four state rules.
2026-09-11: R460 finished running (LCB v2, 120 tasks, 6 blocks × 20).
| Arm | Successes | Rate |
|---|---|---|
| OFF | 70 / 120 | 58.33% |
| CONFORM | 85 / 120 | 70.83% |
| OFF5 | 78 / 120 | 65.00% |
| H-PI | 91 / 120 | 75.83% |
| H-OC | 96 / 120 | 80.00% |
| H-MIX | 101 / 120 | 84.17% |
H-MIX − CONFORM: b=22, c=6, +13.33 pp, conditional interval [4.22, 19.46], Holm p = .0112 → EFFECTIVE.
2026-09-11: R460R's preregistration fixed the claim rule for "five same-task replications."
2026-09-12: all five runs finished.
#The preregistered claim rule
| Judge each run separately | All five must pass |
| Forbidden | Pooling n, averaging, picking one run to report |
| Decision threshold | Conditional interval lower bound > 0 and passes Holm correction |
#Results of the five replications
Delivery rate (%):
| Arm | r1 | r2 | r3 | r4 | r5 |
|---|---|---|---|---|---|
| OFF | 57.50 | 54.17 | 57.50 | 51.67 | 51.26 |
| OFF5 | 60.83 | 61.67 | 59.17 | 66.67 | 68.64 |
| CONFORM | 71.67 | 72.50 | 75.00 | 70.83 | 70.94 |
| H-PI | 76.67 | 79.17 | 75.83 | 80.83 | 70.83 |
| H-OC | 76.67 | 73.33 | 75.00 | 78.33 | 75.00 |
| H-MIX | 77.50 | 76.67 | 75.83 | 73.33 | 74.79 |
H-MIX − CONFORM:
| r1 | r2 | r3 | r4 | r5 | |
|---|---|---|---|---|---|
| Δ | +5.83 | +4.17 | +0.83 | +2.50 | +4.31 |
| Holm p | .6301 | .9165 | 1.0000 | .6776 | .9224 |
| 95% CI | [−2.79, 12.89] | [−5.35, 12.80] | [−7.44, 8.89] | [−5.94, 10.28] | [−4.54, 12.01] |
| Status | INCONCLUSIVE | INCONCLUSIVE | RULED_OUT | INCONCLUSIVE | INCONCLUSIVE |
0 / 5 passed Holm.
#Five mandatory accompanying reports
The decision document requires five things to be written down together; none may be left out:
| ① Same sign in all five | ✓ All five are + |
| ② Zero passed Holm | ✓ |
| ③ All five uncorrected intervals intersect R460 | ✓ R460's interval is [4.22, 19.46] |
| ④ All five upper bounds are below +13.33 | ✓ 12.89 / 12.80 / 8.89 / 10.28 / 12.01 |
| ⑤ The cross-bank result also did not pass Holm | ✓ R529: +1.12 pp, Holm p = .3409 |
#Forbidden wording
The decision document lists the forbidden phrases directly:
"The replication is stable," "the replication failed," "the effect disappeared," "equivalent," "a tie," "the loop is useless."
No pooling n, no averaging, no picking one run to report.
#But two claims hold on the same data
#Loop vs a single draw
| Result | |
|---|---|
| Three H arms vs OFF | All 15 / 15 cells pass their own Holm |
| Δ range | +17.5 ~ +29.2 pp |
#Gate vs a single draw
CONFORM − OFF: +14.17 / +18.33 / +17.50 / +19.17 / +18.97 pp (outside every preregistered family, uncorrected).
The bulk of the gain is "executable acceptance gate + resampling"; the feedback loop is only a small increment stacked on top of it.}}
#Exploratory: voting loses to the gate
| Replication | OFF5 − CONFORM | Uncorrected p |
|---|---|---|
| r1 | −10.83 pp | 0.019 |
| r2 | −10.83 pp | — |
| r3 | −15.83 pp | — |
| r4 | −4.17 pp | — |
| r5 | −1.74 pp | — |
3/5 have uncorrected p < .05 — and the two that are not significant are exactly r4 and r5, the two runs with zero backend coexistence.
Backend coexistence rate: 8.5% / 71.3% / 2.1% / 0% / 0%.
#R529: it does not hold across task banks either
| Task-bank set | n | H−C b/c | C−O b/c |
|---|---|---|---|
| LCB v3 medium | 135 | 6/5 | 14/4 |
| LCB v3 hard | 54 | 5/3 | 3/0 |
| HumanEval+ | 156 | 5/4 | 21/3 |
| MBPP+ | 371 | 15/11 | 26/8 |
| Pooled | 716 | 31/23 | 64/15 |
| Comparison | Difference | Statistics |
|---|---|---|
| H-MIX − CONFORM | +1.12 pp | Interval [−1.02, 3.13], Holm p = .3409 |
| H-MIX − OFF | +7.96 pp | Holm p = 2.02 × 10⁻⁸ |
| CONFORM − OFF | +6.84 pp | Outside the family |
8 predictions made in advance: 4 hit, 4 missed → INCONCLUSIVE.
#R532: negative after switching to a stronger model
2026-09-17/18. qwen3.8-27b Q4_K_M, non-reasoning, 836 tasks, 43 blocks, 0 voided.
H-MIX − CONFORM is negative in all five task-bank sets: −5.83 / −5.19 / −16.67 / −1.28 / −0.54. Pooled −3.23 pp, interval [−5.52, −0.75], Holm p = 0.0101.
#Final status
| Claim | Status |
|---|---|
| The loop delivers more than a single draw | Held (15/15 pass Holm) |
| The gate delivers more than a single draw | Held (all five significant outside the family) |
| The bulk of the gain is gate + resampling | Held (audit judgment) |
| The loop delivers more than resampling with a fresh attempt, without spending more tokens | Same sign, unresolved |
| The other two loop arms also beat resampling | Same sign, unresolved (10/10 did not pass Holm) |
| The loop's gain comes from the loop, not the prompt | Held (D5 attribution) |
How that last one was measured
And why "without spending more tokens" does not hold either
#What this case taught
#Next
- The two claims that hold on the same data: loop-vs-single-shot
- The token ledger: token-cost-ledger
- Loop vs prompt attribution: loop-not-prompt
- The list of downgraded claims: Corrections log
