2026-09-11 · Gain axis · Held

The gain comes from the loop, not from the first-round prompt

We pulled out the complete first-round responses and re-scored them offline: the loop contributes +19.17 / +17.50 percentage points, the first-round prompt only +6.67 / +8.33. 21 tasks were rescued by the loop.

attributionD5experiment design

#The first round of the three arms

The first-round prompts of H-MIX, H-PI and H-OC are almost the same: the original task text, the list of standard libraries allowed to be imported, and one line saying "first check your function by hand against the examples in the task."

The difference is in what happens afterwards:

Arm	After a failure
H-MIX	Compute the failure signature and paste the failure text back to the same worker
H-PI	The original pi-style feedback loop
H-OC	Plan → diagnose → self-test

#Results: split into two effects

Arm	Loop effect (final − first round)	Prompt effect (first round − OFF)	Tasks rescued by the loop	First-round visible pass rate
H-MIX	+19.17 / +17.50 pp	+6.67 / +8.33 pp	21	74.2%
H-PI	+18.33 / +15.83 pp	−0.83 / +1.67 pp	—	71.7%
H-OC	+19.17 / +18.33 pp	—	—	—

#Why this design could do it

#A reservation that has to be written down

#How this result relates to the headline claim

There is something here that looks like a contradiction

But D5 says "the loop contributes +19 percentage points." The two don't contradict each other, because the denominators differ:

D5's denominator is OFF (no resampling, no revision).
The headline claim's denominator is CONFORM (already resampled).

Relative to "no resampling at all," the loop has +19. Relative to "already resampling," it has only +0.8 ~ +5.8 (not measurable).

This is the same mechanism under two different baselines. The honest wording writes out both denominators:

Comparison	Difference	Verdict
Loop vs a single draw	+17 ~ +19 pp	Held
Loop vs resampling with a fresh attempt	+0.8 ~ +5.8 pp	Same sign, unresolved

The full 15/15-cell result

#What this design means in practice}}

#Next

The full verdict on loop vs resampling: five-replications
The 15/15 cells of loop vs single draw: loop-vs-single-shot
What KS-1 is, and why it forced this design: ks-one

After five replications: the headline claim lost to its own stability rule
R460 measured the revision loop at 13.33 percentage points above resampling with a fresh attempt, Holm p = .0112. After five same-task replications, all five were not significant, 0/5 passed correction, and the status was downgraded to "Same sign, unresolved."
The only thing that held stably across five replications: the loop beats a single draw by 17 to 29 percentage points
All 15 blocks pass their own Holm correction, with differences from +17.5 to +29.2 percentage points, across five same-task replications and four mutually exclusive task sets. This is the hardest result in the whole project.
KS-1: On a frozen model, "writing the consequences to look real" and "making the consequences real" are the same thing
On 2026-06-28, the two main validation designs were killed by adversarial workflows of 17 and 10 agents. The fact that killed them later became the methodological core of the whole project.
Our AI review committee lost to "generate five times, take the majority"
Same 5 calls, same set of tasks: a committee of three AI reviewers plus one revision lost to "generate five times and take a majority vote." The point estimate of the difference was −1.98 percentage points — and after we fixed an unrelated bug, it became +2.97.