2026-09-14 · Gain axis · Not re-verified

This is a success-rate gain, not a token-efficiency gain

Recomputed call by call, 197 cells, zero differences. H-MIX uses 2.23–2.78× the tokens of a single draw, and in four of the five replications it was more expensive per correct delivery than the gate. "The same ceiling" is not "the same actual cost."

costtokenstatistics

#Two definitions that are easy to confuse


`reasoning`	A subset of completion, not an extra amount added on top
Failed requests	The 116 failed requests in the core experiment have no usage and are counted as unknown (not filled in as 0)

#R460's cost

Arm	Delivery rate	tokens / task	tokens / correct delivery
OFF	58.33%	2,913	4,994
CONFORM	70.83%	5,805	8,195
OFF5	65.00%	15,004	23,084
H-PI	75.83%	9,448	12,459
H-OC	80.00%	7,432	9,290
H-MIX	84.17%	5,677	6,745

#Three ratios

Comparison	Per task	Per correct delivery
H-MIX vs OFF	2.23 – 2.78×	1.65 – 2.11×
H-MIX vs CONFORM	0.92 – 1.56× (R460R) / 1.07 – 1.69× (R529)	—
OFF5 vs CONFORM	2.16 – 2.85×, and a lower delivery rate every time	—

#"Without spending more tokens" does not hold

Our original headline claim said the revision loop, compared with resampling with a fresh attempt, "delivers more, without spending more tokens."

	r1	r2	r3	r4	r5
H-MIX vs CONFORM per task	0.92×	—	—	—	1.56×
Per correct delivery	—	—	—	—	More expensive than CONFORM in four runs

#Why "the same ceiling" is not "the same cost"

#The R529 by-host finding

R529 used four mutually exclusive task sets, spread across two LM Studio hosts:

Host	HumanEval+ result	H-MIX tokens/task	CONFORM tokens/task	Difference
Reasoning on (0.4.24)	Both arms 94 / 100	4,368	2,291	+90.7%
Reasoning off (0.4.17)	54 / 56 vs 53 / 56	924	1,064	−13.2%

reasoning makes up 63.0% – 96.3% of completion.

#A better design order

This ledger changed our design order

Strategy	tokens/task	tokens/correct	Calls per task
OFF (single draw)	2,913	4,994	1.00
CONFORM (early-stop resampling)	5,805	8,195	1.51
H-MIX (resampling + revision)	5,677	6,745	1.36

The gate's tokens/correct (8,195) is higher than the revision loop's (6,745). But the gate's rules are much simpler — no failure-signature computation, no "try a different approach" prompt, no context management.

And revision's relative advantage disappeared after R532 switched models (−3.23 pp).

R532's amendments

So our recommended order is: do early-stop resampling first, and treat revision as a later add-on.}}

#Honest boundaries

#Next

Delivery-rate results on the same data: loop-vs-single-shot
How cost affects the verdict on a claim: five-replications
The full gate rules: Acceptance gate

The only thing that held stably across five replications: the loop beats a single draw by 17 to 29 percentage points
All 15 blocks pass their own Holm correction, with differences from +17.5 to +29.2 percentage points, across five same-task replications and four mutually exclusive task sets. This is the hardest result in the whole project.
After five replications: the headline claim lost to its own stability rule
R460 measured the revision loop at 13.33 percentage points above resampling with a fresh attempt, Holm p = .0112. After five same-task replications, all five were not significant, 0/5 passed correction, and the status was downgraded to "Same sign, unresolved."
The birth of the judging gate: execution instead of opinion, receipts instead of votes
We deleted the AI review committee and replaced it with "run the customer's acceptance tests, deliver the first candidate that passes, refuse delivery if none passes." On three task banks it delivered 4.58 to 19.17 percentage points more, at only 1.5 calls per task.
Six things we mismeasured
Reading the wrong field, network drops counted as reviewer results, an rng seed that degenerates at the limit, treating `defected <= BUDGET` as an equal budget, pretty numbers at an endpoint, and a rule switched off by default — six real measurement traps and their lessons.