2026-09-06 · Gain axis · Same sign, unresolved

The MDE is 4.31 percentage points, but the task bank has only 378 tasks

For the comparison of the gate against majority vote, we can calculate that 491 pairs are needed for 80% statistical power, but the task bank actually has only 378 tasks. This "Same sign, unresolved" is not unfinished work; it is physically unmeasurable.

statistical powerMDEstatisticsSame sign unresolved

#Calculate first, then run

Before R529 (cross-bank), we first calculated one thing: how many tasks do we need to have an 80% chance of detecting a real effect when one exists?


MDE @ n = 371	4.31 percentage points
Needed for 80% statistical power	about 491 pairs
Actual task bank size	378 tasks

#Measured results

Task bank	n	Δ	Statistics
LCB v2	120	+6.67 pp	p = 0.15
LCB v3	189	+1.59 pp	95% CI [−3.17, +6.35], p = 0.6636
MBPP+ clean replication (new 192 tasks)	192	+4.69 pp	95% CI [−1.11, 9.42]

#The widths of the three intervals

Task bank	95% CI	Half-width
LCB v3 (n=189)	[−3.17, +6.35]	4.76 pp
MBPP+ replication (n=192)	[−1.11, +9.42]	5.27 pp
MBPP+ pooled (n=371)	—	Half-width corresponds to the MDE of 4.31 pp

#What the same concept looks like in other experiments

E10's full numbers

R460's statistical power and its consequences

Experiment	n	Statistical power	Needed for 80%
E10 trust switch	60	0.163	337
R460 for +10 pp	120	0.43 – 0.63	—
R529 pooled	716	(effect size too small)	—
Gate vs majority vote	371	—	491

The last row is the only case that is "113 tasks short" — the others are short by several times over.}}

#Four features of this judgment

#Why the EQ5 comparison does hold

Candidate alignment solves the magnitude problem

Stage / task bank	Gate successes	Majority-vote successes	Δ	Uncorrected p
r446 MBPP+	280 / 371	265 / 371	+4.04	.0135
r448 MBPP+ (new seed)	286 / 371	273 / 371	+3.50	.0241
r449b LCB v2	85 / 120	75 / 120	+8.33	.0414
r449c LCB v3	157 / 189	149 / 189	+4.23	.0963

Three of the four stages are significant, uncorrected. And that comparison, with only n=120 (LCB v2), got +8.33 — because the pairing is tighter (the same candidates, so the variance comes only from the selection rule).

#The general rule from this case

#Next

The six definitions of verdict values: verdict-values
The statistical-power problem in the five replications: five-replications
The same problem in E10: e10-paired-null

Six verdict values: how to say "not measured" without saying "no difference"
Held, Refuted, Overstated, No discernible difference, Same sign, unresolved, Not re-verified — each of these six terms has a strict definition. We keep them apart so that "I didn't detect it" is never read as "I tested it, and there's nothing."
After five replications: the headline claim lost to its own stability rule
R460 measured the revision loop at 13.33 percentage points above resampling with a fresh attempt, Holm p = .0112. After five same-task replications, all five were not significant, 0/5 passed correction, and the status was downgraded to "Same sign, unresolved."
The trust on/off experiment on a real model: +8.3%, but statistically nothing happened
60 paired tasks; switching the trust layer on raised the pass rate from 51.67% to 60.00% — but the 95% interval spans 0, McNemar p=0.332, and statistical power is only 0.163. The same data holds another, more meaningful result.
This is a success-rate gain, not a token-efficiency gain
Recomputed call by call, 197 cells, zero differences. H-MIX uses 2.23–2.78× the tokens of a single draw, and in four of the five replications it was more expensive per correct delivery than the gate. "The same ceiling" is not "the same actual cost."