2026-07-26 · Defense axis · Not re-verified

The trust on/off experiment on a real model: +8.3%, but statistically nothing happened

60 paired tasks; switching the trust layer on raised the pass rate from 51.67% to 60.00% — but the 95% interval spans 0, McNemar p=0.332, and statistical power is only 0.163. The same data holds another, more meaningful result.

real-model experimentMcNemarstatistical power

This was the project's first paired defense experiment on a real model, and the first time we got "a null result on a real model."

#Design


Model	qwen3.6-35b-a3b @ 100.119.113.56:8765
Task bank	EvalPlus MBPP+ v0.2.0 (378 tasks, sha256 pinned)
Tasks	60 paired tasks
Roster	Demo roster, 2 saboteurs planted by hand
Pairing key	`task_id` = `sha256(prompt + tests)`, stable across arms
Quality check	`compile_check(t.check)(answer)`, outside the arm

#Results

Arm	Passed	Valid	Pass rate	Time	ledger events	Calls per pass
off	31	60	0.5167	6,837.5 s	ROUTE 60, ATTEMPT 60, DELIVERED 60	1.94
on	36	60	0.6000	6,168.0 s	+REVIEW 180, AUDIT 60	1.67

Sounds great, but statistically nothing happened


n	60
Only on correct (b)	11
Only off correct (c)	6
Discordant	17
Δ	+0.0833
95% CI	[−0.05, 0.2167]
McNemar p	0.3323

The interval spans 0, p = 0.332. And statistical power is only 0.163; reaching 80% power would need n = 337.

#The first mismeasurement: reading the wrong field

This false result deserves its own section

But the off arm doesn't audit, by design, so its audit.passed is always empty. The result:

	Reading `audit.passed`	Using the outside-the-arm check
off	0 / 11	7 / 12
on	5 / 10	5 / 10
Conclusion	on far ahead	the difference disappears

After the correction it is off 7/12 vs on 5/10 — no difference.

This one is kept in a comment in examples/realmodel_suite.py as a classic trap. The general rule: if you are comparing "with a feature on" and "with it off," the quality criterion must never be a field produced by that feature itself.

#The second outcome variable: more interesting

We also measured the amount of work assigned to saboteurs. This is a direct indicator of "doing bad work and still getting jobs."

	Share assigned to saboteurs
off	20 / 60 (33.3%)
on	11 / 60 (18.3%)

Test	p
All 60 tasks	0.0931 (does not reach .05)
Drop 10 warm-up tasks	0.021 – 0.041
Drop 20	0.021 – 0.041
Drop 30	0.021 – 0.041
Drop 35	0.0703
Drop 40	0.2188

#Honest boundary


The saboteurs were planted by hand	They did not learn to do harm on their own. Attacker capability is underestimated.
Two saboteurs / 8 residents	Very small scale; ecosystem interaction is almost nonexistent.
Single model	qwen3.6-35b-a3b, no cross-model comparison.
Audit rate 0.2, reviewer accuracy 0.7	Both constants are configured values, never measured.
n = 60	Statistical power 0.163. Needs n = 337.

#Memory, three arms (E11, same day)

Arm	Passed	Time
M0 (no memory injected)	8 / 12 (66.67%)	846.2 s
M1 (last k entries injected verbatim)	8 / 12 (66.67%)	1,012.9 s
M2 (distilled into lessons, 2,000-token budget)	8 / 12 (66.67%)	2,144.6 s

#The value of this null result

#Next

Six things we mismeasured: measurement-traps
The three nulls on the gain axis: hypotheses-excluded
How statistical power is computed: mde-and-power

Six things we mismeasured
Reading the wrong field, network drops counted as reviewer results, an rng seed that degenerates at the limit, treating `defected <= BUDGET` as an equal budget, pretty numbers at an endpoint, and a rule switched off by default — six real measurement traps and their lessons.
The only truly tight constraint in the whole defense system: reviewer accuracy
We swept the stake, probation length, ecosystem size, attacker share, and the correlation of reviewer errors — all of them are slack. The only cell where attacking starts to pay is when reviewer accuracy drops to 0.
The MDE is 4.31 percentage points, but the task bank has only 378 tasks
For the comparison of the gate against majority vote, we can calculate that 491 pairs are needed for 80% statistical power, but the task bank actually has only 378 tasks. This "Same sign, unresolved" is not unfinished work; it is physically unmeasurable.
The harder the task, the more the acceptance gate is worth
The early-stop gate delivers 7.8 percentage points more on MBPP+, and 20.9 percentage points more on hard LiveCodeBench problems. And the "leak rate" drops from 44 to 11 — refusing delivery is worth the most on hard tasks.