Six things we mismeasured

This post gathers in one place everything in the project that we mismeasured. Every one of them really happened, and every one of them nearly became a false positive or negative conclusion.

#1. Reading the wrong field produced a false result

The first analysis of the E10 real-model experiment:

The right way: the quality check must be independent of the arm — judge independently with compile_check(t.check)(answer), outside the arm.

This one is kept in a comment in examples/realmodel_suite.py as a classic trap.}}

#2. Network drops counted as "reviewer pass"

2026-06-30, re-verification of crowd routing. The first round of a2hard printed ✓, but a per-call I/O audit caught this:

• Calls 20–23 (exactly good_1's 4 peer reviewers) ran into Connection refused / reset by peer.
• The responses were empty strings, and the matching rule was a negated match on \bYES\b — an empty string counted as "not YES," that is, "fail."
• Result: good_1's peer reputation was wrongly pushed down to 0.50, the peers' hit rate against ground truth fell from 88% to 75%, and the catch rate fell to 71%.

The apparent effect was −4%.

#3. One optional parameter quietly shifted every number

SimConfig.digest() was two things at once:

1. The RNG seed.
2. A hash of every field.

So after a developer "added one optional parameter," every number from E1 to E16 changed, without any error message.

The fix:

• Fields added later are excluded from the digest when they take their default value.
• Three config digests are pinned so that the 15/15 counts reproduce bit for bit.

#4. Treating "within budget" as "equal budget"

Our E19 wanted to compare "which attack strategy is more economical," so it set an upper limit of 12 on the total number of misbehaviors:

The fix: use defected == BUDGET instead, and verify in advance that every arm really is bound.

Rerun at budget = 1 (the only point where all five arms are bound), the ranking reversed:

whitewash is the weakest, not the strongest.}}

The full rules and limits

#5. Measuring an effect at a degenerate endpoint

At blindspot = 1.0, the 30 seeds collapse into the same trajectory:

And our headline numbers — pulse 21.0, patient 79.0 — are exactly this endpoint. With it excluded, the blind-spot effect drops from 8.9× to 3.28×, and from 54× to 5.68×.

#6. A default was switched off, and we thought it was on

arm_off5 hard-coded accepted = True in the program — it had no path to refusal.

So the three-arm "ON vs OFF5" comparison in R437 was really:

an arm with a free gate vs an arm forbidden to refuse

#7 (bonus): a rejected rule counts as a trap too

In the first round of E15 selective attack, the high-value hit rate across 20 seeds was 0.00 — which is mathematically impossible, since the baseline is 20%.

Root cause: task value was a deterministic function of the round index, and the probation quota was "route to a probationer once every 10 rounds," while the attacker's probation period happened to expire every 10 rounds. The two were perfectly correlated across all 20 seeds.

After changing it to sha256(seed, round), a rerun with 30 seeds gave 18–20%.

#What these seven have in common

#Next

• The four formal analysis disciplines: Experimental method and verdict values
• The full numbers of E10: e10-paired-null