Preregistration: why we write down the criteria before running the experiment
Criteria, statistical methods and the four state rules are all frozen and timestamped before the run starts. This post explains how we preregister, and why "adding a criterion after the fact" doesn't count in this research.
#Why this is needed
Because choosing criteria after the fact is one of the most common forms of statistical cheating, and it doesn't require anyone to lie. All you have to do is, after seeing the results, pick a threshold that "was going to pass anyway" and write it into the report — nobody in the whole process does anything wrong, but the conclusion is false.
We ran into this once. §7 of R440P has a premise that reads "visible filtering is lossless." It is a criterion, written in the decision document, and it looks fine.
Nine days later R440Q refuted it:
In LCB v1, 91/91 hidden test sets are supersets of the visible tests (
V ⊆ GT).
So "would pass hidden but fails visible" cannot happen, by construction. This isn't "no counterexample was measured"; it is "this proposition is always true on this task bank." A criterion that is always true has zero statistical power.
#The four state rules
R460's decision document fixed four states on 2026-09-07:
| State | Trigger condition |
|---|---|
| EFFECTIVE | Interval lower bound > 0, and passes Holm correction |
| INCONCLUSIVE | Direction correct, but did not pass any threshold |
| RULED_OUT | Interval upper bound < threshold (= confirms the effect is smaller than the threshold) |
| Not triggered | A falsification condition fixed in advance holds (for example c ≥ b) |
Why "not measured" gets its own cell
#Refused delivery counts as failure
This is the second hard rule, and the one most easily overlooked:
Refused delivery counts as failure; the denominator is all tasks.
The reason is simple. If the denominator only counted "tasks that were delivered," then "refuse everything" would get a 100% success rate. A system that did nothing and delivered nothing could get full marks under that rule.
Our code had a real mechanism that could do exactly this: the early arm_off5 hard-coded accepted = True — it had no path to refusal. So the "ON vs OFF5" comparison in R437 was really "an arm with a free gate" against "an arm forbidden to refuse."
#Disclaimers registered alongside
In R460's decision document, before the +13.33 figure was seen, two items were already registered:
- Statistical power for +10 pp at n = 120 is only 0.43 – 0.63, so the +13.33 must be treated as an estimate toward the upper end.
- 9 advance predictions, one of which expected revision to increase false delivery.
The second later went in the opposite direction: H-MIX 14/120 < CONFORM 29/120. Revision didn't increase false delivery; it reduced it.
#The five replications are the same rule
R460R on 2026-09-12 was the strictest application. The claim rules fixed in advance:
- Judge each one separately; all five must pass.
- No pooling n.
- No averaging.
- No picking one run to report.
Result: 0/5 passed. The verdict was "Same sign, unresolved."
Full details of the five replications
#What preregistration cannot protect
To be honest and clear about it:
| It cannot protect | Selection bias (picking a model you want to win with) |
| It cannot protect | Stopping rules (stop as soon as p < .05) |
| It cannot protect | Data quality (only on 2026-07-09 did we find that model I/O had never been saved at all) |
| It cannot protect | Voiding rules (voiding blocks that are hard to run) |
| What it protects | The criteria are frozen before the numbers are seen |
We tripped over the last one ourselves: R437's ON arm had infra_void 66/179 = 36.9%, and R437 itself logged a VOID-GATE-WARNING line. So we downgraded the claim "the first clean complete dataset" to "no known bug contamination, paired subset n = 101."
#Next
- Full definitions of the six verdict values: Experimental method and verdict values
- Verdicts for all claims: Evidence ledger
