Six verdict values: how to say "not measured" without saying "no difference"
Held, Refuted, Overstated, No discernible difference, Same sign, unresolved, Not re-verified — each of these six terms has a strict definition. We keep them apart so that "I didn't detect it" is never read as "I tested it, and there's nothing."
This is the most important set of terms on the whole site. Every public claim must land in one of these cells, and there is no seventh.
#Why split them so finely
Because "not measured" and "measured, and nothing there" are two different things, and reports often write them as the same sentence.
- Measured, and nothing there → you can say "this has no effect under these conditions." That is an informative conclusion.
- Not measured → you can only say "I can't tell them apart at this scale." That is not a conclusion; it is the limit of the data.
Our R529 happens to contain both situations at once, and the difference is plain to see:
| Figures | Verdict | |
|---|---|---|
| H-MIX − OFF | +7.96 pp, Holm p = 2.02 × 10⁻⁸ | Held |
| H-MIX − CONFORM | +1.12 pp, Holm p = .3409 | Same sign, unresolved |
The same experiment, the same data: one Held, one unresolved. The only difference is whether the size of the difference cleared the threshold.
#Six definitions
- HeldPassed the criterion set in advance (preregistered, run, then checked against the answer)
- RefutedThe claim is wrong; the correct statement is listed under "Corrected"
- OverstatedThe core holds but was stated too strongly, or the original evidence or mechanism explanation has a problem
- No discernible differenceMeasured, and no discernible effect — that is itself the answer, not an unfinished test
- Same sign, unresolvedThe preregistered rule triggered none of its states. Not measured — not the same as no difference
- Not re-verifiedNot yet sent for re-verification — not the same as passing
#Why "Same sign, unresolved" gets its own cell
Our data file claims.json carries this comment of its own:
verdict = 未複驗means nobody has tried to refute it yet; it does not mean it passed. (original: 「verdict = 未複驗 代表還沒有人試過推翻它，不代表它通過了。」)
Many of the 6 未複驗 entries in that claims.json are "I know it's quite likely wrong, but nobody has checked yet." The honest treatment is to label them Not re-verified, not Held.
#What "No discernible difference" looks like
There is only one, but it is the most convincing one in this research.
Its evidence is the same data, two measures, opposite directions:
| Measure | ON | OFF5 | Diff | p |
|---|---|---|---|---|
| Original | 75.25% | 77.23% | −1.98 pp | 0.7905 |
| After fixing the typing-whitelist bug | 84.16% | 81.19% | +2.97 pp | 0.4531 |
#How "Overstated" is judged
The criterion: the core stands, but it was stated too strongly, or the source evidence / mechanistic explanation has a problem.
Four claims, all on the defense axis:
| ID | The core that stands | Where it is overstated |
|---|---|---|
pulse.timing_minor | At the single point of blind spot 0.5, no effect of temporal structure was measured | But that is absence of evidence. Broken down, exposure spans 6.06× and efficiency spans 8.45× — an 8.45× difference in attack capability |
pulse.blindspot_dominates | The blind spot really does matter more than parameter choice | But it is held up almost entirely by the degenerate endpoint blind = 1.0. Interaction η² = 0.264 is 4 times the main effect, so the sentence pattern "the real variable is X" does not hold |
pulse.starvation | An identity that has been slashed really does find it very hard to come back | But "permanently" was confounded with a coincidence: 14 seeds in the same cell were never caught. slash 0.9 came back in round 177 |
gain.clean_dataset | That data has no known bug contamination | But the ON-arm void rate is 36.9%, so "the first clean complete dataset" should become "paired subset n=101" |
"Unimportant" and "not measured" are not the same thing
#"Not re-verified" is nothing to be ashamed of
We have 6 claims labeled Not re-verified:
| ID | Why it is still Not re-verified |
|---|---|
pulse.definition | The attack model isn't new, but "a configuration running four defenses at once" and "treating the blind spot as a second axis" may still be new |
pulse.reaction_lag | Even with zero blind spot it still misses 2.53 entries and takes 27.95 rounds on average to react — this figure has not been independently re-verified |
entry.fee_backfires | A deposit makes attacking profitable (ROI 2.354) — the mechanism is clear, but nobody has rerun it |
entry.reviewer_accuracy_binds | At acc=0, ROI 1.04 is the only cell in the whole table where "misbehaving is profitable" |
gain.signal_exists | The OFF failure rate of 26.44% falls inside the preset 20–60% window |
peerexec.mutual_execution_below_threshold | Naming behavior below the majority threshold holds (273/273), but the full claim has not been sent for re-verification |
#Phrases we ban
The R460R decision document lists a set of banned phrases; we quote it directly here:
"Replication stable," "replication failed," "the effect disappeared," "equivalent," "a tie," "the loop is useless." (original: 「複製穩定」、「複製失敗」、「效應消失」、「等價」、「打成平手」、「迴圈沒用」。)
No pooling n, no averaging, no picking one run to report. (original: 「禁止合併 n、禁止平均、禁止挑一次來報。」)
The reason is that each of these five terms corresponds to a common evasive move:
| Phrase | What it is really saying |
|---|---|
| Replication stable | Picking out the runs where the difference was small |
| Replication failed | Admitting failure, but in a tone that suggests "the environment was at fault" |
| The effect disappeared | Implying the original number was wrong |
| Equivalent / a tie | Turning "I can't tell them apart" into "they're the same" |
| The loop is useless | Jumping from "Same sign, unresolved" to no effect |
We nearly committed the first one ourselves
#Next
- Verdicts for all claims: Evidence ledger
- The full version of the wording discipline: Experimental method and verdict values
