How "temporal structure doesn't matter" went wrong: breaking the aggregate apart
We measured total hits across 15 cells of attack strategies that differed by only 1.81×, then wrote "temporal structure barely affects the result." Broken apart: exposure spans 6.06× and efficiency spans 8.45× — attack capability differs by 8.45×.
#What we originally measured
The attack strategy is the "pulse attack": discharge B consecutive bad deliveries, charge R clean deliveries, repeat. We swept 15 (burst, recover) combinations, at blind spot 0.5:
| Value | Cell | |
|---|---|---|
| Fewest hits | 2.8333 | (1, 20) |
| Most hits | 5.1333 | (8, 20) |
| Span | 1.81× |
Based on this number, we wrote in the report:
The temporal structure of the attack barely affects the total number of hits.
That sentence is wrong. Or, more precisely, it is a reasoning error, not a calculation error.
#What the re-verification agent found
It split "total hits" into two quantities:
total hits = times routed × hit rate per routing| Span | |
|---|---|
| Exposure (times routed) | 6.06× |
| Efficiency (hit rate per routing) | 8.45× |
| The two multiplied | 1.81× |
#The correct statement
At blind spot 0.5 we could not measure an effect of temporal structure on total hits, but that is absence of evidence. Once hits are split into "times routed × hit rate per routing," the former spans 6.06× and the latter 8.45× — the total is flat only because exposure and efficiency cancel in opposite directions. Attack capability differs by 8.45×.
#How serious this is
The re-verification also found three things:
| ANOVA | F = 1.27 |
| η² | 0.039 (main effect close to zero) |
| Permutation test | p = 0.115 |
| Statistical power | only 25% |
#Why this error happens so often
#Three places this rule applies
We later wrote it up as a general rule and used it in two other places:
| Where | Original sentence pattern | After breaking it apart |
|---|---|---|
| Temporal structure of pulses | "doesn't matter" | exposure 6.06×, efficiency 8.45× |
| Blind spot vs temporal structure | "the real variable is the blind spot" | interaction η² = 0.264, 4 times the main effect |
| Equal-budget comparison | "crude strategies are cheaper" | actual misbehavior 11.97 vs 4.87 |
The full breakdown of the second one
The full breakdown of the third one
#A similar trap: main effect vs interaction
The conclusion "the blind spot matters far more than the attacker's parameter choice" was based on these measurements:
| Span | |
|---|---|
| Blind-spot effect for pulse | 8.9× |
| Blind-spot effect for patient | 54× |
| Parameter grid (15 cells) | 1.81× |
So the conclusion was written as "the blind spot is the real variable." But the re-verification found that this ranking was itself wrong:
| Factor | η² |
|---|---|
| Main effect of temporal structure | 0.039 |
| Blind spot × temporal structure interaction | 0.264 |
The interaction is 4 times the main effect. Once an interaction is this large, the sentence pattern "the real variable is X" does not hold — neither factor can be used as an explanation on its own.
#Why we missed it
Honestly, three reasons:
- Our original metric was an aggregate. It had been since the first day of the design.
- The re-verification agent got the conclusions and the data, but not our thought process. So it was not contaminated in advance by the concept of "exposure."
- We had not listed "decompose" as a required step. Before that, the checklist only had "is there a denominator" and "is there a CI."
#Next
- The full version of this rule and the other three: Experimental methods and verdict values
- The full record of sending AI to refute ourselves: adversarial-recheck
