We sent AI to refute our own conclusions, and they succeeded

#Results

Of the six, three were refuted and two downgraded. The two positive conclusions that were kept have both been written into our analysis disciplines as rules.

#Why it is so easy to get wrong

Because when we wrote the report, we subconsciously treated two related factors as orthogonal dimensions. Three sets of examples:

The full breakdown of the first set

The second set: detection probability is a single product

The third set: equal budget has to check defected == BUDGET

#The hardest one to check: a conclusion held up by a degenerate endpoint

For "the blind spot matters far more than parameter choice," the re-verification agent found:

The blind spot and temporal structure have a strong interaction (two-factor decomposition η² = 0.264, 4 times the main effect of temporal structure), so the sentence pattern "the real variable is X" does not itself hold.

And the original numbers were held up almost entirely by one degenerate endpoint:

If a gauge degenerates at an endpoint, the numbers it gives there look the best and carry the least information. This was later written up as the second analysis discipline.

#A bug in a statistics tool was caught

During re-verification we also caught a problem we did not expect: the normal-approximation branch of wilcoxon_signed_rank_exact applied the tie correction twice.

This was not the researcher's mistake; it was the tool's. But the danger is this — it quietly turns non-significant into significant.

Also caught in the same batch: the bootstrap CI used Python's built-in hash() as its seed, so the same data run in different processes gave two numbers, 53,478 and 1,628.

#Three positive conclusions that were kept

Re-verification is not only destructive. Two conclusions survived the attack, and they are stronger than the original six:

#1. Detection probability is a single product

P(caught) = (1 − blind spot β) × sampling rate p × audit accuracy

The three factors are on the same axis. This is not a finding; it is a rule — but it is one we learned separately from three different mistakes.

Measured support: at blind spot 0.5, audit rate 0.1 → 1.0 cut the attacker's total hits by 82% (Wilcoxon p = 5.5e-6); audit 0.5 → 1.0 exactly offsets a 50% blind spot (paired difference 0.03, p = 0.78).

#2. Interaction η² = 0.264

The blind spot × temporal structure interaction is 4 times the main effect of temporal structure. This directly refutes the sentence pattern "the blind spot is the only real variable."

#The cost of doing it this way

To be honest about it:

The last row matters: re-verification is not omniscient. The agents gave six verdicts, and two items we marked "Not re-verified" ourselves, because the re-verification agents didn't touch them either.

#Next

• The full version of the four analysis disciplines: Experimental methods and verdict values
• Verdicts for all claims: Evidence ledger