Lab notes
What we measured,
and what we measured that didn't hold.
Vacant's lab notes, 32 posts: each one an experiment that really ran or a claim that was refuted, with full numbers, denominators and verdicts. The raw record of the defense axis, the gain axis and multi-party execution.
No notes carry this verdict.
#Method4 posts
How to turn a statement into a number that can be recomputed — and how it later got refuted.
-
Six verdict values: how to say "not measured" without saying "no difference"
Held, Refuted, Overstated, No discernible difference, Same sign, unresolved, Not re-verified — each of these six terms has a strict definition. We keep them apart so that "I didn't detect it" is never read as "I tested it, and there's nothing."
-
Preregistration: why we write down the criteria before running the experimentHeld
Criteria, statistical methods and the four state rules are all frozen and timestamped before the run starts. This post explains how we preregister, and why "adding a criterion after the fact" doesn't count in this research.
-
We sent AI to refute our own conclusions, and they succeededOverstated
In early 2026-08, we handed six headline conclusions about the pulse attack to independent agents, with one instruction only: "prove it wrong." The result: three refuted, two downgraded, and only two held up.
-
Six things we mismeasured
Reading the wrong field, network drops counted as reviewer results, an rng seed that degenerates at the limit, treating `defected <= BUDGET` as an equal budget, pretty numbers at an endpoint, and a rule switched off by default — six real measurement traps and their lessons.
#Defense axis9 posts
The cost of getting in, what it saves, and the attack model we refuted ourselves.
-
An attacker that only slanders and never breaks a rule got 95.4% of the workNot re-verified
All three layers of anti-vote-stuffing defense were in place, yet a collusion group that never delivered anything bad and only reviewed others took almost all of the routing — and its score is 0. The harm was not underestimated; it was measured as zero.
-
Why "permanent exclusion" is really the product of two reasonable patchesOverstated
Measured: of 368 penalized seeds, 0 returned to routing. We said "caught once = removed permanently" — but the real mechanism is not a clock; it is that slash both halves the mean and doubles the observation count, and the observation count is the exponential coefficient of the return time.
-
How "temporal structure doesn't matter" went wrong: breaking the aggregate apartOverstated
We measured total hits across 15 cells of attack strategies that differed by only 1.81×, then wrote "temporal structure barely affects the result." Broken apart: exposure spans 6.06× and efficiency spans 8.45× — attack capability differs by 8.45×.
-
Detection probability is one product, not three dimensionsRefuted
We originally claimed that "raising the audit rate cannot rescue cases with a blind spot." Measured: at blind spot 0.5, raising the audit rate from 0.1 to 1.0 cut the attacker's total hits by 82%, p = 5.5e-6. The three factors are terms in the same product.
-
The "equal budget" trap: our ranking flipped completelyRefuted
The criterion was written as defected <= BUDGET, an upper bound that always holds, so it could not catch "this arm was never bound at all." As a result, whitewash misbehaved 2.5 times as much and we read it as "the strategy is cheaper." After switching to == and verifying, the ranking flipped.
-
Pulse attack: not new, and even our headline number was already old newsNot re-verified
Discharge B, charge R, repeat. This model had already been done in 2005 (Srivatsa et al.'s strategic oscillation); on 2026-08-06 we withdrew the "novel attack" claim.
-
The trust on/off experiment on a real model: +8.3%, but statistically nothing happenedNot re-verified
60 paired tasks; switching the trust layer on raised the pass rate from 51.67% to 60.00% — but the 95% interval spans 0, McNemar p=0.332, and statistical power is only 0.163. The same data holds another, more meaningful result.
-
Setting the stake too low let attackers earn a 235% returnNot re-verified
The most counterintuitive result of the 14 mechanism simulations: free entry blocks completely, yet a stake of 2 gave the attacker a return per unit of cost of 2.354. We deleted two defense designs (an exogenous entry fee and endorser liability) entirely.
-
The only truly tight constraint in the whole defense system: reviewer accuracyNot re-verified
We swept the stake, probation length, ecosystem size, attacker share, and the correlation of reviewer errors — all of them are slack. The only cell where attacking starts to pay is when reviewer accuracy drops to 0.
#Gain axis13 posts
Which mechanism actually makes deliveries better — including four batches where no difference was detected.
-
Zero-config, four batches: three measured no difference, and the fourth vanished when we switched enginesNot re-verified
Zero-config was measured on four batches: DABstep, 711 local cells, 920 Colab tasks and u274. Paid batch: no measured difference for either model; local batch p=0.021 but concentrated in 6 tasks; 920 tasks and u274: no measured difference; v3.7 later measured p=0.0028, but did not replicate on GGUF.
-
The gate cuts false delivery by about 40%, not in half
An experiment with 836 tasks, 43 blocks and 0 voided. The thesis's "halved" was wrong — because the two rates used different denominators. This correction changes the gate's effect from "in half" to "roughly 40%."
-
This is a success-rate gain, not a token-efficiency gainNot re-verified
Recomputed call by call, 197 cells, zero differences. H-MIX uses 2.23–2.78× the tokens of a single draw, and in four of the five replications it was more expensive per correct delivery than the gate. "The same ceiling" is not "the same actual cost."
-
After five replications: the headline claim lost to its own stability ruleSame sign, unresolved
R460 measured the revision loop at 13.33 percentage points above resampling with a fresh attempt, Holm p = .0112. After five same-task replications, all five were not significant, 0/5 passed correction, and the status was downgraded to "Same sign, unresolved."
-
The only thing that held stably across five replications: the loop beats a single draw by 17 to 29 percentage pointsHeld
All 15 blocks pass their own Holm correction, with differences from +17.5 to +29.2 percentage points, across five same-task replications and four mutually exclusive task sets. This is the hardest result in the whole project.
-
The gain comes from the loop, not from the first-round promptHeld
We pulled out the complete first-round responses and re-scored them offline: the loop contributes +19.17 / +17.50 percentage points, the first-round prompt only +6.67 / +8.33. 21 tasks were rescued by the loop.
-
The MDE is 4.31 percentage points, but the task bank has only 378 tasksSame sign, unresolved
For the comparison of the gate against majority vote, we can calculate that 491 pairs are needed for 80% statistical power, but the task bank actually has only 378 tasks. This "Same sign, unresolved" is not unfinished work; it is physically unmeasurable.
-
The harder the task, the more the acceptance gate is worthHeld
The early-stop gate delivers 7.8 percentage points more on MBPP+, and 20.9 percentage points more on hard LiveCodeBench problems. And the "leak rate" drops from 44 to 11 — refusing delivery is worth the most on hard tasks.
-
The birth of the judging gate: execution instead of opinion, receipts instead of votesHeld
We deleted the AI review committee and replaced it with "run the customer's acceptance tests, deliver the first candidate that passes, refuse delivery if none passes." On three task banks it delivered 4.58 to 19.17 percentage points more, at only 1.5 calls per task.
-
Only after our own experiments excluded two explanations did the mechanism have room to changeRefuted
"The model is too strong" and "the tasks are too easy" were two explanations proposed by humans. Two experiments excluded both, and only then was the reviewer mechanism forced into a redesign — and the direction was to change the referee, not the reviewer model.
-
The reviewers track task difficulty, not whether the answer is rightHeld
The same reviewer layer judges almost everything as pass on MBPP+ (0.305 error rate) and almost everything as fail on LiveCodeBench hard tasks (0.46 error rate). It follows a global property, not this particular answer.
-
"The hidden tests are lossless" is empty by constructionHeld
A criterion R440P set in advance was refuted nine days later: in LCB, 91/91 hidden test sets are supersets of the visible tests, so "would pass hidden but fails visible" cannot happen by construction. This criterion had zero statistical power.
-
Our AI review committee lost to "generate five times, take the majority"Refuted
Same 5 calls, same set of tasks: a committee of three AI reviewers plus one revision lost to "generate five times and take a majority vote." The point estimate of the difference was −1.98 percentage points — and after we fixed an unrelated bug, it became +2.97.
#Records & multi-party execution4 posts
Signature chains, twins, and the places where the measuring tools themselves measured wrong.
-
The measuring tool got it wrong seven times, and every error leaned against itselfNot re-verified
The A/B harness for the interactive TUI had seven measurement errors, plus two more that exist only in code comments. Every one was caught on a real pi, and every one pushed the numbers to look better than they really were — 8 fabricated results, and 5 cells whose prompts were never sent.
-
An attacker re-mints the whole chain with their own key, and the receipt viewer still says it passesHeld
194 receipt chains, 9,841 entries, all verify. But the offline single-file viewer has a structural limit: if the whole chain is re-minted once, it still passes. This is not a vulnerability; it is the definitional boundary of a signature.
-
The lying key was named all 273 times; honest keys were falsely accused 0 timesHeld
k=3, 1 controlled lying key, 5,519 testimonies. Lying cells named 273/273, honest keys falsely accused 0/1,840, contradictory-signature cells ruled correctly 58/58. How strict these numbers are depends on whether you know the mathematical upper bound ⌊(k−1)/2⌋.
-
After turning the acceptance suite from code into data, the delivery-rate drop went from 75% to 2.7%Held
An attacker wrote a stateful variant of the gauge: the gauge scored perfectly in both directions, yet the delivery rate dropped 75 percentage points. The fix was to leave the suite with only three things: a dialect, an entry-point name, and a literal list of input — expected value pairs.
#Project history2 posts
From definition to verdict: how this line of work got to where it is now.
-
The definition changed five times: from "a responsibility structure" to "no delivery without a receipt is accepted"
Over nine months, "what Vacant is" was rewritten five times, and each time the claim got smaller: from "a responsibility structure agents adopt voluntarily" to "a delivery without a verifiable receipt is not accepted." The shrinking process itself is this project's contribution.
-
KS-1: On a frozen model, "writing the consequences to look real" and "making the consequences real" are the same thingNot re-verified
On 2026-06-28, the two main validation designs were killed by adversarial workflows of 17 and 10 agents. The fact that killed them later became the methodological core of the whole project.
All the verdicts are here
The label on a note is the verdict of that post's main claim. The full verdicts of every public claim are in the evidence ledger; methods and reading rules are in Experimental methods and verdict values.
