Archive
What we did, the numbers we measured, the guesses that were refuted: it is all here. Every cell links to the source record, and you can rerun it yourself.
To go deeper, start here: the Docs explain how it works, the Lab notes are the full record of the things that were refuted, and the Evidence ledger lists the verdicts on 58 public claims.
The conclusions first
- It did not make AI trustworthy.
It turns "who did what, and who checked it" into a record that can be re-verified afterward.
- What improved delivery was "running," not "voting."
Running your tests was more useful than asking another AI to review.
- What we guessed wrong stays here too.
We originally expected reputation and mutual review to bring a gain. In the real-model experiments, it did not appear.
AI works faster and faster, but checking is as slow as ever. So on what grounds do we hand work over to it?
01
What we wanted to answer
- Why we say "accountable," not "trust"
The classic definition of trust includes "no need to monitor," and monitoring is exactly what Vacant does.
- The definition changed five times
Each time, the claim got smaller: from "a responsibility structure that agents adopt voluntarily" to "no delivery is accepted without a verifiable receipt."
I stand at the gate. When it says it's done, I first run your tests.
02
What we built
- A wrapper around your AI assistant
At the moment it finishes its work, it runs acceptance and signs a receipt.
- The acceptance gate
Acceptance is data, not code: it only runs code it generates itself.
- The signed receipt chain
Every entry is signed and locks the one before it.
- The five-level vocabulary
Observed, Decided, Withheld, Mediated, Attested: every receipt states clearly which level it reached.
The rules are fixed in advance and can't be changed after the run. If we guessed wrong, that stays too.
03
How we measured
- Criteria written in advance
Each experiment's criteria are frozen before it starts.
- You can rerun it yourself
How the tasks are rebuilt and how to reconcile the results: it runs on your computer too.
- An index of every run
Which runs are evidence, and which are only tests or were stopped midway.
Some guesses were right, some were refuted. They're all here.
04
What we found
All of these effectiveness numbers share one precondition: "The whole thing rests on 'requirements can be compiled into executable acceptance tests.' … Where a requirement can't be run, this mechanism has no free referee. It degrades into 'ask a model' — and that is exactly the thing that measured badly."整件事建立在「需求可以被編譯成可執行的驗收測資」。……需求跑不起來的場合，這個機制沒有免費的裁判，會退化成「問一個模型」，而那正是量出來很差的東西。(R440P §五-1)
- Acceptance gate + resampling
14–19 percentage points above trying only once (the same set of 120 tasks, repeated five times).
- It holds on a different task bank too
716 tasks pooled: still 6.84 percentage points higher.
- AI reviewers lose to majority vote
Having AIs review each other does not beat "generate five times, take the majority" at the same budget.
- "Revise from the failure message" was not measured as better than rewriting
Slightly higher all five times, and it passed correction zero times; with a larger model it did not appear either.
- Zero-config (v0.9.0): four batches, three of which did not detect a difference
It takes effect as soon as it is installed, and sends feedback back only once, within the same session. The fourth batch did measure a difference; on a different inference engine it did not replicate. It does not contradict the gate above: zero-config does not own generation.
- The receipts verify
194 chains, all 9,841 entries pass; the lying keys were named in 273/273 cases.
- Bad actors don't get work (mechanism simulation)
Bad actors who were not caught got only 0.57%–1.84% of the work that honest participants got.
These aren't done yet, or haven't been fully measured. We'll say so up front.
05
What it still can't do
- Refusal doesn't stop files from leaving
Under the default settings, of 391 cells, 59 were refused-delivery cells, and in 7 of those the files were copied out.
- "Every single call goes through"
That requires removing every other path; right now only a run locked inside the wall achieves it.
- Retrying doesn't rescue them
The exhibit's 54 cells: 30 stopped at the gate, 0 rescued by retrying.
- We haven't measured "people" yet
We have not recruited users, so we don't yet know whether this makes people more willing to hand work to AI.
Next, we want to answer the question we started with.
06
What's next
- Vary the quality of the acceptance tests
How the gate's effect changes when the tests are written well or badly.
- Make omissions impossible to hide
Publish the receipt chain heads, so that truncation and omitted entries become visible too.
- Find real users
Measure it: with receipts, do people rely on AI more willingly, and with more of a basis?
Source code and every record
The code, the record of every run, and every criterion are public on GitHub.
Go deeper
This page is here to help you understand. Once you do, the three places below let you look things up.
- Docs The seven-layer architecture, the gate rules, the receipts' five-level vocabulary, how to rerun. 27 pages.
- Lab notes 32 posts, each one an experiment that actually ran or a claim we refuted ourselves, with full numbers and denominators.
- Evidence ledger 58 things we have said publicly, and whether each one still holds as of today: Held / Refuted / Overstated / No discernible difference / Same sign, unresolved / Not re-verified.
