Evidence & reproduction · Reproduction / commands & hashes · 2026-09-24

Reproduction

The recipes, one by one: mechanism simulation, the real-model suite, zero-API offline replay, and receipt-chain verification. Most statistical conclusions can be rerun with zero-API recomputation scripts, without calling any model.

#Why most results can be rerun at zero cost

Every source record of the gain experiments is written to disk per task in rows.jsonl, with five fields: arm / task_id / delivered / hidden_pass / calls_used. This means the statistical conclusions don't need the model to be called again:

# Gate settlement (CONFORM, three task banks)
ops/gain/replay/conform_settle.py
ops/gain/replay/paired_ci.py          # paired McNemar + fixed-seed bootstrap
ops/gain/replay/pooled_paired_ci.py   # pooled banks

# Candidate cap and early stopping
ops/gain/replay/verify_hidden.py
ops/gain/replay/rows_visible_audit.py  # visible/hidden separation audit

# k curve
ops/gain/replay/off5_k_curve.py

# Loop attribution (turn-1 code re-extracted offline, zero model calls)
ops/gain/analyze_r460.py --rescore-turn1

# Multi-party execution
ops/gain/replay/peer_exec_sim.py
ops/gain/replay/peer_exec_real.py

#Mechanism simulation

python examples/pulse_suite.py --out <DIR>   # E17–E24, pulse attack
python examples/entrycost_suite.py --out <DIR>  # E1–E16, entry cost

Fully deterministic: the rng seed = f'{seed}:{config_digest}', so the same parameters always give the same results.

#Real-model suite

python examples/realmodel_suite.py --out <DIR> --tasks 60


Task bank	EvalPlus MBPP+ v0.2.0 (378 tasks, sha256 pinned)
Quality check	`compile_check(t.check)(answer)`, outside the arm
Pairing key	`task_id` = `sha256(prompt + tests)`, stable across arms
Statistics	McNemar exact test + fixed-seed bootstrap 95% CI + statistical power
Resume	Keyed by `(arm, i)`; an interruption doesn't mean starting over

#Receipt chain and offline viewer

vacant audit <name>                     # re-verify a logbook's signature chain (PASS/FAIL)
vacant verify-att att.json --answer "olleh"   # verify a portable signed attestation

We also host the offline single-file viewers on the site, so you can download them and use them directly:

File	Size	Contents
receipt_viewer.html	0.3 MB	Single-party execution
receipt_viewer_multiparty.html	4.5 MB	Multi-party execution

Both are copied verbatim from examples/ in the Vacant repo:

The receipts are embedded in the file
Zero external resources, zero network requests
Saved and opened directly via file://, they still run full chain verification

#Regression criteria

.venv/bin/python -m pytest tests/ -q

When	Result
Early	179 passed
2026-07-09 (after VM reset)	189 / 189
Independent-review fix branch	361 passed / 1 skipped (baseline 330)
Channel-separation revision	428 passed / 1 skipped (25 new criteria)

#The data index itself can be verified

實驗記錄/_index/files.jsonl is a 3,322-line manifest; each line records a path, type, line count and sha256. So the whole evidence directory (3,322 files, 1,550,582 lines, 325,111,980 bytes) can be independently verified for tampering.

Every entry in claims.json carries a "how to recompute" field that says which script to rerun, which field to read and which denominator to use.

#The site side

npm run build     # content/**/*.md → docs/, blog/, sitemap.xml, search index
npm run check     # validate content integrity only, write nothing
python3 -m http.server 8899   # local preview

The site's live data is produced by a script in the main Vacant repo:

cd ~/Documents/GitHub/Vacant
python examples/publish_status.py \
  --root ~/.vacant-mcp \
  --out ../vacant-docs-web/data/status.json \
  [--run-tests]
```}}

⌗This script deliberately outputs only quantities that can be independently recomputed
The rolling ledger hash, the chain-verification result, event counts, and cost. It does not output any delivered content, prompts, answers or private keys. Wherever the experiments have got to, rerun the script once and commit data/status.json, and the site is updated to that point.

## Evidence grades

The thesis's 121 references are filed by evidence grade:

| Grade | Definition | Count |
|---|---|---|
| **A full text in hand** | PDF in hand; quotations checked word for word | 81 |
| **A web archive** | Official documentation, specification or source-code page saved to disk with its hash registered | 13 |
| **B per abstract** | Only the abstract is in hand; statements based on it are marked "per abstract" in the text | 5 |
| **C bibliography only** | Full text not in the library; bibliographic data checked against DOI, Crossref or the publisher's page | 16 |
| **Project documents** | Unpublished documents of this project | 6 |

⊘Grade C references are not the basis for any numerical claim
Grade C is used only to describe a work's published research setup and main conclusions (for example the design of a task bank, the definition of a statistical method, or concepts such as the test-oracle problem and complete mediation). Before submission for review, the full texts should be obtained and their hashes registered.

## Next

- The content of the data index, claim by claim: {{doc:evidence-ledger|Evidence ledger}}
- The history of this project: {{doc:journey|Research journey}}