Experimental methods and verdict values
Four families of experimental method, the six states a verdict can take, the four rules declared in advance, and the four analysis disciplines that were written only after re-verification. This page is the reading precondition for every number on the site.
#Two axes
From 2026-08-17 on, the project formally split all of its experiments into two axes. This was the most important structural change in the whole study — before it, every experiment we had accumulated was asking "was the bad stuff blocked?", and not one was asking "with this layer added, does what gets delivered fit the requirement better?"
| Axis | Question | Method |
|---|---|---|
| Defense axis | When there are participants who deliberately deliver wrong work, how much can these rules block? | Mainly mechanism simulation |
| Gain axis | When requirements can already be compiled into executable acceptance tests, how much does the delivery success rate rise, and at what token cost? | Paired experiments on real models |
#Four method families
#1. Mechanism simulation (E1–E9, E12–E24, A1–B5, M1–M5, S1–S3)
What it answers: under these rules, what a given attack strategy or mechanism parameter is worth.
What is real: routing (the real Registry.route, UCB + probation cap 0.55 + a probation quota every 10 rounds), auditing (the real Auditor, deterministic sampling via sha256(seed:task_id)), reputation (the real Reputation, five-dimensional Beta, multiplicative slash, per-stream half-life of 200 events), signatures (real Ed25519 and hash-chain).
What is simulated: only "whether this one delivery is good or bad" — that is the experimental treatment in the first place, not the mechanism.
What is not controlled:
- The attacker strategy space is hard-coded to three or four strategies; real attackers may be stronger.
- A fixed ecosystem size of one attacker + 5 honest residents.
- The true value of the blind spot is unknown (no external anchor).
#2. Real-model experiments
What it answers: does the whole pipeline run on real models, and is there a measurable difference between switching the trust layer on and off.
Steps: sample from EvalPlus MBPP+ v0.2.0 (378 tasks, sha256 pinned) → each arm runs the same batch of tasks (the three arms' prompt templates are identical word for word, KS-1) → after delivery, quality is judged independently by compile_check(t.check)(answer), outside the arm → pair by task_id (sha256(prompt+tests), stable across arms) → McNemar exact test + fixed-seed bootstrap 95% CI + statistical power.
#3. Regression criteria (pytest)
What it answers: whether the mechanism itself has regressed.
Write a criterion that fails first, then change the mechanism (red first, then green). Each criterion's description records "the wrong number at the time," so that regression cannot happen silently.
#4. Website audit
What it answers: whether the public website is readable, whether interactions are correct, and whether there is horizontal overflow.
- Contrast: an iframe probe walks every visible element, takes the computed style, composites semi-transparent foregrounds onto the effective background, and computes the WCAG ratio (text 4.5:1, large text and UI borders 3:1).
- Function: JS-driven clicks walk through the interaction flows, checking the DOM state and whether the fingerprint is really recomputed.
- Responsive: measure
scrollWidthvsclientWidthin a fixed-width iframe.
#Verdict values: six states
This is the most important set of words on the site. Every public claim must fall into one of these cells.
- HeldPassed the criterion set in advance (preregistered, run, then checked against the answer)
- RefutedThe claim is wrong; the correct statement is listed under "Corrected"
- OverstatedThe core holds but was stated too strongly, or the original evidence or mechanism explanation has a problem
- No discernible differenceMeasured, and no discernible effect — that is itself the answer, not an unfinished test
- Same sign, unresolvedThe preregistered rule triggered none of its states. Not measured — not the same as no difference
- Not re-verifiedNot yet sent for re-verification — not the same as passing
#Four rules declared in advance
- Criteria are frozen before the run starts. Each experiment's criteria, statistical method, and four-state rule are written in its decision document, with a timestamp.
- Refused delivery counts as failure, and the denominator is all tasks. Otherwise "refuse everything" would become 100% success.
- Infrastructure failures do not count as model failures. A sandbox that won't start is an environment problem, not a model-capability problem.
- Not measured is not the same as measured zero. Every coverage field is three-state (true / false / not measured).
#Four analysis disciplines
These four were added only after adversarial re-verification. Each one corresponds to a conclusion we would otherwise have published wrong.
#1. Decompose before concluding on an aggregate
Before concluding from an aggregate that "dimension X doesn't matter," decompose it first. When it can't be decomposed, you may only write "not measurable at this operating point," not "doesn't matter."
Source: E24's "temporal structure differs by only 1.81×." aggregate-decomposition
#2. Don't measure effect sizes at degenerate endpoints
The numbers at the endpoints are the prettiest, and usually the least informative.
At blindspot = 1.0, 30 seeds collapse into the same trajectory (sd = 0, effective n = 1); at audit_rate = 1 with audit_accuracy = 1, caught == bad is an identity hard-coded in the program.
#3. Detection probability is a single product
P(caught) = (1 − blind spot β) × sampling rate p × audit accuracy. Don't treat the factors of one product as orthogonal dimensions.
Source: "a bigger audit can't rescue the blind spot." blindspot-is-a-product
#4. Equal-budget comparisons must check defected == BUDGET
The upper bound always holds, so it can never catch "this arm was never bound at all."
Source: E19 misread "2.5× more bad actions" as "better timing." equal-budget-must-bind
#Statistical self-corrections
| Event | How it was found |
|---|---|
The normal-approximation branch of wilcoxon_signed_rank_exact applied the tie correction twice | At n=30 with 20 positive and 10 negative, the program said p=0.0288 (significant); the exact value is 0.0987 (not significant) |
The bootstrap CI used hash() as its seed, so it could not be reproduced across processes | The same data produced 53,478 and 1,628 |
SimConfig.digest() was both the RNG seed and a hash of every field | "Adding one optional parameter" would silently move every number in E1–E16. Fix: fields at default values are left out of the digest, and three config digests are pinned to bit-for-bit reproduction |
| Hidden-test audit v1 produced 90 false positives | Changed the criterion to "who wrote this text"; v2 has zero violations |
An indentation bug in codebench.py | 77.2% of the 378 contracts have ≥2 asserts, which caused FAIL claims to be wrongly dropped. Recomputing 99 entries: 38 (38.4%) were real counter-examples eaten by the bug |
#Three layers of honest boundary
| Layer | Content |
|---|---|
| Mechanism layer | Mechanism simulation cannot establish safety; it can only give a lower bound on the upper bound of attack cost. |
| Task layer | The gain experiments are limited to Python code generation, with tasks from local subsets of three public task banks. The main experiments use a single quantized 12B model and local inference hosts. |
| Human layer | Never measured at all. We did not recruit users, so we don't know whether people become more willing to hand work to AI because of this. |
#Next
- Defense axis
- Gain axis
- Records & multi-party execution
- Verdicts on every claim: Evidence ledger
