Evidence & reproduction · Evidence / 58 verdicts · 2026-10-07

Evidence ledger

58 public claims, each with a verdict, supporting figures and a way to recompute them. The refuted ones, the overstated ones, the ones where no difference was measured — all stay on the table. A system that claims to be accountable has to be accountable to itself first.

#Where this table comes from

A single data source

Every cell of this ledger is computed on the spot from two JSON files under content/data/:

File	Claims	Source	Marked
`claims.json`	34	`實驗記錄/_index/claims.json` in the project folder (created 2026-09-14), imported with `npm run import-claims`	—
`claims-repo.json`	24	Transcribed one by one by the site author after reading the verdicts and conclusion files under `decisions/` in the `cosmopig/Vacant` repo (as of 2026-10-03, v0.9.0 / zero-config v3.7)	repo verdict

To change a number, change the JSON once, and the docs pages, lab notes, search index and this table all follow. Nothing needs to be retyped.

Why there are two files

claims.json is overwritten in full by import-claims, so hand-transcribed claims can't go into it — the next import would wipe them.

claims-repo.json only exists from 2026-09-25 on: after that date the Vacant repo had 193 commits and finished two new lines, zero-config v3→v3.7 and accountable trace, but produced no new claims.json — those verdicts are written in decisions/*.md and the conclusion files.

So these 24 claims sit at a different source level from the other 34, and the ledger marks them "repo verdict". Their numbers come from that repo's analysis scripts and their verdicts were written by that repo's adjudicator, but they have not yet gone through the project's official index.

In other words: for now, these 24 were transcribed and assigned values by the site author on the repo's behalf. If you want to treat them as data as authoritative as the other 34, the next step is to export the cosmopig/Vacant verdicts into a claims.json.

Each claim carries four things:

Original wording — what was said publicly at the time.
Supporting figures — the numbers behind it, with denominators.
Verdict value — one of six states.
Corrected — if the verdict is "Refuted" or "Overstated", what the correct statement is.

HeldPassed the criterion set in advance (preregistered, run, then checked against the answer)
RefutedThe claim is wrong; the correct statement is listed under "Corrected"
OverstatedThe core holds but was stated too strongly, or the original evidence or mechanism explanation has a problem
No discernible differenceMeasured, and no discernible effect — that is itself the answer, not an unfinished test
Same sign, unresolvedThe preregistered rule triggered none of its states. Not measured — not the same as no difference
Not re-verifiedNot yet sent for re-verification — not the same as passing

#Counts

Verdicts of all 58 public claims

23145556

Held23/58 Refuted14/58 Overstated5/58 No discernible difference5/58 Same sign, unresolved5/58 Not re-verified6/58

#The full ledger

Click any row to expand its supporting figures, source-record path and how to recompute it.

Axis

Verdict

Showing 58/58 claims

ID	Claim	As stated	Verdict
`pulse.definition`	Definition of the pulse attack	Pulse attack = discharge B consecutive bad actions → accumulate R clean deliveries → repeat; the three existing strategies are special cases of it	Not re-verified
`pulse.timing_minor` Note →	Time structure barely affects total hits	The attack's time structure barely affects the total number of hits: the 15-cell (burst,recover) grid falls within 2.83–5.13 (1.81×)	Overstated
`pulse.blindspot_dominates`	The blind spot matters more than the attacker's parameters	The blind spot matters far more than the attacker's choice of parameters: pulse 8.9×, patient 54×, against 1.81× across the parameter grid	Overstated
`pulse.crossover` Note →	Once the blind spot exceeds 0.5–0.75, pulse does worse than sustained misbehavior	Once the blind spot exceeds about 0.5–0.75, pulsing actually does worse than plain sustained misbehavior (3.8× apart with the blind spot at its maximum)	Refuted
`pulse.audit_cannot_close_blindspot` Note →	Raising the audit rate can't rescue a situation with a blind spot	Raising the audit rate rescues the zero-blind-spot case (100% audit = complete lockout) but not cases with a blind spot (still 0.97 hits, only 57% fully blocked)	Refuted
`pulse.starvation` Note →	Caught once = removed permanently	Caught once = removed permanently. Of the 16 seeds that were caught, all 16 were never routed again afterward	Overstated
`pulse.reaction_lag`	Even with zero blind spot, 2.53 get through and reaction takes 28 rounds	Even with zero blind spot, one wave still lets 2.53 through, and it takes 28 rounds on average to react	Not re-verified
`pulse.crude_wins_under_blindspot` Note →	Under an equal budget, the crude strategy beats pulse	Under an equal budget (12 each) + blind spot 0.5, whitewash at 8.17 actually beats pulse at 4.23	Refuted
`entry.fee_backfires`	An entry fee set too low actually helps attackers outlast probation	An entry fee set too low actually helps attackers outlast the probation period: stake=0 blocks everything, stake=2 yields 5.00 hits and an ROI of 2.35	Not re-verified
`entry.reviewer_accuracy_binds`	The real constraint is reviewer accuracy	The real constraint is reviewer accuracy: at acc=0, 16.65 hits and an ROI of 1.04 (misbehaving starts to pay)	Not re-verified
`realmodel.e10_paired_null` Note →	E10 trust-switch paired experiment: no significant difference	E10 trust switch, N=60 paired: on 36/60 vs off 31/60, difference +8.3%, 95% CI [-5.0%, 21.7%] spans 0, McNemar p=0.332, not significant	Not re-verified
`realmodel.measurement_error` Note →	The false result of the first measurement, which read the wrong field	The first measurement was wrong: it read audit.passed as the quality criterion, while the off arm is by design not audited, producing a false result of 0/11 vs 5/10	Not re-verified
`gain.signal_exists`	The gauge has signal (OFF failure rate falls in the window)	The gauge has signal: without the trust layer (OFF) the failure rate is 26.44%, inside the 20–60% window set in advance	Held
`gain.arms_differ` Note →	The three arms differ discernibly in delivery quality	The three arms (OFF/ON/OFF5) differ discernibly in delivery quality	No discernible difference
`gain.equal_budget_on_beats_off5` Note →	Under an equal budget, Vacant beats five-try majority vote	Under an equal budget, Vacant (ON) beats running the same task five times and taking the majority vote (OFF5)	Refuted
`gain.mechanism` Note →	Why it can't win: reviewer votes are near-constant, and revision makes no net correction	Why it can't win: the reviewer's vote is almost a constant function, and revision produces almost no net correction	Held
`gain.ceiling_hypothesis` Note →	A weaker model would reveal the gain (ceiling hypothesis H-A)	The 35b worker is too strong and causes a ceiling effect; switch to a single 12b model and Vacant will show a gain	Refuted
`gain.hard_bench_hypothesis` Note →	A harder task bank would reveal the gain (difficulty hypothesis H-B)	MBPP+ is too easy for modern models; switch to date-stamped contest problems (LiveCodeBench) and the gain will show	Refuted
`gain.reviewer_tracks_difficulty` Note →	The reviewer tracks task difficulty, not whether the answer is right	The reviewer tracks task difficulty, a global property, not whether an individual answer is right	Held
`gain.clean_dataset`	The decisive run is the first clean complete dataset	The decisive run is the first clean complete dataset: no known bug contamination, void rate within the window	Overstated
`gain.conform_early_stop_beats_single` Note →	The early-stop gate delivers more than taking a single sample	"Run the customer's own acceptance tests, deliver the first one that passes, refuse delivery if none passes" (the CONFORM early-stop gate) delivers more than "take a single sample and accept it" (OFF)	Held
`gain.gate_rule_beats_majority_vote_same_candidates` Note →	Same set of candidates: the gate rule beats a five-way vote	With the same 5 candidates and the same 5 calls, the gate rule (deliver the first one that passes, deliver nothing if none passes) delivers more than a five-way majority vote	Held
`gain.conform_vs_off5_unresolved` Note →	The gate beats majority vote on delivery accuracy	The gate (CONFORM) beats running the same task five times and taking the majority vote (OFF5) on delivery accuracy	Same sign, unresolved
`gain.lossless_visible_filter` Note →	The visible filter is lossless	The visible filter is lossless — no candidate appeared that "would pass the hidden tests but failed the visible acceptance check"	Held
`gain.off5_helps_on_hard_only` Note →	Five times the budget buys something only on hard tasks	Self-consistency at five times the budget (OFF5 vs OFF) buys something only on hard tasks	Held
`peerexec.mutual_execution_below_threshold`	Below the majority threshold, mutual execution can name the liar	Below the majority threshold (corrupted executors ≤ ⌊(k−1)/2⌋), the deliveries of mutual execution without mutual review are bit-for-bit identical to the uncorrupted baseline; the liar is named and honest participants are not falsely accused	Held
`peerexec.suite_fixed_point` Note →	Mutual execution has no defense at all against a corrupted acceptance suite	Mutual execution without mutual review has no defense at all against "the acceptance suite itself being corrupted" — this is a fixed point of the mechanism; R451→R452 shrank it to a residual but did not eliminate it	Held
`peerexec.majority_bound`	Naming has a mathematical upper bound ⌊(k−1)/2⌋	Naming has a mathematical upper bound: majority vote tolerates at most ⌊(k−1)/2⌋ corrupted executors; past a majority it flips	Held
`harness.hmix_loop_beats_resample_same_budget` Note →	The revision loop delivers more than resampling and spends no more tokens	Spending five calls on "run the customer's acceptance tests, paste the failure output back verbatim, let the same worker revise" (H-MIX) delivers more than spending them on resampling with a fresh attempt (CONFORM), and spends no more tokens	Same sign, unresolved
`harness.loop_beats_single_shot_replicated` Note →	The feedback loop delivers more than taking a single sample	Spending five calls on a feedback loop (H-MIX/H-PI/H-OC) delivers more than taking a single sample and accepting it (OFF) — this one holds up across task banks and across replications	Held
`harness.gain_comes_from_executable_gate_and_resample` Note →	The bulk of the gain is gate + resampling; the loop is a small increment	The bulk of the gain is "the executable acceptance gate + resampling"; the feedback loop is only a small increment stacked on top of it	Held
`harness.majority_vote_loses_to_gate` Note →	With the same five calls, voting loses to the gate and costs more than twice as much	With the same five calls, a five-way majority vote (OFF5) loses to "run the acceptance tests, deliver the first one that passes" (CONFORM), and costs more than twice as much	Same sign, unresolved
`harness.hpi_hoc_vs_resample_unresolved`	The other two loop arms also beat resampling	The pi-style raw feedback loop (H-PI) and plan + diagnose + self-test (H-OC) also beat resampling with a fresh attempt	Same sign, unresolved
`harness.gain_is_the_loop_not_the_prompt` Note →	The gain comes almost entirely from the loop, not the prompt wording	H-MIX's gain comes almost entirely from the loop of "paste the failure output back verbatim and let it revise", not from the wording of the first-round prompt	Held
`zero.dabstep_paid_no_difference` Note →	Zero-config on DABstep: no difference measured for either model	On the paid DABstep main batch, agents with zero-config Vacant installed were not measured to have a higher correct-answer rate than agents without it: gemma-4-26b 50/77 vs 44/77 (p=0.21), qwen3.8-27b 68/77 vs 64/77 (p=0.29).	No discernible difference
`zero.dabstep_paid_rejects_all_false` Note →	All 8 send-backs in that round were false positives	None of the 8 send-backs zero-config triggered in the paid DABstep batch would happen after the fix — not a single send-back across 158 runs; that is, once fixed, C equals A: it neither helps nor hurts.	Refuted
`zero.local711_p021_thin` Note →	Local 711-cell batch: p=0.021, but the difference is concentrated in 6 tasks	On the local 711-cell batch, zero-config v3's preregistered primary test against no install gave p=0.0207, but the author states plainly that the significance is thin: with all 6 tasks removed, the mean difference is 0.009, p=0.80; flip the results of two runs and p goes above 0.05.	Overstated
`zero.v3_3_never_fired` Note →	v3-3 "with one round left, send back only for missing files" never fired once	The v3-3 mechanism did not fire even once in 150 local pre-handoff checks; it made no observable contribution.	Refuted
`zero.colab920_no_difference` Note →	Zero-config on the Colab 920-task coding banks: no difference measured	On 920 tasks from five coding task banks on Colab, agents with v3.6.1 installed were not measured to have a higher correct-answer rate than agents without it: 748/920 (81.3%) vs 758/920 (82.4%), p=0.387.	No discernible difference
`zero.colab920_rejects_96pct_false` Note →	96% of the send-backs in the 920-task batch were wrongful	96% of v3.6.1's send-backs on coding tasks were wrongful: 455 of 472 send-backs happened because it recognized only pytest and python -m unittest, not the sh run_tests.sh the agent had actually run.	Refuted
`zero.u274_no_difference` Note →	u274 preregistered batch: 9 vs 10, no difference measured	On the first 100 of 274 tasks in seed order (all hard, never used before), 9/100 were correct without the install and 10/100 with v3.6.1 installed; McNemar exact two-sided p=1.0, no difference measured.	No discernible difference
`zero.u274_missing_file_returns_all_wrong` Note →	u274: 31 runs got a missing-file send-back, and not one was correct afterward	In the u274 batch the missing-file send-back fired in 31 runs (37 times in total), and not one of those 31 runs ended up correct — 17 wrong, 14 still not delivered. The extra answer files C delivered were all wrong, and C also hit the time limit 18 more times.	Refuted
`zero.v37_fixes_three_defects` Note →	v3.7 fixes three defects; send-backs drop from 309 to 29	v3.7 fixes three defects: it recognizes any test script (not only pytest), it also recognizes deliverables from the instruction file a person asks the agent to follow, and a round that ends in a model error is never sent back. On the Colab 920-task batch, the runs sent back at the first check dropped from 309 to 29.	Held
`zero.v37_colab_p0028` Note →	v3.7 on Colab: +11.3 pp measured (p=0.0028)	On Colab's DABstep 711-cell batch, v3.7 was measured higher than no install: A 97/231 (42.0%), C37 123/231 (53.2%), C37R 139/231 (60.2%); primary test C37−A mean difference +11.3 pp, p=0.0028.	Held
`zero.v37_gguf_not_replicated` Note →	The same v3.7 did not replicate on a GGUF backend	On the vacant-dev replication (LM Studio GGUF, gemma-4-12b, 15-round cap), the same v3.7's primary test was not significant: C37−A −0.4 pp, p=0.84. The Colab p=0.0028 did not replicate.	Refuted
`zero.task3_no_difference` Note →	Three task-oriented banks, 94 tasks: no difference measured	On 94 tasks from three task-oriented banks, on Colab with no round cap, v3.7 measured no difference: A 136/188 (72.3%) vs C37 141/188 (75.0%), Wilcoxon p=0.36.	No discernible difference
`zero.reminder_default_off` Note →	The round-budget reminder is off by default, because the extra answers it produces are wrong more often than right	From v3.6 on, the round-budget reminder is off by default; whoever wants it adds --budget-reminder. The reason: the reminder raised the correct-answer rate from 51.9% to 59.3%, but the extra answers it produced were wrong more often than right.	Held
`zero.not_contradicts_past` Note →	Zero-config's negative results do not contradict past data	The negative results of the three zero-config batches do not contradict the gains measured before, because exactly the same setting was never measured before. Everything that showed a gain before was a mechanism in which Vacant itself owns generation.	Held
`trace.two_layer_attribution`	Trace has two layers: only the fact layer can carry consequences	Every trace conclusion has two layers: the fact layer (a rerun flips, success claimed but the command failed, a prohibition violated) may carry reputation consequences; the inference layer ("step k caused it", "it read bad input") only goes into the report.	Held
`trace.lfake_attribution_correct`	Trace end-to-end attribution correct 33/33, 25/25, 16/16	Four real agents ran eight planted-error scenarios end to end, and attribution was correct (status/category/level/step pointed to/source all right) in 33/33; the same scenarios through vacant do were 25/25; the real interactive TUI was 16/16.	Held
`trace.feedback_never_names_actor`	Feedback to the agent never contains an actor identity	In all three end-to-end tables, no actor identity appeared in the feedback given to the agent: native 0/33, vacant do 0/25, TUI 0/16. The targets of blame are steps and actors, but the feedback is shown only to people.	Held
`trace.review47_and_boundaries`	The trace line's 47 adversarial review items and seven honest boundaries	The trace line went through 47 adversarial review items under five lenses: 46 upheld, 1 judged an existing honest boundary but tightened anyway; it lists seven honest boundaries, including three sentences that are explicitly forbidden.	Held
`trace.scale_perf`	The first scan of 40,000 files takes 5.7 seconds; each hook after that 557 milliseconds	Trace scanning the whole workspace (incrementally) on a 4-core local machine stays within the hook's 30-second cap: first scan of 40,000 files 5.7 s, each hook after that p50 557 ms, p95 597 ms, Stop end to end 1.9 s.	Held
`tui.seven_measurement_errors`	Seven measurement errors in the interactive A/B harness, every one making the numbers look better than they were	The interactive TUI A/B harness had seven measurement errors, each one caught on real pi 0.87.1, and every one of them pointed in the direction of making the numbers look better than they actually were.	Held
`pi.possess_three_defects`	The pi extension lacks three things; installing it is the same as not installing it	The pi interactive-mode extension has three defects, two of which make it completely unusable: install reports success but every call sinks, and the model id does not match what is actually sent.	Refuted
`opencode.w2_silent_noop`	opencode has no stop event: a full 37-step case record, zero verdicts	opencode's plugin did not register a stop event, so the end-of-round check was unreachable: one measured cell recorded a full signed case record of 37 steps, zero verdicts, and no delivery.md. The author says this is more dangerous than doing nothing, because the chain looks correct.	Refuted
`wave1.unresolved`	Native platform wave 1: Same sign, unresolved — not a negative result	Native platform wave 1 (20 tasks × three arms): the paired McNemar had only 2 discordant pairs, p=0.50, which the author recorded as unresolved — neither a negative nor a positive result.	Same sign, unresolved
`intake.five_silent_defects`	End-to-end runs of the universal intake found five defects that "make you think it works when it doesn't"	End-to-end runs of the universal intake found five defects, two of which unit tests cannot catch: Codex refuses to start, and OpenCode's additive layer silently drops the user's global rules from the prompt.	Refuted

Claim by claim

pulse.definitionDefinition of the pulse attackNot re-verifiedDefense axis

Pulse attack = discharge B consecutive bad actions → accumulate R clean deliveries → repeat; the three existing strategies are special cases of it

Source record: vacant/entrycost.py::_should_defect, the pulse branch

Prior work

Not new. It was described as strategic oscillation as early as 2005, and in 2009 it entered the attack taxonomy of ACM Computing Surveys under the name oscillation attack.

Round pulse-2026-08-03 · Definition · verdict value: 未複驗

pulse.timing_minorTime structure barely affects total hitsOverstatedDefense axis

The attack's time structure barely affects the total number of hits: the 15-cell (burst,recover) grid falls within 2.83–5.13 (1.81×)

Corrected

At the blind-spot point 0.5 no effect of time structure can be measured, but that is absence of evidence. Splitting hits into "number of times routed × hit rate per routing", the former spans 6.06× and the latter 8.45× — the total is flat because exposure and efficiency cancel each other out in opposite directions. Attack capability differs by 8.45×.

Supporting figures

Minimum: 2.8333
Minimum cell: (1,20)
Maximum: 5.1333
Maximum cell: (8,20)

Fields: cells[].accepted_bad.mean

Source record: 脈衝攻擊_2026-08-03/E24.json

Round pulse-2026-08-03 · Measurement · verdict value: overstated

pulse.blindspot_dominatesThe blind spot matters more than the attacker's parametersOverstatedDefense axis

The blind spot matters far more than the attacker's choice of parameters: pulse 8.9×, patient 54×, against 1.81× across the parameter grid

Corrected

Blind spot and time structure interact strongly (two-way decomposition η²=0.264, 4 times the main effect of time structure), so the sentence pattern "the real variable is X" does not itself hold. At blind spot 1.0 with exposure fixed at 90 routings, time structure alone spans 29× — more than the 8.9× attributed to the blind spot.

Supporting figures

pulse blind=0: 2.3667
pulse blind=1: 21
patient blind=0: 1.4667
patient blind=1: 79

Fields: cells[].accepted_bad.mean

Source record: 脈衝攻擊_2026-08-03/E20.json

Round pulse-2026-08-03 · Measurement · verdict value: overstated

pulse.crossoverOnce the blind spot exceeds 0.5–0.75, pulse does worse than sustained misbehaviorRefutedDefense axis

Once the blind spot exceeds about 0.5–0.75, pulsing actually does worse than plain sustained misbehavior (3.8× apart with the blind spot at its maximum)

Corrected

What changes as the blind spot rises is that **the optimal accumulation length moves toward 0**, not that the pulse family gets weaker. Measured: pulse(20,2)/(20,1)/(50,1) at blind spot 1.0 scored 81/84/87 hits, all above patient's 79.

Supporting figures

blind=0.75: {"patient":8.3333,"pulse":7.7667}
blind=1.0: {"patient":79,"pulse":21}

Source record: 脈衝攻擊_2026-08-03/E20.json

Round pulse-2026-08-03 · Measurement · verdict value: refuted

pulse.audit_cannot_close_blindspotRaising the audit rate can't rescue a situation with a blind spotRefutedDefense axis

Raising the audit rate rescues the zero-blind-spot case (100% audit = complete lockout) but not cases with a blind spot (still 0.97 hits, only 57% fully blocked)

Corrected

The detection probability is a single product (1−blind spot)×sampling rate×accuracy; all three lie on the same axis. Sampling rate and blind spot can substitute for each other (audit 0.5→1.0 exactly offsets a 50% blind spot; paired difference 0.03, p=0.78). What the blind spot changes is the defense's **ceiling** (residual β/(1−β)), not whether the sampling rate is useful.

Supporting figures

blind=0 audit=1.0: {"accepted_bad":0,"shutout":1}
blind=0.5 audit=1.0: {"accepted_bad":0.9667,"shutout":0.5667}

Source record: 脈衝攻擊_2026-08-03/E21.json

Round pulse-2026-08-03 · Measurement · verdict value: refuted

pulse.starvationCaught once = removed permanentlyOverstatedDefense axis

Caught once = removed permanently. Of the 16 seeds that were caught, all 16 were never routed again afterward

Corrected

A single slash(0.5) takes an agent out of routing within any reachable horizon. It is even stronger after clean isolation: with defect_budget=1 over 4000 rounds, 27/30 of the uncaught ran to round 3993, while the 3/3 caught got zero routings in the 3961 rounds that followed. The real mechanism is not "the per-stream clock makes waiting useless" — waiting does work (the UCB exploration term); what is fatal is that slash's β += (α+β) halves the mean and doubles n at the same time, and n is the exponential coefficient of the return time: the penalty narrows its own pardon channel along with everything else.

Supporting figures

Seeds caught: 16
Seeds routed again afterward: 0
Median round of last routing: 171
Total rounds: 600

Source record: 脈衝攻擊_2026-08-03/E17/logs/burst3__p*.jsonl

Round pulse-2026-08-03 · Measurement (from per-round records, not aggregates) · verdict value: overstated

pulse.reaction_lagEven with zero blind spot, 2.53 get through and reaction takes 28 roundsNot re-verifiedDefense axis

Even with zero blind spot, one wave still lets 2.53 through, and it takes 28 rounds on average to react

Supporting figures

hits_per_burst: 2.5333
react_lag_rounds: 27.95

Fields: cells[label=blind=0.0].hits_per_burst.mean / react_lag_mean.mean

Source record: 脈衝攻擊_2026-08-03/E22.json

Round pulse-2026-08-03 · Measurement · verdict value: 未複驗

pulse.crude_wins_under_blindspotUnder an equal budget, the crude strategy beats pulseRefutedDefense axis

Under an equal budget (12 each) + blind spot 0.5, whitewash at 8.17 actually beats pulse at 4.23

Corrected

Once the budget truly binds, the ranking reverses (budget=2: patient/pulse 1.67 vs whitewash 0.97, paired p=0.0015). Whitewash's advantage comes from switching identities for free to regain the right to be routed — it used up 4.7 identities and was still taking jobs at round 542, while pulse was starved by round 160. That is an entry-cost problem and has nothing to do with the pulse's timing structure.

Supporting figures

whitewash: 8.1667
sybil: 6.1333
pulse(5,20): 4.3
pulse(3,10): 4.2333
patient: 3.7333

Source record: 脈衝攻擊_2026-08-03/E19.json

Round pulse-2026-08-03 · Measurement · verdict value: refuted

entry.fee_backfiresAn entry fee set too low actually helps attackers outlast probationNot re-verifiedDefense axis

An entry fee set too low actually helps attackers outlast the probation period: stake=0 blocks everything, stake=2 yields 5.00 hits and an ROI of 2.35

Fields: cells[].accepted_bad.mean / roi.mean

Source record: 入場成本_2026-07-26/E4.json

Round entrycost-2026-07-26 · Measurement · verdict value: 未複驗

entry.reviewer_accuracy_bindsThe real constraint is reviewer accuracyNot re-verifiedDefense axis

The real constraint is reviewer accuracy: at acc=0, 16.65 hits and an ROI of 1.04 (misbehaving starts to pay)

Source record: 入場成本_2026-07-26/E12.json

Round entrycost-2026-07-26 · Measurement · verdict value: 未複驗

realmodel.e10_paired_nullE10 trust-switch paired experiment: no significant differenceNot re-verifiedDefense axis

E10 trust switch, N=60 paired: on 36/60 vs off 31/60, difference +8.3%, 95% CI [-5.0%, 21.7%] spans 0, McNemar p=0.332, not significant

Source record: 真模型_2026-07-26/E10/{on,off}/rows.jsonl

Round realmodel-2026-07-26 · Measurement · verdict value: 未複驗

realmodel.measurement_errorThe false result of the first measurement, which read the wrong fieldNot re-verifiedDefense axis

The first measurement was wrong: it read audit.passed as the quality criterion, while the off arm is by design not audited, producing a false result of 0/11 vs 5/10

Source record: examples/realmodel_suite.py, its comments (which keep a full account of the error)

Round realmodel-2026-07-26 · Self-correction · verdict value: 未複驗

gain.signal_existsThe gauge has signal (OFF failure rate falls in the window)HeldGain axis

The gauge has signal: without the trust layer (OFF) the failure rate is 26.44%, inside the 20–60% window set in advance

Supporting figures

OFF processed: 179
OFF infra_void: 5
measured: 174
Failure rate: 0.2644
Wilson CI95: [0.204,0.334]

Source record: 增益實驗_2026-08-30/DECISION_20260901_R437_DECISIVE_RUN_COMPLETE_FINAL.md

Round gain-2026-08-30 · Measurement (pre-set criterion 1) · verdict value: held

gain.arms_differThe three arms differ discernibly in delivery qualityNo discernible differenceGain axis

The three arms (OFF/ON/OFF5) differ discernibly in delivery quality

Corrected

"The three arms differ" does not hold. On this worker pool (qwen3.6-35b + gemma-4-12b) and MBPP+, no discernible difference in delivery quality could be measured between ON and OFF5.

Supporting figures

raw ON vs OFF5: {"n":101,"ON":0.7525,"OFF5":0.7723,"b":6,"c":8,"p":0.7905}
typing-fixed version: {"n":101,"ON":0.8416,"OFF5":0.8119,"b":5,"c":2,"p":0.4531}

Source record: 增益實驗_2026-08-30/DECISION_20260901_R437_DECISIVE_RUN_COMPLETE_FINAL.md

Round gain-2026-08-30 · Measurement (pre-set criterion 2, paired McNemar) · verdict value: no_effect

gain.equal_budget_on_beats_off5Under an equal budget, Vacant beats five-try majority voteRefutedGain axis

Under an equal budget, Vacant (ON) beats running the same task five times and taking the majority vote (OFF5)

Corrected

What ON does beyond self-consistency (a reviewer + one revision round) delivered no measurable value on this batch of workers and tasks. Boundary: a stronger reviewer, a "revise only when there is a counterexample" trigger, or a different task bank could each flip the result — that is a new experiment (the R440/R440B ladder).

Supporting figures

Full paired set: {"n":101,"p":0.4531}
Hard subset (the OFF failure set): {"n":21,"ON":0.2857,"OFF5":0.2857,"b":1,"c":1,"p":1}

Source record: 增益實驗_2026-08-30/DECISION_20260901_R438_HARD_SUBSET_AND_REVISE_MECHANISM.md

Round gain-2026-08-30 · Measurement (pre-set criterion 3) · verdict value: refuted

gain.mechanismWhy it can't win: reviewer votes are near-constant, and revision makes no net correctionHeldGain axis

Why it can't win: the reviewer's vote is almost a constant function, and revision produces almost no net correction

Supporting figures

Accuracy of effective reviewer votes: 0.7552
almost-PASS baseline: 0.7522
Difference: 0.0029
revise counterfactual no_opportunity: 93/113
hidden pass rate when revised is adopted blindly: 82.3%→78.8%
discarded_win: 0/113
revision_transition=improved: 1/113

Source record: 增益實驗_2026-08-30/DECISION_20260901_R438_HARD_SUBSET_AND_REVISE_MECHANISM.md

Round gain-2026-08-30 · Mechanism analysis (recomputed on the full 179-task data) · verdict value: held

gain.ceiling_hypothesisA weaker model would reveal the gain (ceiling hypothesis H-A)RefutedGain axis

The 35b worker is too strong and causes a ceiling effect; switch to a single 12b model and Vacant will show a gain

Corrected

Under this mechanism design, worker strength is not the binding constraint.

Supporting figures

OFF failure rate: 0.318
ON vs OFF5: {"n":167,"b":11,"c":12,"p":1}
Reviewer grounded − baseline: +4.19pp (95% [0, 8.58])
ON void: 12/179

Source record: 增益實驗_2026-08-30/DECISION_20260903_R440N_E1_INDEPENDENT_AUDIT_CONCUR.md

Round gain-2026-08-30 · Measurement (hypothesis H-A, proposed by the human on 2026-09-01; E1, gemma-4-12b single-model pool, 179 tasks) · verdict value: refuted

gain.hard_bench_hypothesisA harder task bank would reveal the gain (difficulty hypothesis H-B)RefutedGain axis

MBPP+ is too easy for modern models; switch to date-stamped contest problems (LiveCodeBench) and the gain will show

Corrected

Three clean runs, two model families, two task banks, two difficulty levels, and the same answer: under an equal budget this mechanism does not beat self-consistency.

Supporting figures

OFF failure rate: 0.484
Wilson CI95: [0.384,0.585]
ON vs OFF5: {"n":87,"b":15,"c":10,"p":0.4244}
The task bank really is harder: 16.6pp higher than MBPP+'s 31.8%

Source record: 增益實驗_2026-08-30/DECISION_20260904_R440T_E3_WRAPUP.md

Round gain-2026-08-30 · Measurement (hypothesis H-B, proposed by the human on 2026-09-01; E3, LCB 91 tasks medium+hard × gemma-only) · verdict value: refuted

gain.reviewer_tracks_difficultyThe reviewer tracks task difficulty, not whether the answer is rightHeldGain axis

The reviewer tracks task difficulty, a global property, not whether an individual answer is right

Supporting figures

MBPP+: {"Raw FAIL claims":"A minority (almost always PASS)","First-draft error rate":0.305,"Raw layer − constant baseline":"-2.99pp"}
LCB hard tasks: {"Raw FAIL claims":"234/261 = 89.7% (almost always FAIL)","First-draft error rate":0.46,"Raw layer − constant baseline":"+0.77pp"}
Interpretation: Switching the task bank only switches which constant it leans toward

Source record: 增益實驗_2026-08-30/DECISION_20260904_R440T_E3_WRAPUP.md

Round gain-2026-08-30 · Mechanism analysis (comparison across two task banks) · verdict value: held

gain.clean_datasetThe decisive run is the first clean complete datasetOverstatedGain axis

The decisive run is the first clean complete dataset: no known bug contamination, void rate within the window

Corrected

The paired analysis uses the n=101 tasks where none of the three arms is void, so its statistical conclusion is not directly affected by voids; but "clean" should become "no known bug contamination, paired subset n=101", and void rates must be listed as they are. The qwen-only runs after round459/R440D, with 79% void, have separately been ruled unusable.

Supporting figures

ON infra_void: 66/179
OFF5 infra_void: 30/179
OFF infra_void: 5/179
R437's own note: void rate ON=36.2% OFF5=16.9% ⚠VOID-GATE-WARNING
equal_budget_comparison_valid (summary field): false

Source record: 增益實驗_2026-08-30/g_r356_3arm_summary.json

Round gain-2026-08-30 · Data-quality claim · verdict value: overstated

gain.conform_early_stop_beats_singleThe early-stop gate delivers more than taking a single sampleHeldGain axis

"Run the customer's own acceptance tests, deliver the first one that passes, refuse delivery if none passes" (the CONFORM early-stop gate) delivers more than "take a single sample and accept it" (OFF)

Supporting figures

MBPP+ pooled, 371 tasks: +4.58pp (CI [+0.57, +8.04])
LCB v2, 120 tasks: {"delta_pp":19.17,"b":31,"c":8,"p":0.0003}
LCB v3, 189 tasks: {"delta_pp":7.94,"b":25,"c":10,"p":0.0167}
Calls/task: 1.51/1.71/1.55 vs 1.00

Fields: arm/task_id/delivered/hidden_pass/calls_used in rows.jsonl

Source record: runs/g_r444_conform_mbpp/rows.jsonl, runs/g_r445_conform_mbpp_ext/rows.jsonl (MBPP+ pooled, 371 tasks); runs/g_r447_conform_lcb2/rows.jsonl (LCB v2, 120 tasks); runs/g_r461_lcb3_three_arm/rows.jsonl (LCB v3, 189 tasks)

How to recompute: ops/gain/replay/conform_settle.py, ops/gain/replay/paired_ci.py, ops/gain/replay/pooled_paired_ci.py (zero API; reads rows.jsonl only)

Round gain-2026-08-30 · Measurement (three task banks, paired) · verdict value: held

gain.gate_rule_beats_majority_vote_same_candidatesSame set of candidates: the gate rule beats a five-way voteHeldGain axis

With the same 5 candidates and the same 5 calls, the gate rule (deliver the first one that passes, deliver nothing if none passes) delivers more than a five-way majority vote

Supporting figures

r446: {"gate":75.47,"vote":71.43,"b":24,"c":9,"delta_pp":4.04,"ci95":[0.796,6.529],"p":0.0135}
r448: {"gate":77.09,"vote":73.58,"b":21,"c":8,"delta_pp":3.5,"ci95":[0.81,6.47],"p":0.0241}
r449b: {"gate":70.83,"vote":62.5,"b":15,"c":5,"delta_pp":8.33,"ci95":[0.3,13.78],"p":0.0414}
r449c (UNRESOLVED): {"gate":83.07,"vote":78.84,"b":13,"c":5,"delta_pp":4.23,"ci95":[-0.66,7.68],"p":0.0963}

Source record: runs/g_r446_eq5_mbpp/rows.jsonl (MBPP+, 371 tasks), runs/g_r448_eq5_mbpp_seed2/rows.jsonl (MBPP+, 371 tasks, different seed), runs/g_r449_eq5_lcb2/rows.jsonl (LCB v2 hard tasks, 120 tasks), runs/g_r449c_eq5_lcb3/rows.jsonl (LCB v3, 189 tasks, ruled UNRESOLVED)

How to recompute: ops/gain/analyze_eq5.py (zero API; reads rows.jsonl only; the prereg block it prints itself is **not** the arbiter — see the end of R449B §三)

Round gain-2026-08-30 · Measurement (EQ5 equal-budget arm, four runs) · verdict value: held

gain.conform_vs_off5_unresolvedThe gate beats majority vote on delivery accuracySame sign, unresolvedGain axis

The gate (CONFORM) beats running the same task five times and taking the majority vote (OFF5) on delivery accuracy

Supporting figures

LCB v2: {"delta_pp":6.67,"p":0.15}
LCB v3: {"delta_pp":1.59,"ci95":[-3.17,6.35],"p":0.6636}
MBPP+ clean replication (192 new tasks): {"delta_pp":4.69,"ci95":[-1.11,9.42]}
MDE@371: 4.31
Pairs needed for 80% power: 491
Task-bank ceiling: 378

Source record: runs/g_r447_conform_lcb2/rows.jsonl (LCB v2, 120 tasks), runs/g_r461_off_gate_lcb3/rows.jsonl and runs/g_r461_lcb3_three_arm/rows.jsonl (LCB v3, 189 tasks); on the MBPP+ side, runs/g_r445_conform_mbpp_ext/rows.jsonl

How to recompute: ops/gain/replay/paired_ci.py, ops/gain/replay/off5_k_curve.py

Round gain-2026-08-30 · Measurement (independent sampling, two LCB runs) · verdict value: unresolved

gain.lossless_visible_filterThe visible filter is losslessHeldGain axis

The visible filter is lossless — no candidate appeared that "would pass the hidden tests but failed the visible acceptance check"

Supporting figures

MBPP+ total: 0/1630
LCB v1 replay: 0/455
LCB v2 real run: 0/120
LCB v3 real run: 0/189
Overlap (not independent samples): The LCB v1 replay and the LCB v2 real run share 91 tasks

Fields: visible_pass and hidden_pass in rows.jsonl; violation = hidden_pass ∧ ¬visible_pass

Source record: runs/g_r441_gemma_only_mbpp_b/rows.jsonl (0/895), runs/g_r356_3arm_20260830/rows.jsonl (0/735), runs/g_r443_gemma_lcb/rows.jsonl (LCB v1 replay, 0/455), runs/g_r447_conform_lcb2/rows.jsonl (LCB v2 real run, 0/120), runs/g_r461_off_gate_lcb3/rows.jsonl (LCB v3 real run, 0/189)

How to recompute: ops/gain/replay/rows_visible_audit.py, ops/gain/replay/verify_hidden.py, ops/gain/replay/check_lcb_visible_subset.py

Round gain-2026-08-30 · Measurement (six datasets) · verdict value: held

gain.off5_helps_on_hard_onlyFive times the budget buys something only on hard tasksHeldGain axis

Self-consistency at five times the budget (OFF5 vs OFF) buys something only on hard tasks

Supporting figures

MBPP+: {"delta_pp":0.81,"ci95":[-2.78,4.28],"cell":"RULED_OUT"}
LCB v2: {"delta_pp":12.5,"b":22,"c":7,"p":0.0081}
LCB v3: {"delta_pp":6.35,"b":22,"c":10,"p":0.0501}
Comparison (changing how the budget is spent helps more than adding budget): +19.2pp @1.71 calls vs +12.5pp @5 calls
r461 does not count as a hard-task replication: lcb3's OFF failure rate is 27.5%, already back at the MBPP+ level of 31.8%

Source record: runs/g_r441_gemma_only_mbpp_b/rows.jsonl and runs/g_r356_3arm_20260830/rows.jsonl (MBPP+), runs/g_r447_conform_lcb2/rows.jsonl (LCB v2, 120 tasks), runs/g_r461_lcb3_three_arm/rows.jsonl (LCB v3, 189 tasks)

How to recompute: ops/gain/replay/off5_k_curve.py, ops/gain/replay/paired_ci.py

Round gain-2026-08-30 · Measurement (OFF5 vs OFF, cross-bank comparison) · verdict value: held

peerexec.mutual_execution_below_thresholdBelow the majority threshold, mutual execution can name the liarHeldRecords & multi-party

Below the majority threshold (corrupted executors ≤ ⌊(k−1)/2⌋), the deliveries of mutual execution without mutual review are bit-for-bit identical to the uncorrupted baseline; the liar is named and honest participants are not falsely accused

Supporting figures

Simulation, below the threshold: {"Bit-for-bit identical to the uncorrupted baseline":"100 cells","Liar named":1,"Honest participant falsely accused":0,"1 corrupted, k=1":"−7.8pp, detection 0","1 corrupted, k=3":"detection 1.000"}
Real run R453 (k=2): {"Visible labels agree across machines":"1840/1840","Shipped sha":"340/340","Refused delivery":"26/26","Honest executors named":0,"Chain verifies, per machine":"2/2","spec/render sha identical across machines":"368/368"}
Real run R454 (k=3, 1 lying key): {"Lying cells whose dissenters are exactly {K3}":"273/273","Honest keys appearing in the naming column":"0 (denominator 1840)","Self-contradictory cells still given the correct verdict":"58/58","Three chains verify":"3/3 (chain lengths 1840/1840/1899)","Testimony entries failing signature verification":"0 (of 5519 in total)"}

Source record: [Simulation] ops/gain/replay/peer_exec_sweep.json (200 cells: r446 371 tasks × 5 candidates, r443 91 tasks × 5, k∈{1,3,5,7}, corruption ratio 0–70%, five attacks), ops/gain/replay/peer_exec_flake.json (flakiness × corruption); [Real runs] ops/gain/replay/r453/r453_result.json and r453_independent_audit.json (k=2, Mac + vacant-dev), ops/gain/replay/r454/r454_result.json and r454_naming_table.tsv (k=3, 1 lying key)

How to recompute: ops/gain/replay/peer_exec_sim.py (simulation), ops/gain/replay/peer_exec_real.py (real runs), ops/gain/replay/receipt_chain_audit.py (chain verification); the code is in vacant/peerexec.py

Round peerexec-2026-09-05 · Simulation sweep + real runs (which half is which: see "Limits") · verdict value: held

peerexec.suite_fixed_pointMutual execution has no defense at all against a corrupted acceptance suiteHeldRecords & multi-party

Mutual execution without mutual review has no defense at all against "the acceptance suite itself being corrupted" — this is a fixed point of the mechanism; R451→R452 shrank it to a residual but did not eliminate it

Supporting figures

trivial suite: {"Delivery rate":"−6.47pp/−18.68pp","False delivery":"31%/49%","Dispute rate across the four k values":"all 0.0%"}
R451: gauge bound into the commit: the trivial suite was rejected at commit in 371/371 (without spending a single sandbox run)
R451 stateful variant: {"Delivery rate":"−75.20pp","Delivered":"0.00%","gauge_status":"ok (the gauge as a partial solution is voided)"}
R452: suite turned into data: {"exec smuggled onto the chain":"368/371 → 0/371","Oversized hexadecimal integers getting through the gate":"11/11 → 0/11","Lossless":"1840 (MBPP+) + 455 (LCB) candidates identical cell by cell, 0 mismatches"}
Residual (always state both numbers; denominator 366/371 tasks on the chain): {"Achievable generic_gauged":"17.93% → 20.65% = +2.72pp","Hindsight upper bound weak_oracle":"+4.35pp"}

Source record: ops/gain/replay/peer_exec_trivial_suite.json (a suite where "if it loads, it passes"), ops/gain/replay/r451_stateful_gate.json (a stateful variant that scores full marks on the gauge in both directions), ops/gain/replay/peer_exec_suitespec_gate.json and r452b_smuggle_gate.json (exec smuggling), ops/gain/replay/r452c_generic_gate.json and r452c_channel_hunt.json (residual: generic_gauged/weak_oracle), ops/gain/replay/r453/r453_gauge.json

How to recompute: ops/gain/replay/r452_suitespec.py, ops/gain/replay/r452c_generic_gate.py, ops/gain/replay/r452b_smuggle_gate.py; the code is in vacant/suitegauge.py

Round peerexec-2026-09-05 · Limit claim (held = this "cannot be bought" was itself measured) · verdict value: held

peerexec.majority_boundNaming has a mathematical upper bound ⌊(k−1)/2⌋HeldRecords & multi-party

Naming has a mathematical upper bound: majority vote tolerates at most ⌊(k−1)/2⌋ corrupted executors; past a majority it flips

Supporting figures

Tolerance upper bound: ⌊(k−1)/2⌋
False-accusation rate past a majority: 0.175/0.374 (two sweep batches: r446 371 tasks × 5, r443 91 tasks × 5)
Delivery rate for k=1..7 at a fixed corruption ratio: unchanged to the digit (collusion 67.39%, sabotage 0%)
k=3/quorum=2 with one vote missing: 1-1 tie ⇒ undecided, no one named

Source record: ops/gain/replay/peer_exec_sweep.json (companion measurement: at a fixed corruption ratio, the delivery rate does not change by a single digit as k goes from 1 to 7), ops/gain/replay/r454/r454_naming_table.tsv (the naming column of the real run)

Round peerexec-2026-09-05 · Limit claim (a mathematical upper bound of the mechanism, not a measured result) · verdict value: held

harness.hmix_loop_beats_resample_same_budgetThe revision loop delivers more than resampling and spends no more tokensSame sign, unresolvedGain axis · loop

Spending five calls on "run the customer's acceptance tests, paste the failure output back verbatim, let the same worker revise" (H-MIX) delivers more than spending them on resampling with a fresh attempt (CONFORM), and spends no more tokens

Supporting figures

R460: {"OFF":{"deliv":58.33,"false_delivery_n":50,"tokens_per_task":2913},"CONFORM":{"deliv":70.83,"false_delivery_n":29,"tokens_per_task":5805},"OFF5":{"deliv":65,"false_delivery_n":42,"tokens_per_task":15004},"HPI":{"deliv":75.83,"false_delivery_n":17,"tokens_per_task":9448},"HOC":{"deliv":80,"false_delivery_n":18,"tokens_per_task":7432},"HMIX":{"deliv":84.17,"false_delivery_n":14,"tokens_per_task":5677},"HMIX_vs_CONFORM":{"b":22,"c":6,"delta_pp":13.33,"ci95":[4.22,19.46],"p":0.0037,"holm_p_adj":0.0112,"verdict":"EFFECTIVE"},"HMIX_vs_OFF":{"b":34,"c":3,"delta_pp":25.83,"ci95":[17.32,29.78]}}
R460R_HMIX_vs_CONFORM_by_rep: [{"rep":1,"num":93,"den":120,"conform_num":86,"delta_pp":5.83,"b":15,"c":8,"ci95":[-2.79,12.89],"holm_p_adj":0.63,"four_state":"INCONCLUSIVE"},{"rep":2,"num":92,"den":120,"conform_num":87,"delta_pp":4.17,"b":17,"c":12,"ci95":[-5.35,12.8],"holm_p_adj":0.917,"four_state":"INCONCLUSIVE"},{"rep":3,"num":91,"den":120,"conform_num":90,"delta_pp":0.83,"b":12,"c":11,"ci95":[-7.44,8.89],"holm_p_adj":1,"four_state":"RULED_OUT","ci95_upper":8.89,"four-state meaning":"RULED_OUT = rules out ≥+10pp; it does not rule out every effect"},{"rep":4,"num":88,"den":120,"conform_num":85,"delta_pp":2.5,"b":13,"c":10,"ci95":[-5.94,10.28],"holm_p_adj":0.678,"four_state":"INCONCLUSIVE","note":"The upper bound 10.28 is only 0.28pp from RULED_OUT ⇒ the four-state count is extremely sensitive to the interval method"},{"rep":5,"num":87,"den":116,"conform_num":82,"delta_pp":4.31,"b":14,"c":9,"ci95":[-4.54,12.01],"holm_p_adj":0.922,"four_state":"INCONCLUSIVE","denominator":"complete-case 116 (7 infra_void rows voided)","worst-case bound over the voided rows":[0.83,7.5]}]
R460R_rep5_per_arm_den: {"OFF":119,"CONFORM":117,"OFF5":118,"HPI":120,"HOC":120,"HMIX":119}
R460R_rep5_infra_void: {"n":7,"by_arm":{"OFF":1,"CONFORM":3,"OFF5":2,"HMIX":1},"source":"JIT reload after the backend model crashed at 2026-09-13 01:44Z (TTL 1h, unloaded at :07 every hour); notes.jsonl records, entry by entry, HTTP 400 Failed to load model gemma-4-12b-it-qat (5) / HTTP 500 (2)"}
R460R_reps_run: 5
R460R_reps_prereg: 5
R460R_statement_rule: 5/5 same sign: met; ≥4/5 Holm-significant: not met (0/5) ⇒ list each replication as it is; do not write "the replication is stable" or "a majority supports it", and do not write "the replication failed", "the effect disappeared" or "equivalent" either
R460R_holm_significant_n: 0
R460R_ci_all_intersect_R460: All five unadjusted intervals intersect R460's [4.22, 19.46]
R460R_ci_uppers: [12.89,12.8,8.89,10.28,12.01]
R460R_ci_uppers_all_below_13.33: true
R460R_power_expectation: At n=120 the statistical power for +10pp is 0.43–0.63 ⇒ 2–3 of the five are expected to pass; if the true value is +10pp, the probability of 0/5 is about 0.007–0.06 (lower tail)
R460R_co_tenancy_pct: [8.5,71.3,2.1,0,0]
R529_HMIX_vs_CONFORM_by_set: [{"set":"lcb3_medium","num":126,"den":135,"conform_num":125,"delta_pp":0.74,"b":6,"c":5},{"set":"lcb3_hard","num":43,"den":54,"conform_num":41,"delta_pp":3.7,"b":5,"c":3},{"set":"humanevalplus","num":148,"den":156,"conform_num":147,"delta_pp":0.64,"b":5,"c":4},{"set":"mbppplus","num":299,"den":371,"conform_num":295,"delta_pp":1.08,"b":15,"c":11}]
R529_HMIX_vs_CONFORM_pooled: {"num":616,"den":716,"conform_num":608,"delta_pp":1.12,"b":31,"c":23,"holm_p_adj":0.341}
R529_four_state: INCONCLUSIVE
Direction: 9/9 same sign (five replications + four sets); magnitude not established

Source record: runs/g_r460_harness_lcb2_{a1,a2,a3,b1,b2,b3}/rows.jsonl and calls.jsonl (six blocks of 20 tasks each, 120 tasks combined); runs/g_r460r{1,2,3,4,5}_harness_lcb2_{a1,a2,a3,b1,b2,b3}/rows.jsonl (five replications, 120 tasks each; r5 has 713 rows, the 7 infra_void rows are in notes.jsonl); runs/g_r529_{lcb3m_a1..a7,lcb3h_a1..a3,hep_a1..a8,mbpp_a1..a19}/rows.jsonl (four sets, 716 tasks)

How to recompute: ops/gain/analyze_r460.py --run <six blocks> --bank lcb2 --rescore-turn1 (zero API, sandbox only); ops/gain/analyze_r460r.py --reps 1 2 3 4 5 --bank lcb2; written to ops/gain/replay/r460/r460_analyze.json, **ops/gain/replay/r460r/r460r_analyze_5reps.json** (its arbitration block is bit-for-bit identical to that of `_norescore.json`), ops/gain/replay/r529/r529_analyze.json. The R460R numbers in this claim were recomputed independently from rows.jsonl (accepted∧meets_demand) and agree with the analyzer cell by cell.

Round harness-2026-09-08 · Measurement (R460 six arms, LCB v2 120 tasks + R460R **five** same-task replications + R529 four cross-bank sets) · verdict value: unresolved

harness.loop_beats_single_shot_replicatedThe feedback loop delivers more than taking a single sampleHeldGain axis · loop

Spending five calls on a feedback loop (H-MIX/H-PI/H-OC) delivers more than taking a single sample and accepting it (OFF) — this one holds up across task banks and across replications

Supporting figures

R460R_vs_OFF_by_rep: {"HMIX":[{"rep":1,"num":93,"off_num":69,"den":120,"delta_pp":20,"b":29,"c":5},{"rep":2,"num":92,"off_num":65,"den":120,"delta_pp":22.5,"b":34,"c":7},{"rep":3,"num":91,"off_num":69,"den":120,"delta_pp":18.33,"b":28,"c":6},{"rep":4,"num":88,"off_num":62,"den":120,"delta_pp":21.67,"b":33,"c":7},{"rep":5,"num":88,"off_num":61,"den":118,"delta_pp":22.88,"b":34,"c":7,"denominator":"complete-case 118"}],"HPI":[{"rep":1,"delta_pp":19.17,"b":26,"c":3,"den":120},{"rep":2,"delta_pp":25,"b":35,"c":5,"den":120},{"rep":3,"delta_pp":18.33,"b":29,"c":7,"den":120},{"rep":4,"delta_pp":29.17,"b":39,"c":4,"den":120},{"rep":5,"delta_pp":19.33,"b":31,"c":8,"den":119}],"HOC":[{"rep":1,"delta_pp":19.17,"b":28,"c":5,"den":120},{"rep":2,"delta_pp":19.17,"b":30,"c":7,"den":120},{"rep":3,"delta_pp":17.5,"b":28,"c":7,"den":120},{"rep":4,"delta_pp":26.67,"b":38,"c":6,"den":120},{"rep":5,"delta_pp":24.37,"b":31,"c":2,"den":119}],"holm":"all fifteen cells significant (family of 6 tests within each replication)"}
R529_HMIX_vs_OFF_by_set: [{"set":"lcb3_medium","num":126,"off_num":115,"den":135,"delta_pp":8.15,"b":17,"c":6},{"set":"lcb3_hard","num":43,"off_num":38,"den":54,"delta_pp":9.26,"b":7,"c":2},{"set":"humanevalplus","num":148,"off_num":129,"den":156,"delta_pp":12.18,"b":24,"c":5},{"set":"mbppplus","num":299,"off_num":277,"den":371,"delta_pp":5.93,"b":31,"c":9}]
R529_HMIX_vs_OFF_pooled: {"num":616,"off_num":559,"den":716,"delta_pp":7.96,"b":79,"c":22,"holm_p_adj":2.02e-8}
Number of real sources: 3
Denominator warning: HumanEval+ is 156/164 (8 tasks excluded by the sandbox envelope); the three comparison denominators of R460R r5 each differ (118/119/119, 7 infra_void rows); do not backfill them uniformly to 120
Backend load is not homogeneous: The co-tenancy rates of the five replications and R529 are 8.5/71.3/2.1/0/0%; the effect size in this claim is too large for co-tenancy to explain (magnitude ceiling ≤1.7pp/arm), but it must still be stated

Source record: runs/g_r460r{1,2,3,4,5}_harness_lcb2_*/rows.jsonl (five × 120 tasks; r5 has 713 rows); runs/g_r529_*/rows.jsonl (four sets, 716 tasks)

How to recompute: ops/gain/replay/r529/r529_analyze.json; ops/gain/replay/r460r/r460r_analyze_5reps.json; the five R460R replications recomputed independently from rows.jsonl (r5 uses complete-case denominators)

Round harness-2026-09-08 · Measurement (**five** same-task replications × three harness arms + four mutually exclusive task sets) · verdict value: held

harness.gain_comes_from_executable_gate_and_resampleThe bulk of the gain is gate + resampling; the loop is a small incrementHeldGain axis · loop

The bulk of the gain is "the executable acceptance gate + resampling"; the feedback loop is only a small increment stacked on top of it

Supporting figures

R460R_CONFORM_vs_OFF_by_rep: [{"rep":1,"conform_num":86,"off_num":69,"den":120,"delta_pp":14.17,"b":22,"c":5,"p_uncorrected":0.0015},{"rep":2,"conform_num":87,"off_num":65,"den":120,"delta_pp":18.33,"b":28,"c":6,"p_uncorrected":0.0002},{"rep":3,"conform_num":90,"off_num":69,"den":120,"delta_pp":17.5,"b":26,"c":5,"p_uncorrected":0.00019},{"rep":4,"conform_num":85,"off_num":62,"den":120,"delta_pp":19.17,"b":30,"c":7,"p_uncorrected":0.00019},{"rep":5,"conform_num":82,"off_num":60,"den":116,"delta_pp":18.97,"b":28,"c":6,"p_uncorrected":0.0002,"denominator":"complete-case 116 (7 infra_void rows voided)"}]
R529_CONFORM_vs_OFF_by_set: [{"set":"lcb3_medium","conform_num":125,"off_num":115,"den":135,"delta_pp":7.41,"b":14,"c":4,"p_uncorrected":0.031},{"set":"lcb3_hard","conform_num":41,"off_num":38,"den":54,"delta_pp":5.56,"b":3,"c":0,"p_uncorrected":0.25},{"set":"humanevalplus","conform_num":147,"off_num":129,"den":156,"delta_pp":11.54,"b":21,"c":3,"p_uncorrected":0.00028},{"set":"mbppplus","conform_num":295,"off_num":277,"den":371,"delta_pp":4.85,"b":26,"c":8,"p_uncorrected":0.0029}]
R529_CONFORM_vs_OFF_pooled: {"conform_num":608,"off_num":559,"den":716,"delta_pp":6.84,"b":64,"c":15,"p_uncorrected":2.3e-8}
How much more the loop adds: {"R460R":[5.83,4.17,0.83,2.5,4.31],"R529":[0.74,3.7,0.64,1.08]}
All p values uncorrected: The p_raw of CONFORM−OFF is < 0.002 in all five replications, but CONFORM−OFF is not in any preregistered Holm family ⇒ it may not be read as "the gate's effect is significant"
Holm for the four sets, added after the fact (descriptive only): {"humanevalplus":0.0011,"mbppplus":0.0088,"lcb3_medium":0.062,"lcb3_hard":0.25}
Not causal: CONFORM and H-MIX are two separately run arms, not two stages; "gate +14pp, loop +6pp" is a narrative obtained by subtraction, not separated components

Source record: runs/g_r460r{1,2,3,4,5}_harness_lcb2_*/rows.jsonl; runs/g_r529_*/rows.jsonl

How to recompute: Numerators and denominators recomputed independently from rows.jsonl (delivery criterion = accepted ∧ meets_demand); ops/gain/replay/r460r/r460r_analyze_5reps.json

Round harness-2026-09-08 · Attribution (**audit judgment**, not a preregistered primary endpoint; CONFORM−OFF is an exploratory quantity outside the family) · verdict value: held

harness.majority_vote_loses_to_gateWith the same five calls, voting loses to the gate and costs more than twice as muchSame sign, unresolvedGain axis · loop

With the same five calls, a five-way majority vote (OFF5) **loses to** "run the acceptance tests, deliver the first one that passes" (CONFORM), and costs more than twice as much

Supporting figures

OFF5_vs_CONFORM_by_rep: [{"rep":1,"off5_num":73,"conform_num":86,"den":120,"delta_pp":-10.83,"b":7,"c":20,"p_uncorrected":0.019},{"rep":2,"off5_num":74,"conform_num":87,"den":120,"delta_pp":-10.83,"b":9,"c":22,"p_uncorrected":0.029},{"rep":3,"off5_num":71,"conform_num":90,"den":120,"delta_pp":-15.83,"b":4,"c":23,"p_uncorrected":0.00031},{"rep":4,"off5_num":80,"conform_num":85,"den":120,"delta_pp":-4.17,"b":12,"c":17,"p_uncorrected":0.458,"co-tenancy":"zero (1004 had all three streams to itself)"},{"rep":5,"off5_num":80,"conform_num":82,"den":115,"delta_pp":-1.74,"b":11,"c":13,"p_uncorrected":0.839,"denominator":"complete-case 115","co-tenancy":"zero (1004 had all three streams to itself)"}]
Same sign 5/5: true
Replications with uncorrected p < 0.05: 3
The two clean replications are not significant: r4/r5, the two with zero backend co-tenancy, are exactly the two at −4.17/−1.74; in the same period OFF fell to 51.67/51.26%, OFF5 rose to 66.67/68.64%, and CONFORM barely moved (70.83/70.94%) ⇒ part of the cross-replication difference is not sampling
But co-tenancy cannot explain this shape: OFF5 = majority of five OFF samples; as OFF got worse, OFF5 got 7–9pp better, in the opposite direction ⇒ "the two clean replications are not significant" can for now only serve as a description; saying "co-tenancy made the first three look more exaggerated" is a causal statement that does not yet have evidence
tokens_per_task: {"OFF5":[15565,14877,14954,15226,14163],"CONFORM":[6497,6880,5246,5770,6416],"ratio":[2.4,2.16,2.85,2.64,2.21]}
calls_per_task: {"OFF5":[5.04,5.03,5.03,5.03,5.12],"CONFORM":[1.78,1.71,1.53,1.69,1.79]}
OFF5_vs_OFF_by_rep: [{"rep":1,"delta_pp":3.33,"p_uncorrected":0.58},{"rep":2,"delta_pp":7.5,"p_uncorrected":0.16},{"rep":3,"delta_pp":1.67,"p_uncorrected":0.85},{"rep":4,"delta_pp":15,"p_uncorrected":0.0021},{"rep":5,"delta_pp":16.24,"p_uncorrected":0.00088}]
Difference in accounting: The Fable summary gives tokens as 2.4–2.9×; recomputed this round from calls.jsonl, rep2 is 2.16× (the analyzer includes wire_probe; harness_tokens_total in rows does not). r5's 2.21 excludes the calls of voided tasks; counting the 47,102 tokens of the voided tasks (tpc_incl_void) gives 2.25 — the two numbers must not be mixed

Source record: runs/g_r460r{1,2,3,4,5}_harness_lcb2_*/rows.jsonl and calls.jsonl

How to recompute: Delivery recomputed from rows.jsonl; tokens/calls summed call by call from the usage in calls.jsonl (preflight counted separately, not included; r5 excludes the calls of voided tasks)

Round harness-2026-09-08 · Measurement (**exploratory quantity**: OFF5 − CONFORM is not in any preregistered Holm family) · verdict value: unresolved

harness.hpi_hoc_vs_resample_unresolvedThe other two loop arms also beat resamplingSame sign, unresolvedGain axis · loop

The pi-style raw feedback loop (H-PI) and plan + diagnose + self-test (H-OC) also beat resampling with a fresh attempt

Supporting figures

R460: {"HPI_vs_CONFORM":{"delta_pp":5,"ci95":[-4.4,13.3],"holm_p_adj":0.345},"HOC_vs_CONFORM":{"delta_pp":9.17,"ci95":[-1,17.62],"holm_p_adj":0.16}}
R460R_vs_CONFORM_by_rep: {"HPI":[{"rep":1,"num":92,"conform_num":86,"den":120,"delta_pp":5,"b":16,"c":10,"holm_p_adj":0.654},{"rep":2,"num":95,"conform_num":87,"den":120,"delta_pp":6.67,"b":19,"c":11,"holm_p_adj":0.602},{"rep":3,"num":91,"conform_num":90,"den":120,"delta_pp":0.83,"b":16,"c":15,"holm_p_adj":1},{"rep":4,"num":97,"conform_num":85,"den":120,"delta_pp":10,"b":20,"c":8,"holm_p_adj":0.107,"ci95":[0.62,17.16],"note":"The only cell of the thirty across the five replications whose unadjusted lower bound is > 0; not significant after Holm, and H-PI is an ablation arm ⇒ it may not stand in for the claim"},{"rep":5,"num":83,"conform_num":83,"den":117,"delta_pp":0,"b":14,"c":14,"holm_p_adj":1,"ci95":[-9.26,9.26],"four_state":"RULED_OUT","denominator":"complete-case 117","worst-case bound over the voided rows":[-2.5,2.5],"note":"The worst-case bound changes sign ⇒ RULED_OUT is kept per the rule, and the sensitivity must be written alongside it"}],"HOC":[{"rep":1,"num":92,"conform_num":86,"den":120,"delta_pp":5,"b":16,"c":10,"holm_p_adj":0.654},{"rep":2,"num":88,"conform_num":87,"den":120,"delta_pp":0.83,"b":13,"c":12,"holm_p_adj":1},{"rep":3,"num":90,"conform_num":90,"den":120,"delta_pp":0,"b":12,"c":12,"holm_p_adj":1},{"rep":4,"num":94,"conform_num":85,"den":120,"delta_pp":7.5,"b":17,"c":8,"holm_p_adj":0.216},{"rep":5,"num":89,"conform_num":83,"den":117,"delta_pp":5.13,"b":15,"c":9,"holm_p_adj":0.922,"denominator":"complete-case 117"}]}
None of the ten passed Holm: true
rep3 HOC and rep5 HPI are exact ties: 12/12 and 14/14; the direction did not turn negative; do not read this as "confirmed ineffective"
Against single-shot, all five significant: {"HPI":[19.17,25,18.33,29.17,19.33],"HOC":[19.17,19.17,17.5,26.67,24.37]}

Source record: runs/g_r460_harness_lcb2_*/rows.jsonl; runs/g_r460r{1,2,3,4,5}_harness_lcb2_*/rows.jsonl

How to recompute: ops/gain/replay/r460/r460_analyze.json · decision.HPI / decision.HOC; ops/gain/replay/r460r/r460r_analyze_5reps.json; the five R460R replications recomputed independently from rows.jsonl (r5 uses the complete-case denominator 117)

Round harness-2026-09-08 · Measurement (the other two harness arms of R460 + R460R **five** same-task replications) · verdict value: unresolved

harness.gain_is_the_loop_not_the_promptThe gain comes almost entirely from the loop, not the prompt wordingHeldGain axis · loop

H-MIX's gain comes almost entirely from the loop of "paste the failure output back verbatim and let it revise", not from the wording of the first-round prompt

Supporting figures

HMIX: {"loop_effect_pp_replays":[19.17,17.5],"prompt_effect_pp_replays":[6.67,8.33],"loop_gain_n":21,"turn1_visible_pass_pp":74.2}

Source record: runs/g_r460_harness_lcb2_*/calls.jsonl (full turn-1 responses) + rows.jsonl

How to recompute: ops/gain/analyze_r460.py --rescore-turn1; ops/gain/replay/r460/r460_analyze.json · attribution.*

Round harness-2026-09-08 · Attribution (D5: turn-1 code re-extracted offline and rescored, zero model calls) · verdict value: held

zero.dabstep_paid_no_differenceZero-config on DABstep: no difference measured for either modelNo discernible differencerepo verdictGain axis

On the paid DABstep main batch, agents with zero-config Vacant installed were not measured to have a higher correct-answer rate than agents without it: gemma-4-26b 50/77 vs 44/77 (p=0.21), qwen3.8-27b 68/77 vs 64/77 (p=0.29).

Corrected

Per the preregistration, the wording is "no difference measured", not "no effect". Verbatim: "We do not say 'no effect', and we do not say 'worse' either" (original: 「不說『沒有效果』，也不說『變差』」).

Supporting figures

gemma, not installed: 50 / 77 correct (64.9%)
gemma, installed: 44 / 77 correct (57.1%)
gemma p: McNemar exact two-sided p=0.21
qwen, not installed: 68 / 77 correct (88.3%)
qwen, installed: 64 / 77 correct (83.1%)
qwen p: p=0.29
Denominator: 77 distinct DABstep tasks, one run per task per group

Source record: Vacant repo: docs/ZERO_CONFIG_EVAL_REPORT_2026-09-26.md:150-153

How to recompute: Read the index in that repo's ops/eval/evidence_20260925/; the raw request and response bodies are all there

Prior work

This round's result does not contradict past data, because exactly the same setting was never measured before.

Round zero-2026-09-25 · Experimental result · verdict value: no_effect

zero.dabstep_paid_rejects_all_falseAll 8 send-backs in that round were false positivesRefutedrepo verdictGain axis · loop

None of the 8 send-backs zero-config triggered in the paid DABstep batch would happen after the fix — not a single send-back across 158 runs; that is, once fixed, C equals A: it neither helps nor hurts.

Corrected

The conclusion file originally said 7; after an independent recomputation it was corrected to 8.

Supporting figures

Send-backs: 8 (denominator 158 runs: gemma 79 + qwen 79)
Actual false-positive rate: gemma 6/37 (16%), qwen 1/64 (2%); the denominator is runs that reached the pre-handoff check for the first time with an answer that was correct at that moment
Harm: 0 (not once was a correct answer changed into a wrong one)

Source record: Vacant repo: docs/ZERO_CONFIG_EVAL_REPORT_2026-09-26.md:157, 219-220

Round zero-2026-09-25 · Mechanism defect · verdict value: refuted

zero.local711_p021_thinLocal 711-cell batch: p=0.021, but the difference is concentrated in 6 tasksOverstatedrepo verdictGain axis

On the local 711-cell batch, zero-config v3's preregistered primary test against no install gave p=0.0207, but the author states plainly that the significance is thin: with all 6 tasks removed, the mean difference is 0.009, p=0.80; flip the results of two runs and p goes above 0.05.

Corrected

Any sentence that cites this p-value must also say "concentrated in 6 tasks" and "with all 6 tasks removed, p=0.80". Verbatim: "The significance is thin: flip the results of two runs, or remove any two of those 6 tasks, and p goes above 0.05. Not extrapolated to other models, other task banks, or uncapped use." (original: 「顯著性很薄：兩跑的結果翻過來、或拿掉那 6 題裡的任兩題，p 就會超過 0.05。不外推到別的模型、別的題庫、沒有上限的使用。」)

Supporting figures

Primary test: Wilcoxon exact p=0.0207 (reported as 0.021)
Mean difference: +0.074 (about +7.4 pp), bootstrap 95% CI 0.017–0.139
Direction: 15 tasks better, 7 worse, 55 the same; sign test p=0.13
Sensitivity: remove any two of the 6 large-difference tasks ⇒ p≥0.05; remove all 6 ⇒ mean difference 0.009, p=0.80
Denominator: 77 tasks × 3 runs (original design 711 cells = 79 tasks × three groups × 3 runs)

Source record: Vacant repo: decisions/conclusions/CONCLUSION_20260926_ZERO_CONFIG_V3_LOCAL.md:10-15, 38-39

How to recompute: That repo's analysis script; the gauge itself was also fixed (a floating-point tie misjudgment; 0.0189 corrected to 0.0207)

Prior work

v3 was designed after looking at the records of the same 77 tasks from the previous round, so the tasks and the rules are not independent; no human signed off on this preregistration item by item.

Round zero-2026-09-26 · Experimental result · verdict value: overstated

zero.v3_3_never_firedv3-3 "with one round left, send back only for missing files" never fired onceRefutedrepo verdictGain axis · loop

The v3-3 mechanism did not fire even once in 150 local pre-handoff checks; it made no observable contribution.

Supporting figures

Denominator: all 150 pre-handoff checks reached in C2
Times fired: 0
Reason: no pre-handoff check happened at round 14 or later

Source record: Vacant repo: decisions/conclusions/CONCLUSION_20260926_ZERO_CONFIG_V3_LOCAL.md:55

Round zero-2026-09-26 · Mechanism defect · verdict value: refuted

zero.colab920_no_differenceZero-config on the Colab 920-task coding banks: no difference measuredNo discernible differencerepo verdictGain axis

On 920 tasks from five coding task banks on Colab, agents with v3.6.1 installed were not measured to have a higher correct-answer rate than agents without it: 748/920 (81.3%) vs 758/920 (82.4%), p=0.387.

Corrected

The wording available to the author is: "On these 920 tasks, with gemma-4-12b (thinking off) on Colab, pi 0.87.1, and no round cap, adding only the user's install command, no difference in the correct-answer rate was measured."

Supporting figures

Not installed: 748 / 920 (81.3%)
Installed: 758 / 920 (82.4%)
p: McNemar exact two-sided p=0.387
Denominator: 920 tasks × 2 groups = 1,840 cells, 3 tasks excluded by name

Source record: Vacant repo: decisions/conclusions/CONCLUSION_20260928_COLAB_C5_PI_VACANT361.md:8, 37-39

Round zero-2026-09-27 · Experimental result · verdict value: no_effect

zero.colab920_rejects_96pct_false96% of the send-backs in the 920-task batch were wrongfulRefutedrepo verdictGain axis · loop

96% of v3.6.1's send-backs on coding tasks were wrongful: 455 of 472 send-backs happened because it recognized only pytest and python -m unittest, not the sh run_tests.sh the agent had actually run.

Supporting figures

Send-backs: 472 (the denominator is send-backs, not runs)
Wrongful send-backs: test_claim/none 455
Other: failed_step 21, unsourced 1, missing_output 0
Denominator (the other one): of the 920 cells in group C, 858 reached the pre-handoff check and 309 were sent back
Cost: the wrongful send-backs did no visible harm, but cost about 12% more model calls

Source record: Vacant repo: decisions/conclusions/CONCLUSION_20260928_COLAB_C5_PI_VACANT361.md:57, 61-64

Round zero-2026-09-27 · Mechanism defect · verdict value: refuted

zero.u274_no_differenceu274 preregistered batch: 9 vs 10, no difference measuredNo discernible differencerepo verdictGain axis

On the first 100 of 274 tasks in seed order (all hard, never used before), 9/100 were correct without the install and 10/100 with v3.6.1 installed; McNemar exact two-sided p=1.0, no difference measured.

Corrected

Verbatim: "We do not say 'no effect' … even if all 7 pairs were C-only correct, p would be only 0.016; 6 to 1 is 0.125. This batch can only see very large differences." (original: 「不說『沒有效果』……就算 7 對全是 C 單獨對，p 也只有 0.016；6 比 1 是 0.125。這一批只看得到非常大的差。」)

Supporting figures

Not installed: 9 / 100 correct
Installed: 10 / 100 correct
Pairs: A right ∩ C right = 6, A right ∩ C wrong = 3, A wrong ∩ C right = 4, A wrong ∩ C wrong = 87
p: McNemar exact two-sided p=1.0
Time limit: 116 of 200 runs hit the 1800-second time limit

Source record: Vacant repo: decisions/conclusions/CONCLUSION_20260928_NOCAP_UNSEEN_U274.md:13-15, 19-22

How to recompute: That repo records that three independent recomputations agree

Round zero-2026-09-28 · Experimental result · verdict value: no_effect

zero.u274_missing_file_returns_all_wrongu274: 31 runs got a missing-file send-back, and not one was correct afterwardRefutedrepo verdictGain axis · loop

In the u274 batch the missing-file send-back fired in 31 runs (37 times in total), and not one of those 31 runs ended up correct — 17 wrong, 14 still not delivered. The extra answer files C delivered were all wrong, and C also hit the time limit 18 more times.

Supporting figures

Sent back: 31 runs (37 times in total; 6 runs were sent back twice)
Outcome afterward: 0 correct, 17 wrong, 14 still without a file
Trigger reason: 15 runs: the agent said it was done but there was no file; 16 runs: the round ended on a model-side error and was called back (15 runs failed to decode, 1 run Stream ended without finish_reason)
Time limit: 1800-second time limit: A 49 runs, C 67 runs (C 18 more)
Harm: 0

Source record: Vacant repo: decisions/conclusions/CONCLUSION_20260928_NOCAP_UNSEEN_U274.md:37-42

Round zero-2026-09-28 · Mechanism defect · verdict value: refuted

zero.v37_fixes_three_defectsv3.7 fixes three defects; send-backs drop from 309 to 29Heldrepo verdictGain axis · loop

v3.7 fixes three defects: it recognizes any test script (not only pytest), it also recognizes deliverables from the instruction file a person asks the agent to follow, and a round that ends in a model error is never sent back. On the Colab 920-task batch, the runs sent back at the first check dropped from 309 to 29.

Corrected

The author states plainly that v3.7's ceiling on the correct-answer rate is small: "Even if every one of the 10 new missing-file send-back runs on Colab were rescued, that is +10 tasks out of 920." The section's original title is literally "Expectation (stated, not tested)" (original: 「預期（寫明，不是檢定）」).

Supporting figures

Colab (858 runs checked): v3.6.1 sent back 309 runs at the first check → v3.7 29 runs
u274 (51 runs): 36 runs → 20 runs
S36-nocap (65 runs): 12 runs → 12 runs (unchanged)
R530 SOLO (59 cells): 35 cells → 32 cells (fully correct cells 6 → 4); share of send-backs with a real problem 0.83 → 0.875
Adversarial review: all 20 items reproduced; every one that would cause wrongful send-backs in everyday use was fixed; 31 regression tests
Offline fidelity: recomputing v3.6.1: Colab 1327/1327, u274 81/81, S36-nocap 80/80 checks all match

Source record: Vacant repo: decisions/DECISION_20260926_ZERO_CONFIG_V3.md:151-178

Round zero-2026-09-28 · Mechanism fix · verdict value: held

zero.v37_colab_p0028v3.7 on Colab: +11.3 pp measured (p=0.0028)Heldrepo verdictGain axis

On Colab's DABstep 711-cell batch, v3.7 was measured higher than no install: A 97/231 (42.0%), C37 123/231 (53.2%), C37R 139/231 (60.2%); primary test C37−A mean difference +11.3 pp, p=0.0028.

Corrected

The author says explicitly that this cannot be stated as "v3.7 is better than earlier versions"; what can be said is: "In this environment, 'saying it is done without writing the required file' is common, and zero-config's missing_output send-back can recover it." The agent without the install said it was done without writing the file in 42/231 cells.

Supporting figures

A, not installed: 97 / 231 (42.0%)
C37, installed: 123 / 231 (53.2%)
C37R: 139 / 231 (60.2%)
C37−A: +11.3 pp (C37 better on 26 tasks, A better on 12), p=0.0028
C37R−A: +18.2 pp (34 vs 7), p=3.1e-5, Holm 6.3e-5
Harm: 0
Denominator: 231 cells/group; 79 tasks × 3 runs
Share of answer files that were correct: A 81.5%, C37 76.4%, C37R 65.9% (more got delivered, and the share of them that was correct fell)

Source record: Vacant repo: decisions/conclusions/CONCLUSION_20261002_COLAB_DABSTEP_V37.md:9, 15-20

Round zero-2026-10-02 · Experimental result · verdict value: held

zero.v37_gguf_not_replicatedThe same v3.7 did not replicate on a GGUF backendRefutedrepo verdictGain axis

On the vacant-dev replication (LM Studio GGUF, gemma-4-12b, 15-round cap), the same v3.7's primary test was not significant: C37−A −0.4 pp, p=0.84. The Colab p=0.0028 did not replicate.

Corrected

The author's conclusion from the comparison: "one engine has it, one doesn't" — one cannot claim "zero-config v3.7 makes agents do better". The mechanism difference: on GGUF, send-backs cost rounds, the runs cut off at 15 rounds rose from 59 to 74, and the net effect is about 0; on Colab the amount recovered exceeded the amount cut off. The author's attribution: "The Colab p=0.0028 is mainly that inference engine making the agent without the install miss writing the file more often."

Supporting figures

A, not installed: 122 / 231 (52.8%)
C37: 121 / 231 (52.4%)
C37R: 146 / 231 (63.2%)
C37−A: −0.4 pp (C37 better on 11 tasks, A better on 13), exact p=0.84
C37R−A: +10.4 pp (21 vs 7), p=0.0051
Denominator: 231 cells/group; 79 tasks × 3 runs

Source record: Vacant repo (branch docs/gguf-replication-results, not yet merged into main as of 2026-10-03): decisions/conclusions/CONCLUSION_20261003_VACANTDEV_DABSTEP_V37.md:9-21, 28-34

Round zero-2026-10-03 · Experimental result · verdict value: refuted

zero.task3_no_differenceThree task-oriented banks, 94 tasks: no difference measuredNo discernible differencerepo verdictGain axis

On 94 tasks from three task-oriented banks, on Colab with no round cap, v3.7 measured no difference: A 136/188 (72.3%) vs C37 141/188 (75.0%), Wilcoxon p=0.36.

Corrected

The author's cross-batch summary: "On DABstep with 15 rounds, where 18% of A 'said done but wrote nothing', a difference could be measured; on the three task-oriented banks, A had only 5/120 (4%), and none could be measured. Errors in decisions, or runs cut off by the time limit, are things zero-config cannot see in the first place."

Supporting figures

A: 136 / 188 (72.3%)
C37: 141 / 188 (75.0%)
p: Wilcoxon p=0.36
By task bank: DABench 85.0%→88.3%, DataBench 80.0%→86.7%, Polyglot Python 54.4%→52.9%
Harm: 0
Denominator: 188 cells = 94 tasks × 2 runs

Source record: Vacant repo: decisions/conclusions/CONCLUSION_20261002_COLAB_TASK3_V37.md:20-24

Round zero-2026-10-02 · Experimental result · verdict value: no_effect

zero.reminder_default_offThe round-budget reminder is off by default, because the extra answers it produces are wrong more often than rightHeldrepo verdictGain axis

From v3.6 on, the round-budget reminder is off by default; whoever wants it adds --budget-reminder. The reason: the reminder raised the correct-answer rate from 51.9% to 59.3%, but the extra answers it produced were wrong more often than right.

Corrected

The author's reasoning, verbatim: "For someone who does not look at the score, one more unfounded answer is not an improvement." (original: 「對一個不看評分的人，多一個沒有根據的答案不是改善。」)

Supporting figures

Correct-answer rate: A 51.9% → C2 59.3% (local 09-26 main batch, 77 tasks × 3 runs per group)
Share of answer files that were correct: about 82% → 65% (same batch; verbatim from the decision document §十, no denominator given)
The same direction appeared once more: Colab 10-02 v3.7 batch: A 97/119 = 81.5%, C37 123/161 = 76.4%, C37R (reminder on) 139/211 = 65.9%. The denominators are the cells with an answer file in each group of that batch, not those of the batch in the line above

Source record: Vacant repo: decisions/DECISION_20260926_ZERO_CONFIG_V3.md:139-146; decisions/conclusions/CONCLUSION_20261002_COLAB_DABSTEP_V37.md:43

Round zero-2026-09-27 · Design decision · verdict value: held

zero.not_contradicts_pastZero-config's negative results do not contradict past dataHeldrepo verdictGain axis

The negative results of the three zero-config batches do not contradict the gains measured before, because exactly the same setting was never measured before. Everything that showed a gain before was a mechanism in which Vacant itself owns generation.

Corrected

The author explicitly forbids saying "fully consistent with past data" or "past data predicted this result" — only "does not contradict".

Supporting figures

Parts the zero-config C group lacks: no tests to run (DABstep answers have no visible tests), no resampling, no refused delivery — not one of the three parts
Mechanisms that showed gains before: executable acceptance + resampling + refused delivery, all with Vacant itself owning generation

Source record: Vacant repo: docs/ZERO_CONFIG_EVAL_REPORT_2026-09-26.md:25, 273-276, 279-281, 384

Round zero-2026-09-26 · Framing · verdict value: held

trace.two_layer_attributionTrace has two layers: only the fact layer can carry consequencesHeldrepo verdictDefense axis

Every trace conclusion has two layers: the fact layer (a rerun flips, success claimed but the command failed, a prohibition violated) may carry reputation consequences; the inference layer ("step k caused it", "it read bad input") only goes into the report.

Corrected

Honest boundary, verbatim: "'Provable' means this verifier flips on the reconstructed state; the verifier itself may be wrong, so a person can revoke it." (original: 「『可證明』的意思是這個驗證器在重建狀態上翻轉；驗證器本身可能錯，所以人可以撤銷。」)

Supporting figures

Layers: 2 (fact layer / inference layer)
Five levels: provable → record_review(0, weight=1.0) and no slash; lineage_exact → one count recorded against the source; lineage_internal → traced upward to the step that produced it; heuristic → report only; gap → no one is touched
The three necessary conditions for provable: rerun on the reconstructed state; it flips from pass to fail exactly at this step; the wrong value is not in any pinned input or observed read

Source record: Vacant repo: decisions/DECISION_20260924_ACCOUNTABLE_TRACE.md:141-145

Round trace-2026-09-24 · Design · verdict value: held

trace.lfake_attribution_correctTrace end-to-end attribution correct 33/33, 25/25, 16/16Heldrepo verdictDefense axis

Four real agents ran eight planted-error scenarios end to end, and attribution was correct (status/category/level/step pointed to/source all right) in 33/33; the same scenarios through vacant do were 25/25; the real interactive TUI was 16/16.

Corrected

This is L-fake: "The model is a fake upstream answering from a script; there is no real model and no API key." It proves the attribution mechanism is correct; it does not prove that output under a real model gets closer to the requirement.

Supporting figures

Native hooks: 33/33 (four agents × eight scenarios; G on Claude Code only, 1/1)
vacant do: 25/25 (D and H not run)
Real interactive TUI: 16/16 (including multi-round M 4/4, which really used up its rounds)
Extra time per hook: p95 6.3–13.3 ms (small workspace) / 6.7–12.6 ms (TUI batch)
Signed case-record chain verifies: 33/33, 25/25, 16/16

Source record: Vacant repo: ops/accountability/evidence_20260924/README.md:3-87

Round trace-2026-09-24 · Mechanism evidence · verdict value: held

trace.feedback_never_names_actorFeedback to the agent never contains an actor identityHeldrepo verdictDefense axis

In all three end-to-end tables, no actor identity appeared in the feedback given to the agent: native 0/33, vacant do 0/25, TUI 0/16. The targets of blame are steps and actors, but the feedback is shown only to people.

Corrected

This is a designed boundary, not a defect: read coverage is a lower bound, "no source found" may never on its own lead to "the agent's fault", and every report carries a three-state read_coverage.

Supporting figures

Feedback containing an actor identity: native 0/33, vacant do 0/25, TUI 0/16
Feedback with a location appeared in the model request: vacant do 25/25 (TUI batch 16/16; OpenCode's interactive TUI too)

Source record: Vacant repo: ops/accountability/evidence_20260924/README.md:8-87

Round trace-2026-09-24 · Honest boundary · verdict value: held

trace.review47_and_boundariesThe trace line's 47 adversarial review items and seven honest boundariesHeldrepo verdictDefense axis

The trace line went through 47 adversarial review items under five lenses: 46 upheld, 1 judged an existing honest boundary but tightened anyway; it lists seven honest boundaries, including three sentences that are explicitly forbidden.

Corrected

Honest boundary 1: tampering is detectable, not impossible — the chain is signed with a local key, and local root can re-sign the whole chain. Honest boundary 5: parallel steps can be attributed to only one of them.

Supporting figures

Review: 47 items, 46 upheld, 1 judged an existing honest boundary but tightened; 6 of them changed the design
The three forbidden sentences: "Vacant catches all errors", "the trace is always correct", "agents cannot get around it"
Permitted wording: in the recorded steps, this check flips from pass to fail at step k, and step k was done by X; the evidence can be rerun
Reviews at the same level: scale review 8 items, web-fetch instruction review 12 items, round-count/TUI review 12 of 15 items upheld

Source record: Vacant repo: decisions/DECISION_20260924_ACCOUNTABLE_TRACE.md:180-205

Round trace-2026-09-24 · Review · verdict value: held

trace.scale_perfThe first scan of 40,000 files takes 5.7 seconds; each hook after that 557 millisecondsHeldrepo verdictGain axis · loop

Trace scanning the whole workspace (incrementally) on a 4-core local machine stays within the hook's 30-second cap: first scan of 40,000 files 5.7 s, each hook after that p50 557 ms, p95 597 ms, Stop end to end 1.9 s.

Corrected

The boundary the author states: "The files are all small; local disk, warm cache — a network file system would be slower; the first scan of 4×10⁴ files is right up against the 8-second limit, and on a slightly slower machine it would move to the background." Beyond 5×10⁴ files it no longer scans step by step, and the trace is always marked UNOBSERVED.

Supporting figures

100 files: first scan 0.17 s, afterward 136/166 ms, Stop 0.17 s
1,000 files: first scan 0.29 s, afterward 144/194 ms, Stop 0.26 s
10,000 files: first scan 3.1 s, afterward 230/325 ms, Stop 0.56 s
40,000 files: first scan 5.7 s, afterward 557/597 ms, Stop 1.9 s
60,000 files: first scan 8.3 s (moved to the background; if the background cannot finish ⇒ turned off), afterward 135/227 ms, Stop 2.1 s
Diff index: 0.9 KB per step (previously 4.9 MB for 4×10⁴ files)

Source record: Vacant repo: decisions/DECISION_20260924_ACCOUNTABLE_TRACE.md:236-244

Round trace-2026-09-24 · Performance measurement · verdict value: held

tui.seven_measurement_errorsSeven measurement errors in the interactive A/B harness, every one making the numbers look better than they wereHeldrepo verdictGain axis · loop

The interactive TUI A/B harness had seven measurement errors, each one caught on real pi 0.87.1, and every one of them pointed in the direction of making the numbers look better than they actually were.

Corrected

All fixed, with tests added for five real pane shapes (multi-round with a final pass / single-round rejection / multi-round with a final rejection / infra_void / no gate).

Supporting figures

1 Arm A ran on a different model: copying only models.json does not make pi choose that provider; in practice it fell through to openrouter/moonshotai/kimi-k2.6, and not a single byte went through this run's proxy — arm A recorded "model failure" as "agent failure"
2 settle read the wrong signal: arm C reported settled as soon as the first-round banner appeared and then killed the session; the snapshot caught round two still thinking (tokens frozen at ↑5.9k ↓79)
3 Read only the visible height: about 50 lines vs the full 4000-line scrollback, and the earlier verdict had scrolled exactly out of view
4 Took the preamble for the verdict: the gate first prints 'acceptance source… snapshot sha256', which begins the same way as the verdict but contains no verdict, so every cell came back as gate_accepted: null
5 infra_void recorded as refused delivery: bool(None) filed "the gate gave no verdict" as "the verdict was rejection"
6 Two fields of the same record contradicted each other: a cell whose last round was 2/2 reported 1/2
7 A stall counted as agent failure: the model called a nonexistent tool ls_r, fell back to the shell and then produced nothing more, while the backend itself was healthy
2 more: all 8 cells of arm A stalled at 0.0% for the whole round; the prompt was never sent, yet they were recorded as settled + hidden_pass: false — 8 fabricated results; Enter was dropped in 6 cells

Source record: Vacant repo: ab_harness/tui_ab.py:104-427, ab_harness/tui.sh:37-60

Round tui-2026-09-30 · Measurement-tool defect · verdict value: held

pi.possess_three_defectsThe pi extension lacks three things; installing it is the same as not installing itRefutedrepo verdictGain axis · loop

The pi interactive-mode extension has three defects, two of which make it completely unusable: install reports success but every call sinks, and the model id does not match what is actually sent.

Corrected

In the real-machine test the accept leg was not verified: the agent did not write the file because the OpenRouter free tier's 1000 daily requests ran out, not because of a problem in the mechanism.

Supporting figures

Defect 1: install() did not pass upstreams/models/python, rendering KEY_FROM="" and UPSTREAM="" — the proxy came up but every call returned 502 unspecified_upstream_sink, with no error message pointing to the cause
Defect 2: wire_pi did not bake the model preference into default_model, so DEFAULT_MODEL was always gemma-4-12b-it-qat; when the upstream was a different model every call got 404, and the model name on screen did not match what was actually sent
Defect 3: the startup probe at install time reused the current os.environ; run from a non-login shell, node resolved to 18 and it concluded that pi could not start

Source record: Vacant repo: commit ad1282fe (2026-09-29)

Round possess-2026-09-29 · Product defect · verdict value: refuted

opencode.w2_silent_noopopencode has no stop event: a full 37-step case record, zero verdictsRefutedrepo verdictGain axis · loop

opencode's plugin did not register a stop event, so the end-of-round check was unreachable: one measured cell recorded a full signed case record of 37 steps, zero verdicts, and no delivery.md. The author says this is more dangerous than doing nothing, because the chain looks correct.

Corrected

W1 and W2 are the same design error: both put the verdict point inside the agent's process, and when the platform does not cooperate the whole path is not "a smaller effect" but silently ineffective. The author's decision is to move the loop from hooks to the parent process.

Supporting figures

Event counts: trace_genesis 1, prompt 3, session_seen 2, step 37, session_closed 3, unrecorded_change 1
stop/finding/feedback events: 0
delivery.md/delivery.json: do not exist
Denominator: 37 steps, 3 sessions

Source record: Vacant repo: decisions/conclusions/FINDINGS_20260929_NATIVE_PLATFORM_WAVE1.md:54-74

Round loop-2026-09-29 · Product defect · verdict value: refuted

wave1.unresolvedNative platform wave 1: Same sign, unresolved — not a negative resultSame sign, unresolvedrepo verdictGain axis · loop

Native platform wave 1 (20 tasks × three arms): the paired McNemar had only 2 discordant pairs, p=0.50, which the author recorded as unresolved — neither a negative nor a positive result.

Corrected

The author lists five reasons why no effect can be claimed, two of them structural: the free model's 1000 daily requests ran out mid-batch, so no cross-arm difference is attributable to Vacant; and wave 1 has a ceiling effect (arm A, without the install, got all hidden checks right on 95.0%, so the room for the gate to improve was 0).

Supporting figures

A plain: all hidden checks right 19/20 (95.0%), delivered 20/20, refused delivery 0
B zero-cfg: 17/20 (85.0%), delivered 19/20, refused delivery 1
C bridge: 17/20 (85.0%), delivered 17/20, refused delivery 3
A vs B: 2 discordant pairs, p=0.50 → unresolved
Denominator: 20 tasks

Source record: Vacant repo: decisions/conclusions/FINDINGS_20260929_NATIVE_PLATFORM_WAVE1.md:18-24

Round loop-2026-09-29 · Experimental result · verdict value: unresolved

intake.five_silent_defectsEnd-to-end runs of the universal intake found five defects that "make you think it works when it doesn't"Refutedrepo verdictGain axis · loop

End-to-end runs of the universal intake found five defects, two of which unit tests cannot catch: Codex refuses to start, and OpenCode's additive layer silently drops the user's global rules from the prompt.

Corrected

Wording correction: "tamper_proof": true in AGENTS.md/README×3 changed to false + tamper_evident: true (the signing key is a plaintext file under the same account). The class of issues where malicious candidate code shares an interpreter with the driver and the nonce is in argv is not claimed as fixed — "this layer protects against accidents, not against malice".

Supporting figures

1: when the hook wrote the ledger first, it created the verifier key but did not register trust for it, and the intake side correctly refused to let it through
2: vacant do put its isolated workspace under $VACANT_HOME, and pi's global extension correctly blocked the writes
3: the Codex resident install used profile = "vacant", which 0.156.1 treats as a legacy key and refuses to start — an install that keeps the agent from starting is worse than no install
4: OpenCode used OPENCODE_CONFIG_DIR as the per-run additive layer, which also changed where the global AGENTS.md is read from, so the user's global rules silently disappeared from the prompt
5: with the resident install still in place, vacant do injected another copy, so the same event ran twice
Unit tests: 177, all offline, zero models
Adversarial review: of 60 items, every one that would let a non-conforming version reach the destination or make the record say the wrong thing was fixed; another 7 explicitly will not be fixed

Source record: Vacant repo: decisions/DECISION_20260924_UNIVERSAL_INTAKE.md:74-89

Round intake-2026-09-24 · Product defect · verdict value: refuted

#Where the verdicts cluster

This section covers only the 34 claims in the project's official index (every count in parentheses below uses these 34 as the denominator). The 24 repo verdicts are in the full ledger above; use the filter bar to slice by "axis" or "verdict" to see them.

#Refuted (6 of 34) — all on the defense axis and "equal budget"

ID	Claim	Why it was refuted
`pulse.crossover`	Once the blind spot exceeds 0.5–0.75, pulse does worse than sustained misbehavior	Paired difference −0.57, 95% CI spans 0; measured pulse scores at blind spot 1.0 were 81/84/87, all above patient's 79
`pulse.audit_cannot_close_blindspot`	Raising the audit rate can't rescue a situation with a blind spot	At blind spot 0.5, audit 0.1→1.0 cut total successful attacks by 82%, Wilcoxon p=5.5e-6
`pulse.crude_wins_under_blindspot`	Under an equal budget, crude beats pulse	E19 was never an equal-budget comparison; actual misbehavior was 11.97 vs 4.87. Properly pinned at budget=2: 1.67 vs 0.97, p=0.0015
`gain.equal_budget_on_beats_off5`	Under an equal budget, Vacant beats five-try majority	Full paired p=0.4531; hard-task subset p=1.0
`gain.ceiling_hypothesis`	A weaker model would reveal the gain	OFF failure rate 25.5%→31.8%, gain still did not appear; ON vs OFF5 p=1.0000
`gain.hard_bench_hypothesis`	A harder task bank would reveal the gain	The bank really was harder (48.4% vs 31.8%), ON vs OFF5 p=0.4244

#Overstated (4 of 34)

ID	Problem
`pulse.timing_minor`	"Only a 1.81× difference" is an aggregate. Broken down: exposure spans 6.06× and efficiency spans 8.45×, cancelling each other out. Statistical power is only 25%
`pulse.blindspot_dominates`	Held up almost entirely by the degenerate endpoint blind=1.0. Interaction η² = 0.264, 4 times the main effect
`pulse.starvation`	"Permanently removed" was confounded with a coincidence: 14 seeds in the same cell were never caught. slash 0.9 came back in round 177
`gain.clean_dataset`	"The first clean complete dataset" should become "no known bug contamination, paired subset n=101." ON-arm void rate 36.9%

#Same sign, unresolved (4 of 34)

These four are the easiest to misread as "no difference." The correct reading: the preregistered rule did not trigger a single state, so nothing was measured.

ID	Claim	Why it wasn't measured
`harness.hmix_loop_beats_resample_same_budget`	The revision loop delivers more than resampling and spends no more tokens	0/5 pass Holm; all five intervals intersect R460
`gain.conform_vs_off5_unresolved`	The gate beats majority vote on delivery accuracy	MDE@371 = 4.31 pp; 80% power needs about 491 pairs, but the task bank has only 378
`harness.majority_vote_loses_to_gate`	With the same five calls, voting loses to the gate and costs more than twice as much	Exploratory, not in any preregistered family; 3/5 significant uncorrected
`harness.hpi_hoc_vs_resample_unresolved`	The other two loop arms also beat resampling	10/10 cells fail Holm

#No discernible difference (1 of 34)

gain.arms_differ: the three arms differ discernibly in delivery quality. Not significant under either measure, and fixing an unrelated typing bug flipped the point estimate.

#The four hardest of the Held (13 of 34)

ID	Why it's hard
`harness.loop_beats_single_shot_replicated`	15/15 cells all pass their own Holm, across five replications and four mutually exclusive task sets
`gain.gate_rule_beats_majority_vote_same_candidates`	Same set of candidates, same five calls — budget and candidates are both aligned
`gain.lossless_visible_filter`	Six datasets, 2,389 candidates, zero cases of "would pass hidden but fails visible"
`peerexec.mutual_execution_below_threshold`	Liars named 273/273, honest participants falsely accused 0/1,840, 5,519 testimonies with 0 signature failures

#What this table does not do

It does not judge whether this project is good. It only records "which statements still stand as of today."
It does not count Not re-verified as passed. The 6 Not re-verified claims are Not re-verified.
It does not merge, average, or pick one run to report. The banned list is in the five-replications post

#Next

How to rerun each number: Reproduction
The methodological premise of this table: Experimental method and verdict values
The full story of each claim: Lab notes