Zero-config Vacant

Zero-config is the v0.9.0 product line. Its design goal is not to do a lot, but to let you work as usual once it's installed — no contracts to write, no settings to add, no change of workflow.

#What it does

The core is a pre-handoff check: at the moment the agent says "done," it checks whether the files the task asked for exist, whether tests were run, whether a failed step was skipped, and whether there are numbers of unknown origin.

The design principle is "send back only problems that are almost certainly real; write everything else into the handoff note only":

Everything else is written into the "handoff note" for a person to read, not sent back.

The round-budget reminder has been off by default since v3.6. The reason is a trade-off that can be measured:

The reminder raised the correct-answer rate from 51.9% to 59.3%, but the share of delivered answers that were correct fell from about 82% to 65%. The author's reasoning: "For someone who doesn't look at the score, one more answer with no basis is not an improvement."

#Results of the four batches

#① DABstep paid batch: neither model showed a measured difference

77 tasks, one run per task per group:

In the preregistration's terms, this is called no measured difference, not "no effect."

All 8 send-backs in that round were false positives:

Why? Of the 82 failed runs, 54 were "15 rounds used up and the answer file still doesn't exist" — the gate's check never even got its turn. The remaining wrong answers had a basis for every step; they were wrong in "whether this decision is right," which a check that only looks for a basis cannot see, by design.

#② Local 711-cell batch: p=0.021, but concentrated in 6 tasks

79 tasks × three groups × 3 runs, with 77 tasks used in the main analysis:

The preregistered primary test gave p=0.0207, with a mean difference of +7.4 pp. But this p-value cannot be quoted on its own:

The author's sensitivity analysis: remove any two of the 6 tasks and p goes above 0.05; remove all 6 and the mean difference is 0.009, p=0.80. The whole difference comes from the layer "reached round 14 and still has no answer file," and the v3-3 mechanism never fired once:

Moreover, v3 was designed after looking at the records of the same 77 tasks from the previous round, so the tasks and the rules are not independent. No human signed off on that preregistration item by item.

#③ Colab, 920 tasks from five programming task banks: no measured difference

p = 0.387.

The most important finding of this round was not in the correct-answer rate but in the send-back mechanism itself:

Of 472 send-backs, 455 happened because it recognized only pytest and python -m unittest, not the sh run_tests.sh the agent had actually run. The agents in those 293 cells clearly had run the tests.

#④ u274 preregistered batch: 9 vs 10

The first 100 of 274 tasks in seed order, all hard, none used before:

Not installed 9/100, installed 10/100, McNemar exact two-sided p = 1.0.

The original: "Even if all 7 pairs were C-only correct, p would only be 0.016; 6 to 1 is 0.125. This batch can only see very large differences."

And the missing-file send-back did not help at all in this batch:

#v3.7 fixes three defects

After three batches had measured no difference, the author went back and fixed three things:

1. test_claim recognizes any test script — not only pytest; anything executed in the command position whose file name starts or ends with test counts. There is also a finer rule: "a test script goes red, and a later test run goes green = fixed, which does not count as a skipped failed step; one that never goes green is still sent back."
2. Deliverables are also recognized from the instruction files a person asks the agent to follow — only commands directed at the agent count; negated, optional and example items do not.
3. A round that ends in a model error is never sent back — it is only written up for a person to read.

Effect:

But the author states explicitly that v3.7's ceiling on the correct-answer rate is small: "Even if every one of the 10 new missing-file send-back runs on Colab were rescued, that is +10 tasks out of 920."

#v3.7 follow-up measurements: a difference measured, gone on another engine

On 2026-10-02, v3.7 measured a difference on Colab for the first time:

C37 − A mean difference +11.3 pp, p = 0.0028.

On 2026-10-03, the same v3.7 was replicated on an LM Studio GGUF backend:

C37 − A −0.4 pp, p = 0.84.

The author's conclusion from comparing the two is "one engine has it, one doesn't," with a mechanism explanation: on GGUF, send-backs cost rounds, the number of runs cut off at 15 rounds rose from 59 to 74, and the runs rescued and the runs cut off cancel each other out exactly.

The three task-oriented task banks, 94 tasks (no round cap), likewise measured no difference:

The cross-batch mechanism summary: with DABstep at 15 rounds, 18% of agents without Vacant "said they were done but never wrote it," and a difference could be measured; the three task-oriented banks had only 4%, and no difference could be measured.

#Why this does not refute CONFORM

This is the most important section of this page.

Zero-config's group C lacks three parts, and these three are exactly what the mechanisms that measured gain in the past provided:

So the correct statement is: the path of "saying it once within the same session" measured no difference in three of the four batches. Everything that showed gain in the past was a mechanism in which Vacant owns generation. This is not a contradiction; two different mechanisms were measured separately.

The original also explicitly forbids saying "fully consistent with the past data" or "the past data predicted this result" — we can only say it does not contradict it.

#Honest boundaries

1. v3-2 / v3-3 act only when the agent has been told the round cap. In ordinary interactive use (pi has no cap by default) they almost never trigger.
2. The reminder only checks whether a file exists; it does not say whether the answer is right.
3. The simulated numbers for the paid batch assume the model complies, and they are for gemma-26b with thinking on; the local gemma-4-12b without thinking is a different model.
4. The evidence from the simulated-user group is L-fake: a script decided what the model did after seeing a send-back. It proves "the send-back gets through, a fix gets seen, and nothing changes at all when there is no problem"; it does not prove that a real model will comply.
5. R530 closed on 2026-09-28 (32 cells sent back, 4 of them cells that were entirely correct), and is no longer an open item.

The full record of zero-config's four batches, including what each batch cannot be used to say

#Next

• The path that owns generation: Acceptance gate CONFORM
• Where the signal comes from: The executable acceptance layer
• What happens when the platform doesn't cooperate: Native platforms and silent ineffectiveness