Zero-config, four batches: three measured no difference, and the fourth vanished when we switched engines

This post records four batches. None of the four looks good, and each looks bad in a different way — that is the point.

#First, what exactly was measured

Zero-config is the v0.9.0 product line. Its goal is not to do a lot; it is to let you keep working as usual once it's installed. No contracts to write, no settings to add, no change of workflow.

It does just one thing: at the moment the agent says "I'm done," it sends one message back within the same session.

The difference from the acceptance gate CONFORM is not old version versus new. CONFORM owns generation: every candidate is run against the visible acceptance tests, a fail means resampling, up to five candidates, and if none passes it refuses delivery. Zero-config does not own generation: it has no executable signal, no power to retry, and no power to refuse delivery.

Put this difference on the table first; only then do the results of the four batches below make sense.

#Batch one: the DABstep paid batch (2026-09-25 / 26)

77 tasks, one run per task per group, costing US$3.40 / 4.80 and 3,324 calls.

In the preregistration's terms, this is called no measured difference. Not "no effect," and not "worse."

#All 8 send-backs were false alarms

This is the thing most worth recording from this round:

Actual false-alarm rates: gemma 6/37 (16%), qwen 1/64 (2%) — the denominator is "runs that reached the pre-submission check for the first time while the answer at that point was correct."

Harm: 0 times. Not once did it turn a correct answer into a wrong one.

The author wrote a line of self-assessment in the report:

"Before spending money, the false-alarm threshold was built only on replays of real records from 3 correctly answered runs; that threshold is too thin." (original: 「花錢之前的誤報門檻只建立在 3 跑答對的真實紀錄重播上，這個門檻太薄。」)

3 runs. That is a sample size on the design side that is too small, not a statistical problem.

#54 of 82 failures were "never submitted"

Breaking down the causes of failure:

Two thirds of the failures were "the answer file didn't exist yet when the agent was cut off by the turn limit." Vacant's pre-submission check runs only when the agent itself says it's done, so its turn never comes.

The remaining 22 reasoning errors are the most damaging kind: every step has a basis (the values were computed by the agent, the manual was opened, no step failed), and the error lies in "whether this decision is right." The author wrote: "A check that only looks at 'is there a basis' cannot see this, by design." (original: 「只看『有沒有根據』的檢查從設計上看不到。」)

Examples: 7 runs computed the "fraud rate = share of amount" that the manual spells out as a share of transaction count instead; 5 runs computed the correct answer themselves and then chose another interpretation.

#Batch two: 711 local cells (2026-09-26)

79 tasks × three groups × 3 runs = 711 cells. The main analysis uses 77 tasks.

The preregistered primary test gives p = 0.0207, mean difference +7.4 pp, bootstrap 95% CI 0.017–0.139.

It looks like we finally measured something. But this p-value cannot be cited on its own.

#What "the significance is thin" means

The author's sensitivity analysis (post hoc):

Counted by task, 15 tasks were better and 7 were worse — sign test p=0.13, not significant.

The original wording:

"The significance is thin: flip the results of two runs, or remove any two of those 6 tasks, and p goes above 0.05. We do not extrapolate to other models, other task banks, or uncapped use." (original: 「顯著性很薄：兩跑的結果翻過來、或拿掉那 6 題裡的任兩題，p 就會超過 0.05。不外推到別的模型、別的題庫、沒有上限的使用。」)

#None of the secondary comparisons is significant

• C1 − A: +0.013, p = 0.75
• C2 − C1: +0.061, p = 0.083, Holm 0.17

Of the three runs, only run 2 is significant on its own (12 vs 3, p=0.035). That is because "the 6 large differences are all positive, and the 7 negative ones are all −1/3."

#Two self-imposed limits

1. v3 was designed after seeing the records of the same 77 tasks from the previous round. Item 2 of section seven of the preregistration states this limit itself. The tasks and the rules are not independent.
2. No human signed off on this preregistration item by item. The author wrote that it "must be signed off before being presented externally as 'proof'" (original: 「對外當成『證明』之前要補簽」).

#The gauge itself was wrong too

research.wilcoxon_signed_rank_exact used floating-point == to detect ties, so it treated 2/3−1/3＝0.33333333333333337 and 1/3＝0.3333333333333333 as different values and ranked them wrong. The frozen function computed 0.0189; handled correctly, it is 0.0207.

Both are < 0.05, so the conclusion doesn't change. But this is a bug in the measuring tool, not a problem with the data.

#v3-3 never fired once

Of all 150 times C2 reached the pre-submission check, not one happened after turn 14.

#Where the difference came from, and what it cost

The entire net gain sits in the layer "reached turn 14 with no answer file yet": accuracy in that layer was A 4/79, C1 3/78, C2 26/82 (+22 runs). 63 runs wrote the file after the reminder, all of them in the turn right after the reminder.

But: of those 63 runs, 25 were right and 38 wrong (another 19 runs were reminded and still wrote nothing).

The author sums up the trade-off in one line:

"The cost is that more wrong answers were submitted. For someone who doesn't look at the grading, this means receiving more wrong answers." (original: 「代價是多交了錯的。對一個不看評分的人，這代表會收到更多錯的答案。」)

That is why from v3.6 on, the reminder is off by default.

This is an improvement in success rate, not in token efficiency

#Batch three: 920 tasks from five Colab code task banks (2026-09-27)

LCB v1 89, LCB v2 118, LCB v3 189, MBPP+ 369, HumanEval+ 155: 920 tasks in total × 2 groups = 1,840 cells.

McNemar exact two-sided p = 0.387.

#But the most important thing this round was not accuracy

Of the 920 cells in group C, 858 cells reached the pre-submission check and 309 cells were sent back. Of the 472 send-backs:

Those 455 happened because v3.6.1 only recognized pytest and python -m unittest, not the sh run_tests.sh the agent actually ran. In 293 cells the agent clearly had run the tests.

The author's description: "the false send-backs did no visible harm, but cost about 12% more model calls" (original: 「誤退沒有看得出來的傷害，但多花約 12% 的模型呼叫」).

Only the 289 falsely sent-back cells: C passed 264, A on the same tasks passed 253 — C did 11 cells better than not installed. The 20 cells genuinely sent back: C eventually passed 18, A on the same tasks passed 12.

But both of these lines are post hoc description, not a preregistered test, and we draw no conclusion from them.

#Batch four: u274 (2026-09-27 / 28)

The first 100 of 274 tasks in seed order, all hard, never used before.

Not installed 9/100, installed 10/100, McNemar exact two-sided p = 1.0.

Paired table: A right ∩ C right = 6, A right ∩ C wrong = 3, A wrong ∩ C right = 4, A wrong ∩ C wrong = 87.

#"This batch can only see very large differences"

The original:

"We do not say 'no effect': the tasks are very hard (both groups at about 10% accuracy), 116 of 200 runs hit the 1800-second time limit, and there are only 7 discordant pairs — even if all 7 were C-only correct, p would only be 0.016; 6 to 1 gives 0.125. This batch can only see very large differences." (original: 「不說「沒有效果」：題目很難（兩組答對率約 10%）、200 跑裡 116 跑碰到 1800 秒時限，不一致的配對只有 7 對——就算 7 對全是 C 單獨對，p 也只有 0.016；6 比 1 是 0.125。這一批只看得到非常大的差。」)

This sentence is more useful than p=1.0 itself. It tells you where this batch's statistical power lies.

#The missing-file send-back didn't help at all in this batch

What's more, 16 runs were called back after a model error (15 runs failed to decode, ret = 1, 1 run Stream ended without finish_reason) — calling back a model failure does not turn into an answer; it only burns more turns.

Harm: 0 times, but C submitted 13 more wrong answers and hit the time limit 18 more times.

#v3.7: fixing three defects

After three batches measured no difference, the author went back and fixed three things:

1. test_claim recognizes any test script — it counts as long as it is executed in the position of a command and its file name starts or ends with test. There is a further rule: "A test script goes red, then a later test run is green = fixed, and does not count as a skipped failed step; one that never goes green is sent back as before." (original: 「測試腳本紅了、之後又有一次跑測試是綠的＝修好了，不算失敗的步驟被略過；一直沒綠的照舊退回。」)

This item corresponds directly to the 455 false send-backs in batch three.

2. Deliverables are also recognized from the instruction files a human wants the agent to follow — only commands addressed to the agent are taken; negations, optional items, examples, and descriptions of program behavior don't count; files > 64 KB and files changed in this turn are not read; more than 3 in one file = data, and none of them counts.
3. A turn that ended in a model error is never sent back — it is only written down for humans to see. This item corresponds to the 16 runs in u274.

Effect:

All 20 adversarial-review items reproduce, and everything that would be falsely sent back in everyday use is fixed. 31 regression tests.

But the author spells out the ceiling explicitly: "Even if all 10 missing-file send-back runs newly added on Colab were rescued, that is +10 tasks out of 920." (original: 「Colab 新增的 10 跑缺檔退回就算全部救回，也是 920 題裡 +10 題。」) The original heading literally reads "Expected (stated, not tested)" (original: 「預期（寫明，不是檢定）」).

There is also offline fidelity: v3.6.1 recomputed 1327/1327 checks on Colab, 81/81 on u274, and 80/80 on S36-nocap, with actions and categories identical to the records and fully aligned.

#2026-10-02 / 03: measured, then not replicated

The first real measurement after v3.7 went live measured a difference on Colab:

C37 − A mean difference +11.3 pp, p = 0.0028.

The next day, the same v3.7 was replicated on an LM Studio GGUF backend:

C37 − A −0.4 pp, exact p = 0.84.

#Why this happened

The mechanistic difference between the two batches is that "a send-back uses up turns":

• Colab (vLLM): the agent without it said it was done but hadn't written the file in 42/231 cells. What was recovered outweighed what was cut off.
• GGUF: the same missing_output worked (said-done-but-didn't-write went from 20 down to 2), but a send-back uses up turns — the number cut off by the 15-turn limit rose from 59 to 74. Net effect about 0.

The author's conclusion comparing the two: "One engine has it, one doesn't" (original: 「一個引擎有、一個沒有」), with an explicit ban on claiming "zero-config v3.7 makes the agent do better" (original: 「零設定 v3.7 讓 agent 做得更好」).

#The three task-oriented banks also measured no difference

94 tasks, Colab, no turn limit: A 136/188 (72.3%) vs C37 141/188 (75.0%), Wilcoxon p = 0.36.

The mechanism summary across batches:

"DABstep at 15 turns, where 18% of A 'said done but didn't write,' gets a measurable difference; in the three task-oriented banks, where A had only 5/120 (4%), it isn't measurable. Errors in the decision, or being cut off by the time limit, are things zero-config cannot see in the first place." (original: 「DABstep 15 回合、A 有 18%「說完卻沒寫」就量得到差；任務導向三庫 A 只有 5/120（4%）就量不到。錯在決定、或被時限切斷的，零設定本來就看不到。」)

#This does not refute CONFORM

This is the most important section of the whole post.

Zero-config's group C is missing three components:

The report's original text:

"This round's zero-config group C has none of the three components: no tests to run, no resampling, no refusal of delivery." (original: 「這一輪的零設定 C 組三個零件一個都沒有：沒有測資可以跑、不重抽、不拒交。」)

So the correct statement is: "say it once within the same session" — on that path, three of four batches measured no difference; what showed gains in the past were all mechanisms in which Vacant owns generation.

This is not a contradiction — two different mechanisms were measured separately.

The original text also explicitly forbids saying "fully consistent with past data" (original: 「和過去的資料完全一致」) or "past data predicted this result" (original: 「過去的資料預測了這個結果」). We can only say they don't contradict each other.

#What I learned from these four batches

This section is my judgment, not an experimental result.

1. "No measured difference" and "no effect" are far apart, and this batch only supports the former.

In the u274 batch, accuracy was about 10%, more than half of the runs hit the time limit, and there were only 7 discordant pairs. The author worked it out in the report: even with all 7 pairs C-only correct, p would only be 0.016. This batch's statistical power is only enough to see very large differences. Using it to say "zero-config is useless" is just as unfounded as using it to say "zero-config is useful."

2. "False send-backs" should be fixed before "missed send-backs."

In the 920-task batch, 96% of send-backs were false; in u274, the 31 missing-file send-back runs ended up 0 right and 17 wrong. Together these two numbers say one thing: a gate that wrongly hurts can have a negative net effect until it learns not to — even if its mechanism design is right.

3. The same version number is two different things on different inference engines.

Colab's p=0.0028 and GGUF's p=0.84: the same v3.7, the same set of tasks, the same turn limit. The author can explain why (a send-back uses up turns, and how many turns a send-back can use depends on how fast the engine consumes turns), but it means the sentence "v3.7 works" cannot be said without conditions.

4. Measuring tools mismeasure too, and always in the same direction.

The floating-point tie bug in batch two's gauge, batch three recognizing only pytest, the seven measurement errors in the TUI harness — every one of them made the numbers look better than they really were. This is expanded in The measuring tools mismeasured seven times.

#Next

Zero-config Vacant | The acceptance gate CONFORM | The executable acceptance layer