The measuring tool got it wrong seven times, and every error leaned against itself
The A/B harness for the interactive TUI had seven measurement errors, plus two more that exist only in code comments. Every one was caught on a real pi, and every one pushed the numbers to look better than they really were — 8 fabricated results, and 5 cells whose prompts were never sent.
This post records the measuring tool's own errors.
The first sentence of commit b0eb0680 is explicit:
"This is not cosmetic formatting. Every one was caught on the TUI of a real pi 0.87.1, and every one points in the direction of making the harness unfavorable to itself or favorable to the agent — in other words, these numbers would otherwise have looked better than they really were." (original: 「這不是排版修飾。每一條都在真 pi 0.87.1 的 TUI 上抓到，而且每一條的方向都是讓 harness 對自己不利或對 agent 有利，也就是說這些數字原本會長得比實際好看。」)
"The same direction" is the key. If the seven errors had pointed in random directions, that would be noise; if they all point the same way, it is systematic — meaning this harness has a structural problem.
#Why an interactive TUI is especially hard to measure
An interactive interface has no API to call. Everything to be measured (where the gate banner appears, which turn the model has reached, whether it has settled) has to be scraped off the screen.
The method is a tmux pseudo-terminal: open a pane running the agent, wait for it to stop, grab the screen with capture-pane, and find the verdict in it with a regex.
That is where all seven errors came from — every one sits in the layer of "how to read the truth off the screen."
#Error 1: arm A ran on a different model
To measure "how the agent performs without Vacant installed," you need an arm with nothing installed at all. The problem: the model connection comes from configuration, not from launch arguments.
The approach was to copy models.json into the test environment. But that doesn't make pi choose that provider. Without a pin at launch, pi uses its own default.
In practice it landed on openrouter/moonshotai/kimi-k2.6 — a different model, a different host, and not a single byte went through this run's proxy.
The consequence: arm A recorded "model failure" as "agent failure." Every cell of the whole A arm was measuring the wrong thing.
The code comment says:
⚠ THIS IS THE WHOLE A/B, AND I GOT IT WRONG THE FIRST TIME.
The fix: each cell records seen_model, and the report can refuse to compare cells on different models (the INVALID field).
Interestingly, the repo itself had already stepped into the same hole on 2026-09-22, recorded at gateshim.py:578-582. It isn't a new problem; nobody had connected the two.
#Error 2: settle recognized the wrong signal
The original rule: arms B/C report settled as soon as they see the gate banner.
The problem is that arm C is the only arm that runs multiple rounds. It reported as soon as the first round's banner appeared, then immediately killed the session. The snapshot caught the second round still thinking — tokens frozen at ↑5.9k ↓79 — while the first round's verdict had already been pushed out of the visible area.
That cell read settled: true together with gate_accepted: null.
The code comment:
Both halves are the harness lying. Not that arm. (original: 「兩半都是 harness 在說謊。不是那一臂在說謊。」)
The fix: capture the full scrollback, poll and count, and settle only after the TUI goes back to idle.
#Error 3: reading only the visible height
By default, capture-pane gives only the roughly 50 visible lines. We switched to -S -4000 to get the full scrollback.
Why this one makes the numbers look better: in a multi-round session, earlier verdicts scroll off the screen, and those are exactly the variable being measured. Reading only the visible height means systematically seeing only the last verdict — and the last verdict is usually the more favorable one.
#Error 4: mistaking the preamble for the verdict
The gate first prints this line:
Vacant 閘門：驗收來源 dir:tests_visible（…）；快照 sha256 …The real verdict looks like this:
✓ 可見驗收全過（2/2）
✗ 拒交裁決（visible_fail，可見驗收 0/2）——互動模式不擋檔案：…Both begin with Vacant 閘門： ("Vacant gate:") — the openings are identical — but the first one carries no verdict.
The original regex recognized only the opening, so saw_gate was true and arm C declared settled right after the preamble. Every cell came back as gate_accepted: null.
This was the most serious error in the batch, because it is all or nothing: it doesn't misread one cell; it can't read any cell at all.
The fix: both regexes require a marker (✓ / ✗ / ⚠).
Plus a pitfall in the shape of the parentheses: the refusal banner reads 可見驗收 0/2） ("visible acceptance 0/2)") — comma + space, no opening parenthesis (the opening parenthesis was used up on the stop reason) — and only the pass banner is 可見驗收全過（2/2） ("all visible acceptance passed (2/2)"). Write the regex to the pass shape alone, and every refusal parses as None.
#Error 5: infra_void recorded as a refusal
In Python, bool(None) is False. So:
gate_accepted = None (the gate gave no verdict)
↓
False (recorded as "refused by verdict")This is exactly what the repo's own infra_void rule explicitly forbids: turning a void into a counted failure. A cell in which nothing happened and a cell whose delivery was refused mean completely different things.
The code comment:
Noneis a real value here, not a missing one.
#Error 6: two fields in the same record contradict each other
visible was first set from "the last pass," then overwritten by "the last refusal."
The consequence: a cell whose last round was a full 2/2 pass reported 1/2.
The fix: decide it once, from the last banner.
This one is smaller, but the direction is the same — it reports the more pessimistic result.
#Error 7: a stall counted as an agent failure
The measured signature: the model calls a nonexistent tool ls_r, falls back to the shell, and then stops producing output. Meanwhile the backend itself is healthy (it answers say ok 2 seconds later).
This is a void, not a failure: a failed model call is neither a right answer nor a wrong one.
The code comment:
A failed model call is neither a right answer nor a wrong one; it is a void, and a void is re-run, not counted.
The fix: rerun in a clean workspace, never score it, and record attempts / voids. MAX_ATTEMPTS = 3.
#Two that exist only in code comments
The commit message doesn't list these two; I found them while reading the comments in ab_harness/tui_ab.py.
#All 8 cells of arm A were fabricated results
Before anything is sent, pi's status bar shows 0.0%/262k; ↑N ↓M appears only after it starts.
In the 24-cell batch, all 8 cells of arm A stayed at 0.0% for the entire run: the prompt was never sent, and the TUI just sat there. And the old arm-A settle rule ("idle 4 polls" — satisfied immediately) recorded every one of them as settled + hidden_pass: false.
The code comment:
That is 8 fabricated results. (original: 「那是 8 個捏造的結果。」) The only thing that would have caught it is this check.
Every cell of arm A was wrong — not a biased number; the cells themselves did not exist.
#Enter was dropped in 6 cells, and I got the check wrong twice
On startup, pi prints:
Warning: tmux extended-keys is off. Modified Enter keys may not work.In the first 24-cell batch, Enter was dropped in 6 cells, all with the same shape: the prompt left sitting in the input box, the status bar at 0.0%/262k, no turns, no gate, no verdict. Those cells were recorded as timeout.
This is why arm B appeared to void 5 of its 8 cells, when the truth is that in 5 of the 8 cells the prompt was never sent.
The author recorded getting the detection method wrong twice:
- Checking with
grep -F "$prompt"— pi echoes the prompt into the conversation transcript, so the text is always on the screen and this check can never pass. So he resent Enter ten times and didexit 2in every cell — including cells whose screen clearly showed↑2.8k ↓34and the correct provider. That would have reported arm A as 8/8submit_failed, when arm A had in fact sent every time. tmux set -g extended-keys on— measured 3/3 sent with it off and 3/3 sent with it on, so it was never the problem, and it changes how Enter is encoded.
The code comment:
both recorded because the second one is worse than having no check at all
The second wrong method is worse than having no check: it reports right answers as wrong.
#After the fixes
read_verdict has tests for five real pane shapes:
- Multi-round, last round passes
- Single round, refused
- Multi-round, last round refused
infra_void- No gate (arm A)
ruff is clean.
But that batch's numbers cannot be used — 8 of arm A's cells were fabricated, and in 5 of arm B's cells the prompt was never sent. Now that it is fixed, the batch has to be rerun.
After 3 arm-A cells, the remote vacant-dev became entirely unreachable (port 22, ICMP, and tailscale ping all unresponsive, while other machines on the same subnet were fine). "Those cells have no backup; once it recovers, pull them back first." (original: 「那些格沒有備份，恢復後要先拉回來。」)
#What I learned from these seven
This section is my own judgment.
1. Errors that point the same way are the point.
If the seven had pointed in random directions, that would be noise. But "every one made the numbers look better than they really were" means a structural problem — specifically, this harness systematically prefers "it looks like something happened" over "it looks like nothing happened."
This is a common ailment of scrapers. When they catch nothing, people (and programs) tend to record it as "failure" rather than "didn't happen," because the former feels like having data.
2. "Silently not taking effect" deserves to be reported as an error even more than "a smaller effect" does.
If the W2 item (opencode has no stop event) were read as "tracing has a smaller effect on opencode," that would be an acceptable conclusion, and nobody would go and fix it.
But what actually happens is a full signed case history of 37 steps, zero verdicts, and no delivery.md. It looks completely normal.
A full signed case history of 37 steps, zero verdicts. That is more dangerous than doing nothing, because the receipt will look right.
3. Bugs in the measuring tool should be held to the same standard as bugs in the product.
In the commit messages these seven are "eval(tui)," which looks like the same category as "feat(eval)." But methodologically they are no different from product defects — both are "things that make you think something holds when it doesn't."
Named dissent: why we write disagreements into the record | R455's structural upper bound | Native platforms and silently not taking effect
