Gain axis

#What this axis answers

Given that requirements can already be compiled into executable acceptance tests: how much gain do the acceptance gate, resampling, and failure-feedback revision each bring to the delivery success rate, and at what token cost.

#The start: the committee design lost

#The early G series (exploratory)

Equal-budget comparison (same tasks, same number of calls):

Key finding: the OFF5 arm has accepted = True hard-coded — it has no path to refuse. So the comparison above is really "an arm with a free gate" against "an arm that is forbidden to refuse."

#R437, the decisive three-arm run

Three criteria set in advance (frozen 2026-08-24/28):

#R438 mechanism analysis (Held)

Why the reviewer layer is almost a constant function

#Both hypotheses were ruled out

On 2026-09-01 the human proposed two explanations; both were ruled out by our own experiments.

#H-A ceiling: the worker is too strong

We switched to a single 12B model (gemma-4-12b, 179 tasks). An independent audit recomputed it row by row and matched R516 exactly (rows.jsonl 525 lines sha 440d973c, calls 1,986 sha 3d24b73d).

ON vs OFF5: n=167, b=11, c=12, p = 1.0000. The reviewer's grounded lift is +4.19 pp, 95% [0.00, +8.58].

Making the worker weaker did raise the failure rate (25.5% → 31.8%), but the gain still did not appear.

#H-B hard tasks: the task bank is too easy

We switched to date-stamped contest problems (LiveCodeBench v1, 91 medium+hard tasks, gemma-only).

ON vs OFF5: n=87, b=15, c=10, p = 0.4244. The task bank really is harder (16.6 pp higher than MBPP+'s 31.8%), but the gain still did not appear.

Three clean runs, two model families, two task banks, two difficulty levels, one consistent answer: at equal budget, this mechanism cannot beat self-consistency.}}

Two explanations ruled out first, so the mechanism had somewhere to change

#Incidentally: the leak rate is the gate's steadiest benefit

The harder the task, the more refusal is worth. This is the earliest mechanism conclusion on the whole axis to hold.

#The turn: R440P, execution replaces opinion

2026-09-03. We deleted the reviewer committee and switched to running the customer's own acceptance tests. Zero-API offline replay: we recomputed the saved five-candidate data.

#Two self-audits

• R440P's own audit caught its own mistake: the first version of the "equal budget" rule took a majority vote on the hidden tests — a violation of V/GT separation, and not a shippable mechanism. It was changed to look only at the visible tests.
• R440U gate cost (Held): ON is the only arm that can stop a delivery at the gate (coverage 0.829 vs 1.0 for the others). Middle segment, n=77: 14 stopped at the gate, number of correct answers thrown away a = 0. The criterion set in advance was a ≤ 2 → closed.

Every number and verdict on the gate line

#R445: convergence across three task banks

A post-hoc audit of the tasks where delivery was refused: the five drafts were wrong every time — r447 7/7, r449b 8/8, r449c 10/10, r461 15/15. What the gate refuses is not bad luck.

The harder the task, the more early stopping is worth

#EQ5: the same candidates, gate vs vote

conformance-gate

#R460: six arms, four states fixed in advance

2026-09-07 to 09-11. LCB v2, 120 tasks, 6 blocks × 20. gemma-4-12b-it-qat Q4_0, temperature 0.7, six personas, LM Studio 0.4.17, reasoning tokens all 0.

H-MIX − CONFORM: b=22, c=6, +13.33 pp, conditional interval [4.22, 19.46], Holm p = .0112 → labeled EFFECTIVE.

#R460R: five replications, and the headline claim loses to its own rule

This is the most important page of the whole project. For the detailed breakdown see five-replications; here are only the conclusions:

Forbidden wording: "the replication is stable," "the replication failed," "the effect disappeared," "equivalent," "a tie," "the loop is useless." No pooling n, no averaging, no picking one run to report.

#But two claims hold on the same data

The bulk of the gain is "executable acceptance gate + resampling"; the feedback loop is only a small increment stacked on top of it.

#Exploratory: voting loses to the gate, and costs more than twice as much

3/5 have uncorrected p < .05 — and the two that are not significant are exactly r4 and r5, the two runs with zero backend coexistence. This one is in no preregistered family; its status is "Same sign, unresolved."

#R529: four sets across task banks

8 predictions made in advance: 4 hit, 4 missed → INCONCLUSIVE. Heterogeneity χ² = 0.135, df = 3, p = .987 (descriptive only; smallest expected cell count < 5).

False delivery did not reproduce: 6 vs 9, 7 vs 11, 6 vs 6, 58 vs 56 (MBPP+).

#Cost: this is a success-rate gain, not a token-efficiency gain

R460 tokens per task / tokens per correct delivery:

• H-MIX vs OFF: 2.23–2.78×/task, 1.65–2.11×/correct
• H-MIX vs CONFORM: 0.92–1.56×/task (R460R), 1.07–1.69×/task (R529)
• In four of the five replications, H-MIX cost more per correct delivery than CONFORM
• OFF5 vs CONFORM: 2.16–2.85×/task, and a lower delivery rate every time

The token ledger, recomputed call by call

"The same ceiling" is not "the same actual cost."

#R532: switching to a stronger model

2026-09-17/18. qwen/qwen3.8-27b Q4_K_M, non-reasoning, 836 tasks, 43 blocks, 0 voided, 2,508 rows; V/GT fingerprint 199,019 items CLEAN; 86 receipt chains / 3,895 entries all verify.

H-MIX − CONFORM is negative in all five task-bank sets: −5.83 / −5.19 / −16.67 / −1.28 / −0.54. Pooled −3.23 pp, interval [−5.52, −0.75], Holm p = 0.0101 (b=38, c=65).

#Statistical power: why some questions can never be measured

MDE 4.31 pp, but the task bank has only 378 tasks

#What this axis still owes

#Next

• Verdict by verdict: Evidence ledger
• The details of each replication: five-replications
• Records and multi-party execution: Records &amp; multi-party execution