The gate cuts false delivery by about 40%, not in half
An experiment with 836 tasks, 43 blocks and 0 voided. The thesis's "halved" was wrong — because the two rates used different denominators. This correction changes the gate's effect from "in half" to "roughly 40%."
#What false delivery is
| Term | Definition |
|---|---|
| Delivery success | The system actually lets it through, and the saved candidate passes the hidden evaluation |
| False delivery | The system lets it through, but the candidate does not pass the hidden evaluation |
So there are three possible ways to count a "false delivery rate." Each is correct, but they give different numbers.
#The R532 data
| Model | qwen/qwen3.8-27b Q4_K_M, non-reasoning |
| Tasks | 836 |
| Blocks | 43 |
| Completion | 43 / 43 |
| Data rows | 2,508 |
| Voided | 0 |
| V/GT fingerprint | 199,019 items CLEAN |
| Receipt chains | 86 chains / 3,895 entries, all verify |
This is the experiment with the cleanest data quality in the project — 0 voided, 0 V/GT violations, all 86 chains verify.
#Three ways of counting
| Arm | Accepted | False deliveries | False deliveries / all 836 | False deliveries / accepted |
|---|---|---|---|---|
| CONFORM | 811 | 120 | 14.4% | 14.8% |
| OFF | 625 | 211 | 25.2% | 33.8% |
#The corrected claim
#Why this number matters
#How it relates to other numbers
| Figure | Value | Source |
|---|---|---|
| False delivery rate (CONFORM) | 14.4% (14.8% of accepted) | R532 |
| False delivery rate (OFF) | 25.2% | R532 |
| Coverage (share CONFORM can let through) | 811 / 836 = 97.0% | R532 |
| Candidate-pool ceiling (r441) | 82.68% | R440P |
| Residual realizable attack (generic_gauged) | +2.72 pp | R452 |
| Gauge upper bound (weak_oracle) | +4.35 pp | R452 |
#This experiment's two amendments
AMEND1 (about the H-MIX label):
| Task-bank set | H-MIX − CONFORM |
|---|---|
| 1 | −5.83 |
| 2 | −5.19 |
| 3 | −16.67 |
| 4 | −1.28 |
| 5 | −0.54 |
| Pooled | −3.23 pp, interval [−5.52, −0.75], Holm p = 0.0101 |
The analyzer checked only significance, not direction, so it labeled this EFFECTIVE. That label is voided.
Only RULED_OUT may be cited (this round's threshold: interval upper bound < +2.0 pp), and saying "the loop is harmful" is explicitly forbidden.
AMEND2 (about the premise):
The premise "a stronger model" does not hold. The 27B OFF arm got 74.8% vs the 12B's 75.2%.
We did not actually compare against a stronger model. So R532 cannot be used to say "the gain disappears when the model is stronger" — because the model was not stronger.
#Generalizing the denominator lesson
#Next
- The gate's full rules and ceiling: conformance-gate
- R532's full verdict: five-replications
- The structural problem with visible/hidden tests: v-gt-is-not-lossless
