The "equal budget" trap: our ranking flipped completely
The criterion was written as defected <= BUDGET, an upper bound that always holds, so it could not catch "this arm was never bound at all." As a result, whitewash misbehaved 2.5 times as much and we read it as "the strategy is cheaper." After switching to == and verifying, the ranking flipped.
#What we wanted to do
E19 set out to answer a very intuitive question:
Given the same misbehavior budget, which timing pattern is the most economical for an attacker?
So we set a cap: the total number of acts of misbehavior cannot exceed 12. Then we compared the "efficiency" of five attack strategies.
#What we measured
| Arm | Efficiency under the criterion | Bound? |
|---|---|---|
| whitewash (misbehaves all the time, switches identity when caught) | 8.1667 | 29 / 30 |
| sybil | 6.1333 | 30 / 30 |
| pulse(5,20) | 4.30 | 1 / 30 |
| pulse(3,10) | 4.2333 | 1 / 30 |
| patient (misbehaves slowly) | 3.7333 | 1 / 30 |
Our conclusion:
Under an equal budget (12 each) + blind spot 0.5, whitewash at 8.17 actually beats pulsing at 4.23.
"So a crude continuous attack is more economical than a carefully paced pulse." That sentence reads as fairly insightful.
#Where it went wrong
So whom did that budget of 12 actually bind?
| Arm | Share of seeds that reached the budget of 12 |
|---|---|
| whitewash | 29 / 30 |
| sybil | 30 / 30 |
| patient | 1 / 30 |
| pulse(3,10) | 1 / 30 |
| pulse(5,20) | 0 / 30 |
And defected <= BUDGET is an upper bound that always holds — it passes as long as misbehavior does not exceed 12. So by design this criterion cannot tell "bound" from "not bound."
#After the correction
We switched to defected == BUDGET, and verified beforehand that every arm was really bound.
Swept over budget ∈ {1, 2, 3, 4, 6}:
| Budget | whitewash | patient | sybil | pulse(3,10) | pulse(5,20) |
|---|---|---|---|---|---|
| 1 | 1.00 / 0.533 | 1.00 / 0.867 | 1.00 / 0.533 | 1.00 / 0.900 | 1.00 / 0.900 |
| 2 | 1.00 / 0.967 | 0.87 / 1.667 | 1.00 / 0.967 | 0.90 / 1.667 | 0.90 / 1.667 |
| 3 | 1.00 / 1.667 | 0.77 / 2.300 | 1.00 / 1.600 | 0.77 / 2.300 | 0.77 / 2.300 |
| 4 | 1.00 / 2.233 | 0.60 / 2.900 | 1.00 / 2.033 | 0.63 / 2.933 | 0.63 / 2.900 |
| 6 | 1.00 / 3.667 | 0.27 / 3.600 | 1.00 / 3.233 | 0.47 / 3.833 | running |
Only budget = 1 binds all five arms. Relaxing to "80% reach the budget" only adds budgets 1 and 2.
At the only point where all arms are bound (Wilcoxon exact, grouped by seed):
| Arm | Efficiency | vs whitewash | p |
|---|---|---|---|
| pulse(3,10) | 0.900 | +0.367 | 0.0074 (13/15/2) |
| pulse(5,20) | 0.900 | +0.367 | 0.0074 |
| patient | 0.867 | +0.333 | 0.0309 |
| sybil | 0.533 | 0.000 | identical across all 30 seeds |
| whitewash | 0.533 | — | — |
The ranking flipped: whitewash is the weakest, not the strongest.}}
#Consistency check
After the change, we reran budget = 12 and it reproduced the original E19 bit for bit:
| Value | |
|---|---|
| Share reaching the budget | 29 / 30 = 0.9667 |
| Actual misbehavior | 11.9667 |
| Claimed efficiency | 8.1667 |
#Four places this rule applies
| Where | Wrong equal budget | Correct |
|---|---|---|
| Comparing attack strategies | defected <= 12 | == 1 (the only point where all are bound) |
| ON vs OFF5 | accepted hard-coded to True | denominator changed to all tasks |
| Token comparison of equal-budget arms | compares only the call-count cap | sum usage call by call (cap ≠ actual cost) |
| V/GT audit | compares against hidden tests | compares only against visible tests (that one is not a shippable mechanism) |
The full accounting for the third one
#The general rule
#Next
- The full version of this rule: Experimental methods and verdict values
- Six things we measured wrong: measurement-traps
