VacantAn accountability layer for AIGitHub

Experiments · Results / defense ·

Defense axis

How much work an attacker can get. Of this axis's 12 claims in the project index, 3 were refuted by ourselves and 3 were overstated — but the mechanism that holds off bad actors does work, and what works is not "catching them" but "starving them."

This page cites 4 verdicts: Not re-verified 4click one to jump to its claim card

#What this axis answers

In an agent population that contains participants who deliberately deliver wrong work, how far can signed records, reputation routing, peer checks, sampled audits, and penalties reduce the chance that a wrong delivery is accepted?

Defense axis: verdicts of 16 claims

Held4/16Refuted3/16Overstated3/16Not re-verified6/16

#The most effective one: being seen, not being caught

This is the defense axis's strongest positive result, and it is not "catching them."

GroupWork received over 9,600 rounds
215 seeds that were never caught7 – 24 jobs
Control group that never violates1,290 jobs

In other words: bad actors who were never caught received only 0.57% – 1.84% of an honest participant's workload.

The conclusion in one sentence: what stops attackers is "being seen," not "being caught." Reputation routing alone is enough to starve them, without catching them even once.

(Source: runs/s5_v1_FINDINGS.md)

#Entry cost: neither path should be taken

2026-07-26, 14 mechanism simulations (20 seeds per cell, 40 for E9 and 30 for E15; 400 rounds; 5 honest + 1 attacker; audit rate 0.2, 0.05 from E2/E3 on; reviewer accuracy 0.7, chosen by the author, not measured).

#The real constraint is reviewer accuracy

Reviewer accuracyRouted to attackerHitsROI
1.018.01.90 ± 0.300.119
0.7—2.35—
0.5—2.70—
0.3—4.300.269
0.033.616.65 ± 12.891.041

acc = 0 is the only cell in the whole table where "doing harm starts to pay." None of the other parameters — entry fee, endorser liability, probation length — comes close to the constraint that reviewer accuracy imposes.

#Setting the entry fee too low actually helped the attacker

Exogenous stakeHitsROIFull-block rate
stake = 00.00—100%
stake = 25.002.3540%
stake = 53.050.5600%
stake = 101.950.1770%
stake = 201.900.0920%

Free entry = complete blocking. Adding a stake actually made attacking profitable. This is one of our most surprising results.

#Probation length: all the value is in the first step

Probation mHitsFull-block rate
m = 06.85 ± 1.800%
m = 1, 2, 3, 5, 80.00100%

In one sentence: all of the defensive value is in the m = 0 → 1 step. Making it longer adds no gain at all.

#Verdict

Neither path — an exogenous entry fee or endorser liability — should be taken. The premise is wrong.

Endorser liability is worse: at λ = 1.0 honest participants lose 0.000 but there is no gain at all; at λ = 0.9 honest participants lose 0.691 and the gain is still 0.

#Pulse attack: six conclusions, three refuted

2026-08-03, E17–E24. 30 seeds × 600 rounds, 1,740 per-round files, 1,041,246 lines.

After writing the report, we sent independent agents whose only job was to refute it. Re-verification results for the six headline conclusions:

Original conclusionVerdictWhat re-verification found
Temporal structure barely affects total hits (1.81× spread across 15 cells)OverstatedANOVA F=1.27, η²=0.039, permutation test p=0.115, statistical power only 25%. Broken apart: exposure spans 6.06× and efficiency spans 8.45×, and the two cancel in opposite directions. Attack capability differs by 8.45×.
The blind spot matters far more than parameter choice (pulse 8.9×, patient 54×)OverstatedHeld up almost entirely by the degenerate endpoint blind=1.0. Excluding it gives 3.28× and 5.68×. Interaction η² = 0.264, 4 times the main effect of temporal structure.
Beyond a blind spot of 0.5–0.75, pulsing does worse than sustained misbehaviorRefutedPaired difference −0.57, 95% CI spans 0. Measured pulse(20,2)/(20,1)/(50,1) at blind spot 1.0 got 81/84/87 hits, all higher than patient's 79.
Raising the audit rate cannot rescue cases with a blind spotRefutedAt blind spot 0.5, audit rate 0.1→1.0 cut total hits by 82% (Wilcoxon p=5.5e-6). Audit 0.5→1.0 exactly offsets a 50% blind spot (paired difference 0.03, p=0.78).
Caught once = removed permanently (16/16 seeds)OverstatedConfounded: the 14 uncaught seeds in the same cell stalled just the same. "Can't come back" is wrong — slash 0.9 came back at round 177, 0.8 at round 520. The real mechanism is that slash both halves the mean and doubles n, and n is the exponential coefficient of the return time.
At equal budget, crude strategies beat pulsing (whitewash 8.17 > 4.23)RefutedE19 was never an equal-budget comparison. Budget 12 bound only whitewash (29/30) and sybil (30/30); patient 1/30, pulse(3,10) 1/30. Actual misbehavior counts 11.97 vs 4.87 — whitewash won because it misbehaved 2.5 times as much. Correctly bound at budget=2: patient/pulse 1.67 vs whitewash 0.97, p=0.0015.

#Two positive conclusions that were kept

  1. Detection probability is a single product (1 − blind spot) × sampling rate × accuracy, with the three factors on the same axis. The blind spot changes the defense's ceiling (residual β/(1−β)), not whether the sampling rate is useful.
  2. The blind spot × temporal structure interaction is η² = 0.264, 4 times the main effect of temporal structure. So the sentence pattern "the real variable is X" does not itself hold here.

#Three that are not yet re-verified

Reaction lagWith zero blind spot, one wave still leaks 2.53 items, and reaction takes 27.95 rounds on average (blind spot 0.5 → 3.95 / 34.53; 0.75 → 4.55 / 25.14)
Selective attackFewer total hits but 4.8 times the high-value damage (see above)
The real crossover between burst and sustainedMeasured at about 0.90; for an attacker who picks its parameters, there is no crossover

#Corrections from prior work

Even our headline had been anticipated: Srivatsa et al. (2005, TrustGuard) measured attacker cost ratios across four oscillation models of 1 : 2.28 : 2.08 : 1.36, and proved that the optimal strategy for an attacker who knows the memory window maxH is to oscillate with period = maxH. Our "1.81×" is both older news and weaker, and we have already judged it ourselves to be noise. Hoffman et al. (2009, ACM Computing Surveys §5.4) list it as a named category, oscillation attack.

#New attack surface: the only load-bearing layer

A1–A3, B1–B5 (A series 12 seeds, B series 10 seeds; means only, no CIs, no paired tests).

#Layer-by-layer ablation (B2)

Layer turned offSlander score
All on4.2
Endogenous weighting off3.8
Self-reported same-origin off4.2 (bit-for-bit identical = pure decoration)
Behavior-inferred same-origin off11.4 (2.7×)
Unproven decay off3.6
All off10.7

#Internal collusion (B3) is the cheapest, and hurts the most

AttackerScore
Self-reported same controller3.8
Anonymous5.1
Newly registered10.9 / 7.5

A 1.5 – 2.9× gap. The report's conclusion says it plainly: the only load-bearing layer cannot stop the cheapest attacker.

#B5: harm measured as zero

A collusion group that only slanders and never breaks a rule took 95.4% of the routing share, and its "score" is 0. (baseline collusion share 0.331)

This is not harm being underestimated — this is harm being measured as zero. Any score-based routing will do this.

#B4: the optimal slander intensity is 0.75

Score 9.0, routing share 0.094, honest participants' reputation drops 0.30. At 1.0 it falls back to 4.2 (caught by behavior inference).

#Permanent exclusion: 0 / 368

ParametersReturned to routing
Current (f = 0.5, λ = 1)0 / 368
Move only the mean, not the observation count (λ = 0)214 / 368
f = 0.7, λ = 0308 / 368
f = 0.9, λ = 1308 / 368

Why "permanent" is the product of two reasonable patches

The closed form (total ≥ exp(n'·((target−mean')/c)²) − 1, m=0.9, target=0.95, c=0.3) at n=10: f=0.5/λ=1 needs 10¹² observations, f=0.9/λ=0 needs 10^0.95 ≈ 9 observations.

That 10¹² is not "a long wait"; it is unreachable.

#Channel separation and multi-directional environments

#Channel separation (passed both criteria)

  • Expertise profile: expert hit rate 0.1653 → 0.6200 (random 1/6 = 0.1667; 3.72 times random), quality +24.7 pp. The null control group went 0.4330 → 0.4330, identical to four decimal places, with the same standard deviation.
  • Sealed review: herd_overrides went from 7,889 cumulative flips to constantly 0, and the spread across herds is exactly 0.0. False positives 0 / 340.
  • The scaled-up version refuted its own overly strong claim: masking disappears monotonically as the group grows (2/10: 0/6→6/6; 4/10: 10/12→12/12; 6/10: 18/18→18/18). It applies only to "small collusion groups."
  • Regression: 428 passed / 1 skipped; the three pinned digests reproduced 15/15 bit-for-bit identical; all 180 review envelopes from E10 still verify.

#Multi-directional environments (M1–M5, 300 rounds × 24 seeds)

M1 asks: how much of the shared blind spot does the architecture create by itself?

Blind spotOne-directionalTheoretical floorCaused by architectureShare
0.00.1916 ± 0.00690.0270+0.164686%
0.0 + sealed review0.12830.0270+0.101379%
0.0 + hidden reputation0.08180.0270+0.054867%
0.0 + multi-directional0.0236 ± 0.00270.0270−0.0034≈0
0.3one-directional 0.5003 / multi-directional 0.30900.318936% / ≈0
0.6one-directional 0.7602 / multi-directional 0.61000.610820% / ≈0

How to read it: with a one-directional architecture at zero blind spot, 86% of the shared errors are manufactured by the architecture itself. After switching to multi-directional, it falls below the theoretical floor. M1b swept cascade_p / authority_w and got an excess ranging from −0.010 to +0.316 — the magnitude is unreliable; the direction is robust.

M2's add-one ladder (blind spot 0.3): one-directional 0.6935 → multi-directional 0.8252, +13.2 pp, and the bad items that slipped through were halved. Breakdown: hidden reputation +8.0 pp, refusal primitive +6.1 pp, asking for help +0.5 pp, sealed review alone slightly worse.

M3 splits the intervention into two factors: "where to look" contributes 70%, "when to look" contributes 30%.

#Defense experiments with real models

#E10 trust switch (paired, N = 60)

qwen3.6-35b-a3b; 60 paired tasks from EvalPlus MBPP+; 2 saboteurs planted by hand in the roster.

ArmPassedValidPass rateTimeCalls per pass
off31600.51676,837.5 s1.94
on36600.60006,168.0 s1.67

Paired: n=60, b_on_only = 11, c_off_only = 6, 17 discordant, Δ = +0.0833, 95% CI [−0.05, 0.2167], McNemar p = 0.3323. Statistical power is only 0.163; an MDE at 80% would need n = 337.

The second outcome variable is more interesting: work assigned to saboteurs went from off 20/60 (33.3%) → on 11/60 (18.3%), McNemar p = 0.0931 (did not reach .05). Removing 10–30 warm-up tasks puts p at 0.021 – 0.041; removing 35 or 40 gives 0.0703 / 0.2188.}}

#E11 three memory arms (pilot)

Same model, 12 tasks: M0 8/12, M1 8/12, M2 8/12 (time 846.2 / 1,012.9 / 2,144.6 seconds). The original text says: "Not enough to support any effect claim." (original: 「不足以支撐任何效果宣稱。」)

#Independent review: six attack PoCs

2026-07-26, target commit f9a381a. The verdict in one sentence: this architecture currently raises accountability, not trust.

See Attribution. Only the conclusion is listed here: of the three necessary conditions, two fail outright (the signal can be cheaply forged; the penalty bites, but for low scorers it is a reward), and one partly holds.

#What this axis still owes

#Next

Last updated: · about 8 min read · this page cites 4 adjudicated claims

Shih Hsin UniversityDepartment of Information and Communications, Shih Hsin UniversityVacant

Department of Information and Communications, Shih Hsin University · 2026 capstone project · on-site exhibition