Under Watch
A small, fully inspectable test of Goodhart's law: learned monitors catch a hidden saboteur in a cooperative team until the saboteur is trained against them.
Results
- 01Unpressured sabotage is easy to catch: held-out AUROC 0.999 for the MLP monitor and 0.998 for the RGCN.
- 02At λ = 20 the frozen MLP monitor falls to a mean AUROC of 0.58 across 5 seeds, while the saboteur's damage falls 58% to 77%.
- 03A monitor retrained on the evasive saboteur scores 0.97 to 0.99 across 5 seeds: evasion exploited one frozen monitor's blind spots.
Engineering details
A monitor that works against an unpressured adversary may stop working once the adversary is optimised against it. This study asks that question in a setting small enough to inspect completely, on a laptop CPU in under an hour.
Rebuild the world
A batched JAX port of PettingZoo MPE simple_spread_v3 matches the reference physics, rewards and observations step by step. One seat per episode is secretly a saboteur rewarded by the team's failure.
Train two monitors
An MLP reads one agent's trajectory; an RGCN reads typed agent and landmark relations. Both are trained on 4,096 worlds and every score is reported on 2,048 held-out worlds.
Apply pressure
The saboteur pays for being flagged by a frozen monitor, at penalty weights from 0 to 40, as a black box with no gradients through the monitor. A three-round arms race then retrains the monitor on every saboteur seen so far.