Who Is Responsible When "AI" Decides to Punish You Unjustly?
Two scoring systems shipped in the same week: Substack's Pangram AI-writing detector and Meta's layoff-ranking that ignored protected leave. Both removed a human from a judgment call, and when the machine is wrong, the burden of proof lands on the person it misjudged.
This picks up a term I coined earlier: accountability debt, the gap between whoever captures a system's efficiency win and whoever pays when it gets something wrong. Two July launches make the point. Substack's Pangram scans your writing and tells readers whether it thinks you're human. Meta's layoff process pulled from keystroke data, AI-tool token usage, and algorithmically-assisted rankings, and 26 employees say it structurally couldn't be satisfied by anyone who was out on protected leave.
Key takeaways
- Low false-positive rate still means real false accusations at scale, with no way to un-flag someone after
- Burden of proof shifts to the person the machine misjudged
- Opt-out defaults dump the work on whoever is least equipped to push back
- Pattern-matching measures pattern, not truth: train on your own back catalog and it flags you as AI
- No malice required. The system targets the person on leave as a side effect of removing human judgment
- The fixes weren't technical: show the disclosure by default, exclude the leave window before ranking. Both just needed someone to decide the exception mattered
Who this is for
Anyone building or deploying a scoring, ranking, or detection system, and anyone on the receiving end of one. Short opinion essay on accountability and automation.
The full piece is on Substack.