Who Is Responsible When "AI" Decides to Punish You Unjustly?

Let me start this with saying I have written about this before (just it was about a 40k line PR). I called it accountability debt.

Accountability Debt is: when the efficiency of a system is captured by whoever built or deployed it, and the cost of what that system gets wrong is paid by whoever it was never built to see.

On July 21st Substack rolled out Pangram. An AI tool that scans your own writing and tells your readers whether it was written by a human or an AI. I hope you see the irony in this. And almost immediately people are realizing the flaws in it. We'll get more into that later.

Tangentially, Meta is facing a layoff lawsuit. Why? Because their AI ignored protected leave when sourcing.

Two systems, same architecture. One scores whether you wrote your own words. One scores whether you worked hard enough while giving birth. Neither was built with malice (possibly). But both were built to remove a human from a judgment call.

But then what happens when the machine makes the call, and it is wrong?

Pangram and its 0.01% False Positive Rate

Let's talk about what shipped July 21. Pangram-powered scan on posts, notes, and comments over 100 words. Human vs. AI-assisted vs. AI-generated estimate, shown only when a reader requests it. The framing Substack used was "Claudefishing," not a ban. Chris Best's own words position this as consumer protection, not policing. Sure.

Now let's talk about Pangram claim of a 0.01% false positive rate. Let's say if this is true, a rate that low still produces real false accusations at Substack's volume, and there's no visible mechanism for how a wrongly flagged writer gets unflagged in the reader's eyes. The scan already happened. The impression is already made. Now the reader says "hey you use AI" you're a phony. And the burden of proof goes to the wrongly accused writer.

Now there is an opt-out, but it's opt-out, not opt-in. Burden sits on the writer to know the feature exists, know it's on by default, and go turn it off or write a disclosure statement nobody asked them to write.

And one more thing, the Pangram's tool can't tell if you trained on your own back catalog and it started sounding like you. It measures pattern, not truth. So I could go grab all my essays, research articles, short stories, and random diary entries as a teen, and make an AI that sounds like me 100% and it wouldn't know. Because it's my pattern.

Meta's Maternity Leave Problem

So now let's talk about Meta's lawsuit. Twenty-six Meta employees filed to challenge how they were selected for layoff. They're alleging that the process ranking them pulled from keystroke and activity monitoring data, from dashboards tracking usage of the company's internal AI tools, specifically how many tokens people were spending talking to the models, and from performance rankings that were algorithmically assisted rather than written by a human who'd actually watched them work all year. And their core claim is that how that scoring worked, it structurally could not be earned by someone who was out on protected medical, parental, or family leave during the window it measured.

Now as much as I don't particularly like Meta, I doubt someone sat in a room and decided to target people on leave. The system did that on its own, quietly, as a side effect of being built to remove exactly the kind of judgment that would have caught it.

The kind of judgement Pangram's tool was built to do as well, and then when the tool is wrong (because every tool has a flaw) people are unjustly punished.

So who is responsible?

For some reason it's never the team that shipped the feature. It's the writer defending their authorship, or the employee defending their leave. Their livelihood is affected because the "efficiency" gain happened and someone got to avoid confrontation.

And guess what? The fix isn't complicated, it's the decisions nobody made. Substack could show the disclosure statement by default instead of behind three dots. Meta's system could exclude the leave window from the denominator before the ranking runs. Neither required new technology. Both required someone deciding the exception mattered more than the throughput.

Every scoring system has a person standing exactly where it wasn't built to look. The question isn't whether that person exists. It's whether anyone decided, in advance, to go find them.