blame: she left in 2021 and her bash scripts are still running

It's 2:07 in the morning. The pages start. Checkout is failing. Not all of it. Maybe one in five requests. The internal ledger service is returning errors. Too many requests. So, rate limiting.

The on-call engineer does what everybody does at 2:07 in the morning. They look outside. Is it a traffic spike? Is it a bot? Is it a payment processor? They check the vendor status pages. All green. They get a security team on the call because somebody's hammering the ledger API hard, and it looks a lot like an attack.

For 40 minutes, five people on a call look out the windows. And then someone finally pulls up the source address of all that traffic. It starts with a 10 dot. It's an internal IP. It's one of ours. It's a box called util-02 that nobody on the call has ever logged into. And at 2:00 AM, like it does every night, it ran a bash script.

The call is coming from inside the house.

Welcome back to Chaotic Commits. I'm Joanne Skiles. I'm an engineer. I teach computer science. I advise companies on how their AI systems are actually designed, and my research is on transparent systems, which mostly means I'm the person on the incident call asking, "Okay, but who owns that box?" And then waiting through a very long and uncomfortable silence.

It's October, and this month we're spending time with things living inside our systems, the stuff we didn't build, can't fully see, and are a little afraid to touch. This one is about a ghost. A specific kind of ghost. The engineer who left and whose work is still running.

A quick note before we go in. The story I'm about to tell you is a composite of many versions that I have lived through at more than one company. I've changed the names, the systems, and enough details that nobody's former coworker is getting a phone call because of this episode. But every part of it is real somewhere. And if you've been in this industry longer than about three years, at least one part of it is real for you.

Here's what we are going to talk about: the bus factor. The bus factor is a number. The haunting is what happens when it hits zero. When one engineer is the only person who understands a system, their leaving doesn't end the system. It ends the understanding of the system, because the system keeps running. And I wanna be really clear upfront about whose fault that is, because it's not the engineer who left.

So let's get into it.

It's 2:50 in the morning, and the team knows where the call's coming from. They now need to know what it is. Somebody finds the SSH key for util-02 in a shared password vault in a folder called misc. They log in. They run crontab -l, which lists the scheduled jobs on the machine. There's a line that runs at 2:00 AM every night. It calls a script called partner_sync.sh.

They find the script in a repository called ops-scripts, and somebody does the thing you do when you're looking at code you don't understand. You run git blame. If you've never used it, git blame goes line by line through a file and tells you who last changed each line and when. It's a tool for tracing authorship. It has the most dramatic name of any command in Git, and at 3:00 in the morning, it kinda earns it.

Every line comes back to the same name. Sarah.

Nobody on the call has worked with Sarah. Someone checks the org chart. She's not on it. Someone checks Slack. Her account is deactivated. Her last commit to that file is from the spring of 2021. She left in 2021. Her script has run every night since, something like 1,500 nights in a row without her.

Here's what the script does, once they've read it and understood it. Every night, it pulls a settlement file from a payments partner. It parses it line by line, and for every line, it calls the ledger service to mark a transaction as settled.

Here's what changed. Last week, the partner added a column to their file, which is normal. They sent an email about it to an address that forwarded to Sarah. So tonight, every single line fails to parse, and the script has a retry loop. It's literally a loop that says, "Until this call succeeds, sleep for one second and try again," with no limit. It's sitting on util-02 at 2:00 in the morning, calling the ledger service over and over, forever, for a transaction that's never going to be able to settle. The ledger service starts rate limiting, and because checkout talks to the same ledger service, checkout starts failing, too.

Nobody attacked anybody. A script got a file it didn't expect, and it did exactly what it was written to do.

So let's go back to the term bus factor. The bus factor is a morbid name, but it's a very useful number. It's how many people on your team would have to get hit by a bus, or maybe we'll be a little more positive here and say win the lottery and move to a beach house, before a system becomes unmaintainable. If one person leaves and nobody can run the thing, your bus factor is one.

For partner_sync.sh, the bus factor had been one for years, and in the spring of 2021, it hit zero. Nothing happened that day. No alarm went off, and the script ran that night, and the next night, and the night after that.

And that's the thing that people miss about the bus factor. It doesn't fail when it hits zero. It fails later on, on a random night when the world changes a little bit and the one person who would have noticed isn't there anymore. So most teams never calculate it, and what ends up happening is they find it out by accident, and that accident is usually an incident.

I wanna talk to you about Sarah now, because the most important part of the story is: how did she end up as the only one? You can actually reconstruct it, because like with most ghost stories, the ghost leaves a trail. You just have to read it in the right order.

If we look at partner_sync.sh, the first commit is from 2018. The commit message says, "Temporary script for a partner launch. Will move into pipeline V2." Temporary. If you've been doing this for a while, you are laughing with me. Nothing is more permanent than a temporary script that works.

And it did work. That's the thing. The partner launched on time. The script ran, and everybody was happy. And Sarah became the person who knew how partner settlement worked.

Then she became the person who got asked about it. A finance analyst had a question about a missing settlement, so they messaged Sarah. A new partner needed to be onboarded, so they looped in Sarah. The script needed just a little tweak. Sarah did it, because it's faster than explaining it to someone else.

Then she became the person who got paged about it. It's not in anyone's on-call rotation, because it's not really a service. It's just a script on a box. So when it breaks, people DM her.

None of that is a decision. Nobody sits down and says, "Okay, let's make a single point of failure for partner payments." It happens one little reasonable favor at a time. Small little dependencies, then it becomes tacit knowledge, then informal consulting, and then it becomes an entire person holding it up. By the time anyone could have noticed, it was just part of the org.

And here's the part of the trail that you really need to hear. In 2019, Sarah opened a ticket titled "Move partner sync off util-02 and into the data pipeline." The description is three paragraphs long. It's good. It explains what the script does. It explains why running it from a cron on a single box is risky. It even says, and I'm paraphrasing because it's been a few years, "If the partner changes their file format, this will fail in a way that's hard to see."

The ticket has a priority field: medium. It was moved to the next quarter, and then the next one, then a third time, with a comment from a product manager saying, "Not customer facing. Deprioritizing for now."

The last comment on that ticket is from Sarah in early 2021, and it says, "Happy to pair with someone on this before I go."

And no one replied.

So when people say she didn't document anything, we gotta push back here. She did document it. She documented the risk, the fix, and the exact failure mode that took down checkout at 2:07 in the morning. She put it in the place the organization told her to put it, and the organization filed it under later, because medium priorities are meaningless here. We only work on stuff that's high.

She tried to give the knowledge away. The organization just didn't have anywhere to put it, because they were too busy.

So let's go back to the call now. It's 3:15. They know what the script is. They know why it's failing. They just have to stop it. This is where the haunting really starts showing its teeth, because stopping it turns out to be terrifying.

Can you just kill the process? Maybe. But the script has been marking transactions as settled one at a time all night. If you kill it in the middle, is half of tonight's file settled and half not? Does that double pay somebody tomorrow? Does finance's reconciliation break? Nobody knows. The person who knows is long gone.

Somebody then finds a runbook. The good thing is there was one. It's a wiki page from 2020. Under "If partner sync fails," it says one thing: "Ask Sarah."

That's it. That's the whole runbook. And I don't think whoever wrote that was being lazy, 'cause in 2021, it was the correct runbook. It was the fastest, most reliable path to a fix. It just had a dependency that, well, left, and nobody updated that part.

At 3:15 in the morning, five people make a judgment call about payments with no information. They kill the retry loop, not the cron job. They let the ledger recover. Checkout comes back. They decide to deal with the file in the morning, when finance is awake.

The actual fix the next day is one line: skip the new column. One line that took about 90 minutes of an incident and a morning of finance people reconciling by hand to figure out. That's what the haunting costs. Not the line. Everything around the line.

And then the next morning, somebody does the thing nobody wants to do. They go back and look at that crontab again, properly this time. There are six other lines in it. Six other jobs, all Sarah's, running every night or every hour or the first of every month or the second week. Nobody on the current team knew any of them existed until the one that broke dragged everyone into the house to look.

This is the specific cruelty of on-call in a haunted system. You get paged for something you've never seen, written by somebody you've never met, at an hour when your judgment is at its worst, because we all make great judgment calls at 2:00 AM, right? And the only document that exists tells you to call the person who isn't there.

And you know what the post-incident review says, the one I've seen in some form at a lot of companies? It says, "Root cause: undocumented legacy script. Action item: find someone who knows this system."

There's no one who knows this system. That was the root cause.

Okay. What do you actually do about it? And I want to resist something here, because the easy version we can get to is a documentation PSA. Write things down, make a wiki. And look, documentation helps. It does. But Sarah wrote things down. She wrote the most accurate risk assessment anyone at that company ever wrote about that system. Documentation was not the missing piece.

The question isn't, did she document enough? The question is, did the organization create the conditions for knowledge to spread? And that's a different kind of fix.

The first thing is to go find your calls from inside the house before the incident does it for you. Every scheduled job, every cron entry on every box, every Lambda on a timer, every scheduled task in a CI system, every Kubernetes CronJob. Make a list. For every line, write down a name. A current name, a person who's still on the org chart. The lines where you can't write a name, that's the list of ghosts. That's your real bus factor, and it's almost always worse than you'd guess.

The second thing is that knowledge spreads through people doing the work, not through people reading about it. If one person is always the one who handles partner settlements, then one person knows partner settlements. So you gotta pair on it. Rotate. Put it in an actual on-call rotation, even if it's just a script. Actually, especially if it's just a script. The scripts are where the ghosts live.

The third thing is offboarding. Most companies treat it like a form. Return the laptop, revoke the badge, deactivate Slack. And they deactivate the Slack account of the person who's the only receiver for a partner's change notifications. Offboarding a senior engineer should be a project, with a list of every system they're the only person for, and someone named on the other end of each one before the last day. Not after an incident.

The fourth thing is gonna be uncomfortable, and it's for the leaders. Sarah's ticket got deprioritized because it's not customer facing. That was true right up until 2:07 in the morning, when it took down checkout. Knowledge concentrated in one person is a risk, an actual business risk, the same as a single database with no replica. It belongs in the same place you track your other risks, with someone accountable for it. If you're a director and you've got a "happy to pair with someone on this before I go" comment sitting on a ticket somewhere with no reply, that's not her backlog. That's yours.

And if you're the Sarah right now, if you're the only one who knows how the thing works, here's what I say to you. It's not your job to fix this alone. You can't fix it alone, because the fix requires someone else to have the time to learn it. But you can make it really hard to ignore. Say it in writing to your manager, with the words "single point of failure" in it. Ask for a pairing partner by name. And then if the organization still files it under later, you've done your part. Whatever happens after you leave, that belongs to them.

You know why the babysitter story still works all these years later? It's not the breathing on the phone. It's that the whole time, she was checking the windows, locking the doors, watching the dark yard. She was doing everything right for the wrong threat. The danger wasn't trying to get in. It was already in the house. It had been there the whole time.

That's what a ghost like Sarah's script is. It's not an intruder. Nobody snuck in. It was invited and welcomed and thanked back in 2018, when it got the partner launched on time. It just stayed after everyone forgot it was there.

She left in 2021. Her bash scripts are still running. They did something important every night for years, and almost nobody knew what. That's not her failure. That's the system.

So go check your crontab. Not the one in the architecture diagram. The real one, on the box nobody's logged into. And if you find a name on it that isn't on the org chart anymore, don't wait for 2:07 in the morning to find out what it does.

That's this episode of Chaotic Commits. I'm Joanne Skiles, and I've definitely been somebody's Sarah, which means somewhere out there is a cron job of mine I'd really rather not hear about.

New episodes wherever you get podcasts. No highlight reel, just the commit history.