debug: your codebase is haunted and I can prove it

It's Monday, your first day. The laptop is brand new, still has the plastic on the trackpad. You peel it off, and it's the most satisfying thing that's going to happen to you all week.

You clone the repo. It takes eleven minutes. That's your first clue, but you don't know it yet. You open the README. It says, "Getting started: run make setup." You run make setup. It fails, on a dependency that was deprecated before you finished college.

You ask the team channel. Somebody replies with a thumbs up emoji and a link to a Confluence page. The Confluence page has a banner across the top: "This page is out of date. See the README."

There's a movie from 2001, Thirteen Ghosts, a remake of William Castle's movie from 1960. A guy named Arthur, a widower with two kids and not a lot of money, finds out his uncle Cyrus has died and left him a house. And it's an incredible house. Glass walls, steel, strange writing etched into every pane. Arthur walks in thinking, okay, this is weird, but it's free and it's mine.

What he doesn't know yet is that it isn't a house. It's a machine. It's powered by ghosts his uncle trapped inside it. The writing on the glass is what's holding them in, and you can't see a single one of them unless you're wearing a special pair of glasses.

Arthur doesn't have the glasses. So for the first part of the movie, he just walks around his new house admiring the architecture, while it's absolutely full of things that want to hurt him.

That's your first week at a new job.

Welcome back to Chaotic Commits. I'm Joanne Skiles, and it's October. This month on the show, we're spending time with things living inside our systems. The stuff we didn't build, can't fully see, and are a little afraid to touch.

I've started at a lot of companies. Defense, Disney, golf, healthcare, big bank, AI. And I want to tell you that some of those were beautiful, well-documented codebases. Clean architecture, clear ownership, tests that actually ran.

I want to tell you that. I cannot tell you that. Every one of them was a haunted house.

Every inherited codebase is a haunted house. You didn't build it. The people who did are mostly gone. Something is making that noise and you don't know what. And every room has a decision in it that made sense to somebody once.

The haunting isn't supernatural. It's just undocumented.

We're gonna do this the way you actually experience it. One week, Monday to Friday, your first week in the house. And by Friday, I want you to be able to tell whether you're living in one. Just a spoiler: you are. So let's get into it.

Monday. We've already met the welcome mat. The README is the welcome mat. "Home Sweet Home" on a door that's been warped for six years. It's still technically there. It's still technically welcoming you. It just no longer describes the house.

And I want to be fair to the README, because nobody sets out to write a bad one. Somebody wrote it on a good day, and it was accurate on that day. And then the house kept changing and the welcome mat didn't, because no sprint ever had a ticket called "update the welcome mat."

You can date the README like a tree. You count the rings. The setup instructions are from 2019. The section on environment variables is from 2021. There's one line at the bottom that says "TODO: document the new auth flow." And the new auth flow is three auth flows ago.

Monday afternoon, someone gives you access to the dashboards. And this is the portrait in the hallway, the one whose eyes follow you. There are 47 dashboards. They watch everything. Request count, CPU, memory, something called "queue depth (old)". They follow you from room to room. They see everything that happens in the house. They have never once told anyone anything useful.

You ask, "Which one do you look at when something breaks?" And the senior engineer thinks about it for a second and then says, "Honestly? I just check the logs."

Tuesday. You're shadowing on-call, which is a nice way to say you're sitting next to the person who gets woken up so you can learn what it feels like to be woken up.

At 3:12 in the morning, the lights flicker and an alert fires. Elevated error rate, checkout service. Your on-call buddy rolls over, opens their laptop, and by the time the laptop wakes up, the alert has resolved itself. Gone. Green. Nothing to see.

You ask what that was. They say, "Oh, that one. It does that." And you say, "How long has it done that?" And they respond, "Since before I got here."

Flickering lights are the most normal thing in a haunted house, which is exactly why they're dangerous. The first time, everyone gets up and checks. The tenth time, someone mutes the channel. By the hundredth time, the flicker is just part of the house. It's ambiance.

Flaky tests are the same thing. The test that fails one run in twenty: you hit rerun, it passes, you merge. Everyone on the team has muscle memory for that rerun button, like flipping the light switch twice when you walk into the bathroom.

And the thing about the flicker? It isn't random. Lights don't flicker for no reason. There's a loose wire somewhere, a connection that's almost good enough. The alert that resolves itself is telling you something real about the system. It's just telling you at 3:12 in the morning, in a whisper, and then taking it back. You learn to stop listening because it keeps taking it back.

Wednesday. Your first real ticket. It's small. That's the point. They always give the new person something small. Add a field to the order confirmation email. Easy, half a day.

You trace the code. The email gets its data from a function called calculate_order_totals, and you open it. It's 400 lines long and has a comment at the top that says "Do not modify without talking to Ray."

There's no Ray. You check. There hasn't been a Ray for some time.

The function takes eleven arguments. Three of them are booleans. One of them is called legacy_mode, and it's set to true everywhere. Every single call site. So it really isn't a mode. It's just how the function works now, with a name that suggests it's temporary.

And you think, I could fix this. I could clean this up while I'm here. It'd take a day. So you start. And you've changed about four lines when the senior engineer walks up, looks at your screen, and makes a sound. Not a word, just a sound.

That's the creaky floorboard. Every haunted house has one. Everyone who lives there knows exactly where it is. They step over it without thinking. They don't talk about it, because talking about it is how new people find out where it is and step on it. There's an unspoken agreement: you don't touch it, it doesn't break anything new.

I've watched this exact agreement in more codebases than I can count, and the thing nobody says out loud is that the agreement is completely rational. That function works. Badly, mysteriously, but it works. Nobody's been paged for it in two years, and the last person who tried to fix it caused an incident that people still describe with a specific facial expression.

So the team made a deal with the floorboard. We won't fix you. You won't hurt us.

The problem is that the deal has a cost, and the cost just keeps adding up. Every new feature that needs order totals has to route around the floorboard. People write a second function that does almost the same thing, so they don't have to touch the first one. And now there's two floorboards.

And the new person, in this case you, learns the most important lesson in any haunted house, and that is where not to step. That's not onboarding. That's just a ghost story getting passed down.

So you put your four lines back. You add your field to the email somewhere else. It takes two days instead of half a day, and you have no idea how you'd explain why.

And now it's Thursday. You're starting to get comfortable. That's always what happens in the movies. You kind of settle down and you're like, oh yeah, nothing bad's gonna happen.

So you're reading through a service list, trying to build a map in your head, and you find one that you haven't heard anyone mention. It's called Reconciler. It's in the architecture diagram. It has arrows going into it, arrows coming out of it. So you go look at it.

You don't have access. So you ask for access, but nobody can grant it. The team that owned it got re-orged into a different part of the company two years ago. The people who had admin rights are now in the other org, and when you message them, they say, "Oh wow, that's still running?"

That's the locked room in the east wing. You know, three people tried to get into it. Two of them left the company. One of them is still trying, and if you ask about it, they get a faraway look.

And here's what makes the locked room scary. It's not that something bad is in there. It's that something important could be in there. The arrows go into it. The arrows come out of it. Whatever it does, the rest of the house is depending on what it's doing, and you can't see inside.

And then Thursday afternoon rolls around and you notice the cold spot. You're looking at latency on one of the core services. The median is fine. The p95 is fine. And the p99 is, um, not fine. Every hundred requests, one of them takes eight seconds. Not sometimes. Always. And it's been that way since before anyone on the team got there.

Someone filed a ticket about it. The ticket has fourteen comments, and the last comment was from 2023, and it says, "Still seeing this."

That's the cold spot. You walk through one particular doorway and the temperature drops ten degrees and you shiver. And everyone in the house knows about it. They wear a sweater in that hallway, because nobody has found out why.

But I want you to notice something about Thursday, because it's the most important day of the week. The locked room and the cold spot, those are not just technical problems. They're actually, more importantly, organizational ones. The room is locked because ownership moved and the keys didn't move with it. The cold spot is still cold because finding out why would take a week, and there's just not any time on the roadmap for that mystery. So this becomes architecture.

There are two more rooms I'm not touring today. Somewhere in this house, there's an engineer who left years ago whose scripts still run every night. And somewhere underneath it, there's a basement: a table that something writes to every morning, and nobody knows what. You find those on your own. Everybody does.

Friday. You've survived the week. You've got a small change ready. Not the floorboard, but something safe. You test it in staging, it works perfectly, and this is the mirror. The one in the upstairs hallway that shows you a slightly different room than the one you're standing in.

Staging is a portrait of what production looked like in 2022. Same services, mostly. Some of the same data, sort of. A different version of the database. A feature flag that's on there, but it's off in prod. Staging is the mirror that tells you everything is fine, because in the room it's showing you, everything is.

But the door that opens by itself, it's right next to the mirror. So your change works every time in staging, and in production it fails 30% of the time. You can't reproduce it. It's a race condition, or a timing thing, or a load balancer is having a feeling. The door just opens. Nobody touched it.

So it's Friday afternoon and you're about to deploy, and somebody on the team quietly says, "I wouldn't deploy on a Friday."

That's the neighbor who warned you. Every horror movie has one. The old guy at the gas station, the neighbor leaning on the fence. She says it once, specifically, with reasons, and everyone ignores her because the movie needs to happen. Friday deploys continue across our industry. The neighbor is still at the fence.

And then the last room of the week. The one that's new to the house. It wasn't here five years ago.

You still don't understand calculate_order_totals. There's no documentation. Ray is gone. So late Friday afternoon, alone, you paste the whole thing into an AI assistant and ask it what the function does.

Did you know that's our Ouija board? Because it answers you instantly, confidently, in full sentences, probably some bullet points in there. And it tells you what the function does, why legacy mode exists, what the three booleans control. And you have absolutely no way to know if any of that is true.

That's what makes it a Ouija board, not a flashlight. A flashlight, you turn it on and you see what's actually in the room. A Ouija board gives you an answer that sounds like it came from the room. It might have. Or it might have come from your own hand moving the little plastic piece, because you really, really, really wanted an answer before the weekend.

I use these tools. I advise companies on how to build them. They're useful for reading code. But "the model explained it" and "the code does that" are two different claims. And in a house, or a codebase, with no documentation, there's nothing to check one against the other.

So here's your Friday checklist. Signs you're living in a haunted house:

The README describes a house that no longer exists. The dashboards watch everything and tell you nothing. An alert resolves itself, and the explanation is "it does that." There's a function people step around. There's a service nobody can get into. There's a latency number that's been wrong since before anyone got there. Staging and production disagree, and staging is the one telling you what you want to hear. Somebody warns you about something and nobody listens. And the most complete explanation of your system was written by a chatbot this afternoon, and nobody's checked it.

If you got three or more, congratulations. You live here now.

So let's go back to Arthur. The thing that makes Thirteen Ghosts work as a horror movie is that the house isn't random. It isn't some cursed building that happened to be full of ghosts. Somebody built it on purpose. His uncle designed every wall, every pane, every bit of writing on the glass. Every room was a decision. It's a machine. It was always running. Arthur just didn't know what it was for.

Your codebase is the same. It's not cursed. It's not random. Every floorboard and locked room and cold spot is a decision somebody made, for a reason that made sense to them, on a day you weren't there. The haunting isn't that the decisions were bad. It's that the reasons didn't get written down and the people who knew them moved on.

And here's my fun nerdy part. When the original 13 Ghosts came out in 1960, William Castle, who was a showman before anything else, handed every person in the theater a little cardboard viewer with red and blue filters. Look through one side and you could see the ghosts. Look through the other and they disappeared. He called it Illusion-O. The audience got to choose whether they saw the ghosts.

That's what this whole job is. Observability, tracing, ownership, a README with a date on it. It's the viewer. The ghosts were always in the house. The only thing that changes is whether you're looking through the side of the glasses that lets you see them.

Most teams are holding the viewer the wrong way around. Not because they're bad at their jobs, but because the other side is scarier, and the house has been running fine. Mostly. Mostly fine. Except for that flicker at 3:12.

So this week, pick one room. Just one. The alert that does that. The floorboard. The cold spot. Put on the glasses and actually look at it. You don't have to fix it yet. Just find out what it is.

The haunting isn't permanent. But you can't deal with a ghost you decided that you just didn't want to see.

That's this episode of Chaotic Commits. I'm Joanne Skiles, and if your team has a new hire starting this month, send them this episode before they find the floorboard the way the rest of us did.

New episodes wherever you get podcasts. No highlight reel, just the commit history.