The Calibration Room

When I worked at a specific company (that may have considered themselves a "tech" company but actually was a bank) we had calibrations twice a year. Once in the summer, at the year's six month mark, and then again in late October, early November (for the second half of the year, even though the year wasn't over yet).

What the heck is calibration?

Calibration is where every manager+ (that means Manager, Senior Manager, Director, Senior Director, etc.) in the organization gets together and stack ranks everyone per level in the organization. These are huge organizations, mind you. We are talking twenty plus people per level from Junior to Manager. Then the managers who ranked them get ranked by their own managers, in a different meeting, and so on up the chain until it hits VP. I have no idea how VPs got ranked. That happened somewhere I never saw.

These meetings ran a minimum of four hours per level (but lasted far longer). This was because we literally stacked people in a spreadsheet and argued about their "impact" with other managers we only knew from these meetings. We didn't know their direct reports. What we had was five bullet points on a slide, and whatever the manager had written for what they'd coach the person on going forward, a required field that most people hand waved past.

It was by far the least efficient use of my time in the history of meetings I've sat through in my life. Because it was full of extreme biases and managers that would hide/lie about their engineers because they either didn't know how to coach someone or were afraid to PIP someone.

I did this twice a year, for multiple years. And if you told me you could automate the whole thing, I would have said yes immediately.

The "Process"

We had a forced distribution, and we knew the distribution going in. It started with a kickoff meeting. Someone reminded the room of the process, assigned the roles, a note taker, a timer, someone whose job was to call out bias when they heard it, though honestly that should have been everyone's job. Then they read out the distribution.

For example they would say, "This calibration cycle the big wigs," they didn't say big wigs but that's what I heard they said something like 'Leadership', "decided 10% Exceptional, 20% Above Strong, 60% Strong, 10% Below Strong. Remember, most of the group should land in Strong. We don't need to fill Exceptional, that can come up short. But 10% has to be Below Strong."

Just calling this out here:

10% had to be Below Strong.

This meant 10% of the organization would be PIP'd once calibration wrapped. And if the person who landed in that bucket was on your team, you were the one delivering it.

And here's the part that made the whole thing feel absurd. This was a forced distribution for every org, no matter the performance. It didn't matter if your team had an incredible cycle and nobody on it was performing anywhere close to Below Strong, 10% still had to go. That meant an org with twenty Senior Engineers PIP'd two Senior Engineers twice a year. And it didn't matter if there's a Joe Schmo in "Organization Does Nothing At All" who was ranked as Strong and the work he did was way less than what those two outputted. Joe is safe. He happened to be in a org where the curve didn't need him. Your two weren't as lucky, and it had nothing to do with what they'd actually built that year.

Now, let me explain what a PIP actually is, for anyone who's lucky enough to have never been on either side of one. PIP stands for Performance Improvement Plan. You get a 30/60/90 plan to hit certain marks, improve your performance, or you are let go. You can accept this plan and try to pass, or you can opt-out and take a severance. The severance at this company was 90 days of continued employment, paycheck and benefits only, you didn't have to show up or log in, plus a number of additional weeks of pay based on your level. The open secret was that almost nobody passed a PIP here. So the real choice was whether you wanted to spend three months trying to survive a process built for you to fail, or just take the severance and go.

None of this was unique to where I worked. It's a known pattern. It's the GE vitality curve, the same forced-ranking philosophy that made Jack Welch famous, cut the bottom 10% every year, no exceptions. A lot of companies still run some version of it. And what it mostly taught people, everywhere it's been tried, wasn't how to compete better. It was how to sabotage each other instead.

And now reality happens

All of that was the process, the mechanisms, and now it becomes a problem. A tiny snag.

People have lives outside of their job.

Somewhere in the middle of one of these calibrations, we'd talk about someone, and the numbers next to their name would look weak. Fewer commits, less visible output, a thinner story than the person next to them. And next to everyone else's numbers, they'd start to look like a good scapegoat. Not your team member. If they're the bottom, you don't have to have the awkward conversation. But somewhere in the room, someone would know, or half know, or suspect, that the reason wasn't performance. The reason was that person had been out. Medical leave, parental leave, some stretch of months where they simply weren't at a keyboard.

Maybe someone might say, "Oh John had a baby."

And the room would go quiet, someone would say, "We aren't supposed to talk about leave."

People would get awkward, because nobody knew what to do with that. We're calibrating on six months' worth of work (or the "whole" year if it's the end of year). Leave is twelve weeks. So they should have output for the other three months or the other six or seven months (remember we calibrated a whole year when it wasn't "done").

Someone would say, "We calibrate based on the work done."

And nobody paused the ranking. The spreadsheet didn't have a column for this person was on leave, exclude this window, or weight it somehow. It had a column for output. And output was what got argued about.

That's the part that's stuck with me longer than the forced curve. Long after I left that company. It wasn't that anyone in that room wanted to punish someone for being on leave. Nobody I sat across from was that person. It's that the process gave nobody a clean way to do the right thing. You could hesitate. You could push back informally, vouch for someone, try to shield a name without being able to say exactly why. But that was on you, individually, in the moment, with no real backing from the process itself. No one knew how to not-count a leave window.

I remember clearly one of my engineers planning to take leave for a new baby and worrying out loud that they'd be punished for it. They had me. I knew the process. I knew how to play the "game" in the calibration room. I knew the work they'd done, and I knew how to tell that story with enough weight behind it to protect them. I also knew what everyone else in the org was doing, because it was a small org, and I had that visibility.

Most people don't get that manager. They don't get that certainty.

So forget about the algorithm

I want to say I got lucky. But I also created that luck. I'm a "loudmouth", someone who knows how to talk to people, network, volunteer for events, speak at conferences, basically not the stereotypical engineer. I was highly involved in several parts of the org. People knew me, and I knew them, for better or worse.

I have a very distinct memory of a new manager. They just started maybe two months before the calibration window (which also meant they personally weren't going to get calibrated). But they weren't there long enough to really know their team, and they were sitting in calibration without their director there to back them up (and calibration is brutal trust me, I would end days opening a bottle of wine).

So they didn't have the history. They didn't know the stories to tell or even the stories people cared about. For example, nobody cared about Salesforce contributions (it wasn't considered engineering) so you wouldn't waste time talking about it (don't get up in arms, that's just what it was like). They also didn't have anyone senior in the room willing to spend political capital vouching for a name that looked weak on paper. So when a thin number came up, there was nobody positioned to say, "Wait! Here's the context you're missing." The room did what the room does when nobody pushes back. It moved on, and their engineers drifted towards the bottom.

That's the "what" I want you to sit with. Not the algorithm. This... "mechanism". Whether or not someone survived calibration with a leave gap in their record depended on who happened to be in the room. It depended on tenure, relationships, whether your manager knew the org well enough to tell a good story, whether their director or VP showed up. None of that is written down anywhere. None of it is even a rule.

It's luck.

So let's go to why I am writing this. Meta is being sued by twenty-six employees who say an AI system (which used activity data and token usage to rank them), couldn't account for protected leave. I'm not shocked. Honestly, it is a completely predictable outcome. Because, as I just told y'all, I sat in the human version of that system for years. The only difference is that my flawed version had a chance of being caught, because a person with enough seniority in the room might notice and say something. Meta's version doesn't have that chance built in at all. There's no manager to vouch, there's no human hesitation, there's no senior leader walking in late and asking why someone's numbers look off. The dashboard doesn't know there's a story it's missing.

And that's the part that should worry you more than the automation itself. It's not that AI replaced a biased human process. It's that AI replaced the one part of the biased human process that occasionally worked. Someone in the room having a conscience and the standing to act on it. We didn't fix the gap. We just made it so nothing sits in that gap anymore.

Not even luck.

I talk about the Meta lawsuit itself in last Friday's episode of Chaotic Commits. This piece is the part I couldn't fit in cleanly (what I lived through before any of it had a model attached to it).