
fix: removed nodeAffinity to the CEO
About this episode
In 2015 I was in the final rounds of a SpaceX interview when someone told me, very casually, that Elon doesn't like people who started in defense. I started in defense. I thought about that interview all week, because this week Nvidia lost hundreds of billions of dollars in a few days, not because the product broke but because one man stopped being CEO. That is not a scandal, it is an architecture diagram: a health check pinging /ceo instead of /health, node affinity pointing thirty thousand pods at one machine, sixty direct reports drawn as a star topology. Then the people who built their single points of failure on purpose, and four things to do about the one in your own org.
Watch
Show Notes#
It's 2015 and I'm in the final rounds of an interview at SpaceX. It's going fine. Good conversations, real engineering questions. Then I get to the expectations talk: plan for ninety-hour weeks, and you'll want a command station at home, minimum of three monitors. At my house. Like the company had hardware requirements for my living room. And somewhere in the middle of all that, someone says to me, as casually as if they were telling me where the break room is, oh, Elon doesn't like people that started in defense.
I started in defense. My first job was at Lockheed. Nobody could point me to a policy or a document. Everybody just seemed to know what one man liked. I didn't go.
I thought about that interview all week, because this week Jensen Huang resigned as CEO of Nvidia and the company lost roughly half a trillion dollars in market value in under a week. Not because the product broke. The GPUs are the same GPUs. The engineers went to work Thursday morning. The board said operations and the product roadmap continue without interruption, and the market said half a trillion dollars changed. One of those two statements is wrong, and I think both of them are true at once, which is worse.
So this episode is not a gossip episode. It's an architecture review. Nvidia's health check has been hitting /ceo instead of /health, and when one endpoint stopped answering the whole fleet got pulled out of rotation. I get into node affinity, what happens when thirty thousand pods are pinned to one machine, and why sixty direct reports with no one-on-ones is a star topology with one very famous property. There's a fairness beat for Apple and Steve Jobs, the real counterargument, and a detail I can't get past: analysts said he had no natural successor, and then named five internal candidates by lunch.
Then the part where I stop being gentle. Nvidia is the least bad of the three, because its board could eventually act. I go through the public record on Elon Musk (the 2018 SEC settlement that forced him off as chair, the Delaware pay ruling and the move to Texas, the 10-K that lists him as a risk factor, the printed-out code review at Twitter) and on Mark Zuckerberg (around thirteen percent of the economics, around sixty percent of the vote, and a shareholder proposal to fix that which is voted on by the person it would constrain). When the reviewer rejects your pull request, the normal response is to fix the code, not to move the repo somewhere without branch protection.
The episode also breaks mid-recording, because it did. Reuters reported a third affair while I was recording, and the throwaway hobbit joke from twenty minutes earlier came back to collect.
It ends with four things to do on Monday: find the node affinity, give the root account a reviewer, run the failover drill by sending your hero on a real vacation, and read the expectations talk in your next interview as an architecture review. A company is a distributed system. If it can't survive losing one node, it was never distributed. It was just big.
Read full transcript
It's 2015, and I'm in the final rounds of an interview at SpaceX. And it's going fine. There's good conversations. There's real engineering questions. I'm thinking, okay, this might be something. Maybe I will move to California. It could be a thing. And then I get to the expectations talk. Plan for 90-hour weeks. Okay. And you'll want a command station at home. Minimum of three monitors. Minimum three monitors. At my house. Like the company had hardware requirements for my living room. And somewhere in the middle of all that, someone says to me, very casually, like they're telling me where the break room is, oh, Elon doesn't like people that started in defense.
I started in defense. My first job was at Lockheed. Defense is literally where I learned to do this. And I remember sitting there thinking, why? Why is that a thing? Why does where I have started had anything to do with whether I can build the thing you need built.
I still don't know. And I'm not really sure the person who said it knew either. It wasn't a policy. Nobody pointed me to a document. It was just a thing. Everybody seemed to know what one man liked. I didn't go. 90-hour weeks in a home command station. It just wasn't my jam. But I thought about that a lot this week because this week I watched one of the most valuable companies on the planet lose hundreds of billions of dollars in a few days. Not because the product broke, but because one man stopped being the CEO.
Welcome back to Chaotic Commits. I'm Joanne Skiles. I've been writing software for over 16 years across defense, entertainment, healthcare, finance, and AI, and I've been a director of engineering. I teach computer science, and these days I help companies design their AI systems, the real architecture underneath, not the thin layer on top. Today, we are talking about Jensen Huang, who resigned as CEO of Nvidia this week, effective immediately. And yes, we're going to talk about why, and yes, there will be jokes. But mostly we're going to talk about something I think that's a lot more interesting than the gossip and a lot more useful to you when a company can lose that much value because one person walked out the door. That's not a scandal. That's an architecture diagram. And it's one you probably have a smaller version of at your own job.
Here's how we'll do this. We're going to walk through what happened, then why it's a single point of failure, then the people in tech who built theirs on purpose, and then what you can actually do about the one that's in your org right now.
Let's start with what happened. Last week, Jensen Huang sent an email to Nvidia staff admitting to an affair. The report around it is that the woman is a businesswoman who US intelligence reportedly suspects of working for Chinese intelligence. And I'm gonna be clear here, that part is an allegation. It hasn't been proven and I'm not going to pretend I know more than reporting does. The board opened an investigation and then the board said publicly that it had no reason to believe Nvidia's secrets or national security had been compromised, and that it still had confidence in him. So: one affair, board says we're good.
Then, Wednesday night, the New York Times reports a second affair. This one with a senior employee at the company. A subordinate. The board met that night. And Thursday morning, he's out. Effective immediately. The CFO, Colette Kress, is interim CEO. And look, most of us listening here know about Hobbits. breakfast, second breakfast, elevenses There's a whole system. Nvidia's board, it turns out, is not a hobbit. One breakfast, they'll sit with that. They'll issue a statement of confidence about the breakfast. Second breakfast, second breakfast is where they draw the line. You're leaving the Shire.
Okay, I'm going to say one serious thing about the second one and then we're going to move on because this is not a gossip show. Okay, this is a tech show. A relationship with someone in your own company below you when you're the CEO isn't a private matter in the way people like to pretend. It's a conflict of interest sitting in the middle of the org chart. In engineering, we have a name for when a person with power to approve something has a personal stake in it. We call it a problem and we build systems so it can't happen quietly. OK. Now the part I actually want to talk about.
The stock.
As of 4 o'clock Thursday afternoon, right as the market closed, here's where it stands. Nvidia opened down at about 13%. It clawed some of that back during the day and it finished the afternoon down about 5%. around $165 a share, just under $4 trillion in market value. And since last Friday, before any of this broke, that's roughly half a trillion dollars gone in less than a week. Half a trillion. And here's what the board said in its statement same morning, operations, customer commitments, and product roadmap continue without interruption. Earnings are still happening on schedule next week. So I want you to hold two things next to each other. The board says nothing about the product change. The market says half a trillion dollars changed.
The GPUs are the same GPUs they were last Friday. The software stack is the same. The engineers who build it went to work Thursday morning. The data centers did not notice. If you ask a GPU how it felt about the news, it would just keep doing matrix multiplication because that's what it does. And it's very good at it.
One of those two statements is wrong. Either the board is wrong and the company really was that dependent on one man or the market is wrong. And we have a trillion dollar habit of pricing companies based on a personality instead of what they build.
And unfortunately, here's the actual thing. Both answers are bad. And I'm gonna argue both of them are actually true at the same time. But I want to tell you a quick story first. A few episodes ago, I did an episode about Jensen Huang going on stage at Dreamforce and saying that safety is an engineering problem and that market forces are already there. So we don't need any new laws. And I spent the whole episode reviewing that claim like a pull request.
And near the end I said, here's what I tell him if I ever got two minutes with him. And I also said, I won't. Well, his calendar just opened up and I find I don't want two minutes anymore. If I'm going to get two minutes with anybody at Nvidia right now, it's Colette Kress because she's the one running the thing that actually matters, the company, the thing with tens of thousands of people in it still shipping. And I think the fact that that sentence feels a little strange to say out loud that the person who matters now is someone most people have never heard of is the whole episode.
So let me put this in terms I can actually trust. If you run anything in production behind a load balancer, you know what a health check is. The load balancer asks each instance every few seconds, are you okay? And the instance answers. If it says yes, it gets traffic. If it doesn't answer, it gets pulled out of rotation. The whole art of a health check is deciding what it checks. And there's a classic mistake. You write a health check that doesn't ask is this service okay? It also calls some other dependency and then one day that dependency hiccups and every single healthy instance fails its health check at once and the load balancer pulls your entire fleet out of rotation.
Your service was fine. Your service was serving requests and it went down anyway because you wired something into the health check that never should have been there. That's what the market did this week. Nvidia's health check has been hitting /ceo instead of /health. Every service returns 200. The roadmap is green. The customers are still customers and the whole thing got pulled out of rotation because one endpoint stopped answering.
Now is that the market's fault? Partly, but somebody wired it that way. And I don't mean one person. I mean a whole industry and a whole culture that spent years telling the story of the company as the story of the guy, the keynotes, the leather jacket, the magazine covers, the one of a kind founder. When the story of a company is one person, the market believes you. Of course it does. You told that for 30 years.
Here's another way to think about it. In Kubernetes, there's a setting called node affinity. It lets you say, I want these pods to run on this particular node. There are real reasons to use it.
Sometimes a workload needs special hardware. But imagine a deployment with 30,000 pods and every single one of them has node affinity to one node. You got a whole cluster, hundreds of healthy machines, and everything is pinned to one. When that node goes down, nothing reschedules. All that capacity sitting right there, and the scheduler can't use any of it because you told it everything has to live in one place.
That's not a distributed system. That's a big system that happens to have a lot of machines on it. And at Nvidia, this wasn't a story the market told. It was a management style. Huang had talked openly about having around 60 direct reports and about not doing one-on-ones with them. He said the point is that everybody hears the same information at the same time and there are fewer layers that way.
And I can kind of see the appeal. I've worked in companies with so many layers the information was stale by the time it got anywhere. But if you draw that org chart as a network diagram, do you know what you've drawn? A star, one hub, 60 spokes, and star topologies have one very famous property. They're fast and simple right up until the hub fails, and then every spoke is talking to nothing.
I spent years of my PhD working on vehicle networks. That's cars talking to other cars. And the first thing you learn in that world is that any node can drive away at any time. A car can turn left. It can park. It can go into a tunnel. So you never ever design routing that depends on one particular car staying put. If your protocol only works because one specific node never leaves, that's not a protocol. That's just a lot of hope. Companies are like that too. Because people leave. They retire. They get sick. They get hit by a proverbial bus. And sometimes they have a second breakfast.
Now, I want to be fair here.
Because there is a real counter argument. Sometimes the founder really is that important. They're that load-bearing. The best known example is Apple. Steve Jobs got pushed out in 1985. And over the next decade or so, Apple got very close to running out of road and it took Jobs coming back to turn it around.
So the market isn't crazy to wonder whether vision walks out the door with a person. But I'd say this, if that's true, it's not a defense. That's a confession. If a company that size truly can't function without one specific human. Then somebody built a four trillion dollar single point of failure and they called it strategy. And notice what Apple did the second time. When Jobs got sick, Tim Cook had already stepped in as acting CEO more than once before he took the job for real. The second time Apple had a failover and they tested it.
Here's the detail I think tells you the most. In the coverage this week, analysts quoted saying Huang had no natural successor inside the company. And then within hours, those same news cycles listed five internal candidates by name. The CFO, the head of the data center business, the chief scientist, the CTO, a head of applied deep learning research. The bench was so deep, analysts named five people by lunch.
They just never had their names advertised. And one more. When the board announced this, it also named a longtime director, someone who's been on the board since 1993, as the company's first ever board chair. First. Ever. 33 years. They added a reviewer the morning after the incident, which, if you've ever worked anywhere, is exactly when everyone adds the reviewer.
And I'll say one fair thing about the failover too. The interim CEO is also still the CFO. So the backup node is now running two workloads at once. That's not a criticism of her. That's what it looks like when the failover was never actually provisioned. You grab the most capable person you've got and you ask her to carry both.
One person isn't a company. If he was, why did he need all those people? Okay, wait, actually, I have to stop now. While I was recording this, literally while I was recording the part you just heard, Reuters reported a third affair.
This one is another Nvidia employee. Reuters didn't name her and neither am I. It's based on people familiar with the situation. Nvidia declined to comment and he's only publicly acknowledged one. So that's where it stands right now. So I did the hobbit bit earlier, breakfast, second breakfast, and I said elevenses as a throwaway because I really thought it was going to be funny and I didn't think I would need it. But we're at elevenses now, everybody.
OK, so back to the engineering part and it's a real one. He's acknowledged one. The reporting is at three. That gap is the difference between an incident report and the logs. Anyone who's run a real incident knows this pattern. You find the first problem and you write it up and you think you're done. And then somebody actually goes and reads the logs and there's a second one. And once there's two, you stop asking what happened and start asking, how long has this been going on and how wide does it go? Because now you have widened the scope and you gotta go looking.
The first apology was scoped to the first incident. The investigation wasn't. It never is if it's a real one. And that's why you don't close a ticket on the first root cause you find. Not in prod and apparently not in the C-suite either. Okay, so I've been relatively gentle so far, believe it or not, and I'm going to stop, because Nvidia honestly isn't the worst version of this problem. At Nvidia, the systems eventually worked. Late, after second breakfast, but a board looked at the situation and took action. A reviewer existed when it mattered. It hit the button. There are a couple of people in tech who built their companies so that the button doesn't work.
And I don't like either of them, and I'm going to tell you why. And it's going to be about architecture. So let's go back to that SpaceX interview. Elon doesn't like people that started in defense.
Think about what that actually is as a system. It's a filter in the hiring pipeline, a config value, something like reject if started in defense, except there's no ticket, no design doc, no owner. Nobody can tell you why it's there. The person enforcing it never heard him say it. Probably. They just knew. Everyone knew. That's what happens when a company's culture is one person's taste.
His preferences get cached, copied, served from the edge by hundreds of interviewers and managers guessing what he'd want. And nobody ever invalidates the cache. And here's the part that makes me laugh. Because SpaceX is now one of the biggest national security contractors in the country. It won the majority of a recent multi-billion dollar batch of military launch missions. It builds satellite networks for defense and intelligence customers. The company that didn't like people who started in defense is kind of looking like a defense company. The filter didn't age well. Filters built on a mood usually don't.
And the 90-hour weeks in the home command station with three monitors, I want to say something about that because it's the same failure. It's just wearing a lanyard. When everything routes through one person's pace and one person's vision, You don't build failover. You make every human in the building run hot all the time and call it intensity.
It's a GPU that only benchmarks when it's overclocked. You've seen those charts. Incredible numbers. And then it's August and the room's really warm and the thing thermal throttles and the fans start to sound like a leaf blower trying to take off. It's not fast. It's running past spec and calling the heat a culture. And a 90-hour work week isn't just a schedule. It's a hiring filter too because it quietly screens out anyone who has somebody waiting at home, anyone caring for kids or parents, anyone who would like to still be employable in 10 years. So now your one person's taste is enforced by a pipeline that only lets through people who can live like him and for him. That's how you get a room where everyone looks the same, thinks the same, and misses the same things.
Now, the public record.
In 2018, Musk tweeted that he had funding secured to take Tesla private. He didn't. The SEC settled with him and with Tesla. Fines for both. And he had to step down as chairman of the board for three years and have his communications about the company reviewed. So Tesla got a separate board chair the same way Nvidia did. After an incident, except at Tesla, it took a federal regulator to add the reviewer.
Then there's the pay package. A Delaware judge looked at his roughly $56 billion compensation package in 2024 and threw it out. And a big part of her reasoning was that she didn't think the board negotiating it was really independent of him. And what did Tesla do? It moved. It reincorporated in Texas. To be fair, the Delaware Supreme Court later restored the package at the end of 2025. So he won that round in the end. But I keep coming back to the move. When the reviewer rejects your pull request, the normal response is to fix the code, not to move the repo somewhere without branch protection.
And Tesla's own annual filings have for years listed it as a risk factor that the company is highly dependent on him. So when your SEC filing has to warn investors that one guy is a dependency, your architecture diagram has a problem, and at least they're honest about it. And then Twitter, when he bought it in 2022, engineers were reportedly asked to print out recent code so it could be reviewed, printed on paper. The most expensive code review in history done in a conference room with no diff and no context by one man deciding who stayed and then most of the company was gone. I think every one of those is the same decision made over and over. Every time somebody tries to add a reviewer, he routes around it, and people call it vision.
Zuckerberg is a different version, and honestly it's the scarier one. Meta has two classes of stock. Zuckerberg owns something like 13 or 14 percent of the company's economics. But because his shares carry 10 votes each, he controls around 60 percent of the vote, which means he can't really be voted out by anyone ever. And every year shareholders put up a proposal to move one share, one vote. And every year it fails because the vote on whether he should control the vote It's controlled by him, guys. This year, it got about a quarter of the vote. The pull request to add a reviewer gets reviewed by the author. Every year. Closed, won't fix.
In security terms, that's a root account with no audit trail and no way to revoke the key. And the key holder voted to keep it that way.
And here's what that architecture costs. In 2021, he renamed the entire company after a bet on the metaverse.
The metaverse, guys. We know where that went, right? Because by the end of 2025, the division doing that work, Reality Labs, had lost something like $77 billion. $77 billion over about five years. Most of us would have been in a very uncomfortable meeting after the first billion.
I'm not saying no one at Meta raised a hand. I'm sure a lot of people did. I'm saying that there is no mechanism that could make a raised hand matter. That's what you get when a single point of failure has a hobby.
He has a very expensive hobby. So here's how I rank them. Not as people, but as systems. Nvidia has a single point of failure and a board that could eventually act. Too late after second breakfast, but it did act.
Tesla has a single point of failure and when regulators and courts tried to add reviewers, the company moved the repo. Meta has a single point of failure with a deadbolt on the inside. And Nvidia is the best of those three. So sit with that for a second. The company that just lost hundreds of billions of dollars in a week is the good example.
OK, you're probably not on the board of a trillion dollar company. And if you are, hi, I actually have some notes. So, you know, email me. I'm all for it. But you almost certainly have a smaller version of this at work. Every org I've ever been part of has had one. So here's what I do on Monday.
One, find the node affinity. Ask: if this one person or this one service disappeared tomorrow, what stops working? Because it's not always the CEO. It could be your staff engineer who's the only one who understands the billing system, the founder who still personally approves every pricing change, the one service account everything authenticates through, the vendor whose contract nobody has read. If you can name it in five seconds, guess what? That's real. Two. Give the root account a reviewer. The most powerful person or system in your org should be the most observed, not the least. That doesn't mean you don't trust them. It means trust isn't the control. In any decent cloud setup, even your admins can't quietly grant themselves new permissions. It shows up in the audit log that somebody else reads. So do the same for your decisions. Who can say no to the most senior person in your org? Not in theory, in practice. If the answer is nobody, you found a bug.
Three, run the failover drill. This one's my favorite. The best chaos engineering experiment in most companies is sending your hero on a vacation. I mean a real vacation. Laptops closed, no slack, two weeks, preferably up on a mountain or in the middle of the ocean so they get no cell phone service. Okay, watch what breaks and write it down. That's your runbook and you just got it for the price of a vacation instead of a crisis. Four, if you're interviewing, read the expectations talk as an architecture review.
When a company tells you to plan for 90 hours a week and keep a command station at home, it's not just telling you about the job. It's telling you it has no failover. You're it. That's the useful thing to know before you sign. And for the leaders listening, if you built a team where everything routes through you, that probably feels like you're important. It's not the same thing as being good at your job. Actually, you might not be good at your job. The job is building something that keeps working when you're not in the room.
I think about the person in that SpaceX interview who told me Elon doesn't like people who started in defense. I don't think they were a bad person. I think they were doing what everybody in the system like that learns to do. Predict what one man wants and filter for it before anyone has to ask him. And this week, I watched thousands of people at Nvidia wake up to find out that the person they had been predicting was gone.
And the company was still there. The chips still work. The roadmap still exists. The people were still the people. And that's the part that the stock price can't see. It was never one person. It just looked that way from the outside. Because that's the story we kept telling. Here's the lesson. And it's the one I want you to take back you're building. A company is a distributed system.
If it can't survive losing one node, it was never distributed. It was just big. So go find the node everything is pinned to in your codebase, in your team, in your org, start writing the runbook while they're still around to help you write it. And one more thing, as I'm recording this, the count is three. This episode comes out tomorrow. Who knows how many will have been reported by the time you're hearing this?
Hobbits for the record. Eat seven meals a day. Breakfast, second breakfast, elevenses, luncheon, afternoon tea, dinner, supper. I'm not saying that's a forecast. I'm just saying I've left room in the show notes. That's this episode of Chaotic Commits. I'm Joanne Skiles. Still running exactly two monitors at home, one of them extra long, purely out of spite since 2015. New episodes wherever you get podcasts, no highlight reel, just the commit history.