
log: the bug was older than the headline
About this episode
At the end of October the US government gave every civilian federal agency three days to patch two bugs in PHP. Amazon's honeypots had caught somebody trying it four days before the fix existed. That was the headline. The part that wasn't: the oldest piece of that code path reportedly goes back to March 2012. Every bug has two birthdays, the day somebody wrote it and the day the rest of us found out, and this episode lines up four of them. PHP this fall, Shellshock from around 1989, Heartbleed written on New Year's Eve 2011, and Log4Shell, a 2013 feature that turned a Minecraft chat message into code execution eight years later. Then the pattern proved itself again while I was finishing the episode, and four things a team can actually do about it.
Watch
Show Notes#
At the end of October, the US government gave every civilian federal agency three days to patch two bugs in PHP, the language that runs a large share of the websites where anyone can tell what's running on the server. No login required. Send the right request, and you're running your own code on somebody else's machine. People were already trying it. One of Amazon's honeypot servers caught somebody attempting it on October 17th, four days before the fix existed.
That's the headline. Here's the part that didn't make it: according to the research teams that found it, the oldest piece of that code path goes back to PHP 5.4, which came out in March of 2012. It wasn't news in 2012. It was just code. It became news in 2026.
Every bug has two birthdays. The day somebody wrote it, and the day the rest of us found out. The gap between those two days is what this whole episode is about.
This is a history episode, not a history of PHP exactly, a history of a certain kind of bug: the kind that's older than the news about it, sometimes by a decade or more, living in software everybody uses and almost nobody is paid to look at. I line up four of them. The PHP bugs from this fall, including the parts that came after the patch, the chain through a Linux kernel bug to root that CISA documented in its own catalog, and the campaign that ended in wiped VMware hosts. Shellshock, a bash bug that was news in 2014 and went back to around 1989, maintained for decades largely by one volunteer. That's not a bug, that's a bug with a mortgage. Heartbleed, committed on New Year's Eve 2011, shipped in March 2012, the same month as PHP 5.4, found in April 2014, in a project that was getting about two thousand dollars a year in donations. And Log4Shell, the one that ruined my December in 2021 and still makes me a little wary every December for no reason, where a logging library would read what it was writing, go fetch code from wherever the text pointed, and run it. Some of the earliest exploits were people typing the attack into Minecraft chat.
There are two images in this one and then I stop. The first is starlight. When you look up at a star tonight you're not seeing the star, you're seeing light that left it a long time ago, and the star could have changed completely since. Security news works the same way. The headline is the light arriving. The second is the volunteer fire department: people with day jobs and a pager on the nightstand, in towns that have never once asked who's driving the truck.
Then the episode got its own live proof. A few weeks after the October patch, PHP shipped another critical security release, and one of the bugs in it has a name. October's fix closed off the route the attack was taking and left the underlying file write alone, so a team went and looked at the part nobody fixed and found another way in. If you patched in October and closed the ticket, you were patched against the exploit, not against the thing underneath it, and you stayed that way for four weeks with no way of knowing. That isn't a knock on anybody. It's what it looks like when the light arrives in pieces.
It ends with four things to do that aren't exotic: know what you're actually running rather than what you think you're running, put end-of-life dates on a calendar the way you'd track a lease (PHP 8.2's own support ends December 31st, about six weeks after this one airs), budget upgrades as routine work instead of a heroic one-time patch, and fund the fire department. You can't stop old bugs from existing. You can decide how you find out: from a headline with a three-day deadline, or from a list you already keep.
Read full transcript
At the end of October, the US government gave every civilian federal agency three days. Three days to patch two bugs in PHP, the language that runs, depending on whose count you trust, about 70% of the websites where anyone can tell what's running on the server. No login required, send the right request, and you're running your own code on somebody else's server.
People were already trying it. One of Amazon's honeypot servers caught somebody attempting it on October 17th, four days before the fix existed. And that's the part that made the news. That's the headline. Here's the part that didn't make the headline.
According to the research teams that found it, the oldest piece of that code path goes back to PHP 5.4. PHP 5.4 came out in March of 2012. That code path has been on the internet longer than my kids have been alive. It's been through dozens of PHP releases, a couple of hosting migrations for basically everybody, and the every "Is PHP dead?" blog post written in the last fourteen years.
It wasn't news in twenty twelve. It was just code. It became news in twenty twenty-six. Every bug has two birthdays: the day somebody wrote it, and the day the rest of us found out. And the gap between those two days is what this whole episode is about.
Welcome back to Chaotic Commits. I'm Joanne Skiles. I'm an engineer. I teach computer science. I advise companies on how their AI systems are actually designed, and my research is on transparent systems, which mostly means I spend a lot of time asking, "How would anyone know about things that are already running?"
This one's a history episode. Not a history of PHP exactly, a history of a certain kind of bug, the kind that's older than the news about it, sometimes by a decade or more. The kind that lives in the software everybody uses and almost nobody is paid to look at. We're gonna look at four of them, the PHP bugs from this fall, and then three older ones that I think are the same story told in earlier years: Shellshock, Heartbleed, Log4Shell.
One of those ruined my December, and we'll get there. And then we'll talk about what the pattern says about how we build and who's driving the truck. So let's get into it.
PHP is a great place to start a history episode because PHP's history is actually a little ridiculous. In 1994, a programmer named Rasmus Lerdorf wrote some small programs in C to keep track of who was visiting his online resume, personal stuff, a hit counter, basically, plus some form handling.
He called the bundle Personal Home Page Tools. That's what PHP originally stood for, personal home page. He put it out publicly in 1995. Other people started using it. They asked for features, he added features, and it kept growing one reasonable next step at a time. Lerdorf has said versions of this for years, that there was never a plan to write a programming language. He just kept adding the next logical step.
And this is where I want to be clear because I'm not dunking on that. That's how most software actually gets built. Somebody solves their own small problem, the solution is useful, other people show up, and the small problem turns into everybody's problem. Fast-forward 30 years, PHP runs WordPress.
WordPress alone runs a huge share of the web, plus a long list of stores, school sites, church sites, local government sites, and the website for the plumber who came to your house last spring. And I'm not saying any of this from the outside. I've written a lot of PHP, unfortunately.
And I maintained my share of legacy PHP, the kind where you open a file, see who wrote it and when, and then you just quietly close it and move on.
So now the October bugs, two of them, both critical, both found by outside researchers, including teams from Wiz and Calif, and researchers at OpenAI. I'm not going to read you the whole advisory, but the shape of them is worth knowing.
One of them lives in what PHP does when it runs out of memory halfway through building a request. It bails out, and then the cleanup code comes along afterward and treats the half-built thing as if it were finished. The other is a leftover. One request leaves something behind on a worker, and then the next request, which should be completely separate from the first one, picks that thing up and uses it after it should have been gone. Neither of those is a typo.
They're plumbing.
And then the part that I think matters most for this episode, PHP shipped versions for the fixes it still supports 8.2 and up. If you're on 8.1 or 8.0 or 7.4, you're past end of life. There's no release coming for you. So here's what the PHP security team did instead. They hand wrote the patches for those three dead branches anyways, and pushed them up as source code, not releases, not packages, source, which you have to go apply yourself.
And somebody else in the community went further and built packages going back to PHP 5.6. That's work nobody owed anybody, on versions nobody supports, done on a day that I promise you was not going well for any of them.
And by one count, a real slice of PHP sites out there are still running PHP five, not seven, five. A code path from twenty twelve, a bug found in twenty twenty-six sitting in versions some people never left. And then the ordinary predictable next thing happened.
CISA put both bugs on its known exploited list on October twenty-sixth. That's the three days I opened with, and within about a week, somebody had worked out how to chain the PHP bugs together with a Linux kernel bug and go all the way to the root. I'm not inferring that. CISA wrote the chain down in its own catalog, implant a web shell, run commands as root.
And researchers at Checkpoint reported a campaign that started on October thirtieth, where the people who got in through this didn't stop at the web shell. They moved sideways into VMware hosts and wiped them on the way out. Here's the thing I keep coming back to. The bug didn't get worse this October.
The code sat there unchanged the whole time. What changed is that somebody looked and then somebody worked out how to reach it. If you think the PHP story is weird, wait until you see how normal this story actually is. When you look up at a star tonight, you're not seeing the star. You're seeing light that left it a long time ago.
Years, sometimes a lot of years. The star could have changed completely since then, and you have no way of knowing from looking. Security news works kind of like that. The headline is the light arriving. The bug left the star a long time ago. So let's go back to September twenty fourteen, Shellshock.
A researcher found that Bash, the shell that's on basically every Linux machine, every Mac at the time, and an enormous number of servers would run commands hidden inside an environment variable if you shaped it the right way.
Bash had a feature where you could pass a function to a new shell through an environment variable, and the code that read that variable didn't stop reading where the function ended. Anything you tacked on after, Bash just ran. And web servers at the time were constantly stuffing request data into environment variables.
So you could put your attack in, for example, the user agent header of a web request, and a server somewhere would run it. Here are the two birthdays. It was news in twenty fourteen. The bug went back to around nineteen eighty-nine. Twenty-five years. That's not a bug. That's a bug that has a mortgage.
'Cause I had a mortgage at that time, and I was born in nineteen eighty-nine. And Bash at the time, was maintained by essentially one person, Chet Ramey, for decades, largely as a volunteer. One person, every server on the internet. Now, go back six months before that, April twenty fourteen, Heartbleed.
OpenSSL the library, a huge chunk of the internet used to do the S in HTTPS. It has an extension called heartbeat, which is exactly what it sounds like. One side says, "Are you still there? Here's a word.
Send it back to me." The other side sends the word back, except the code trusted you about how long the word was. You could send a three-letter word and say, "This is sixty-four thousand letters long. Please send it back," and the server would. It'd send you back your word and then the next chunk of whatever happened to be sitting in its memory: passwords, session cookies, sometimes the private keys.
That bug had two birthdays again. It was news in April 2014. The code was committed on New Year's Eve 2011. It shipped in OpenSSL 1.0.1 in March of 2012. And if you're paying attention to the dates, yes, March 2012 is the same month PHP 5.4 came out.
Two of the bugs in this episode shipped to the world in the same month in two different projects, and nobody would know about either of one for years. And OpenSSL, the thing protecting a big chunk of the world's web traffic, right after Heartbleed, the head of OpenSSL Software Foundation wrote publicly that it was getting about $2,000 a year in donations.
$2,000 a year. Here's the second image, and then I'll stop with images. Most people assume that when you call about a fire, a truck full of full-time paid firefighters show up, and in a lot of places, that is true. But in a lot of towns, more than most people realize, the fire department is volunteers.
They're people with day jobs, an accountant, a mechanic, a teacher, who keeps a pager on the nightstand and gets up at 2:00 in the morning for free. And most people in those towns have no idea. They've never asked. They never had to. When you call, the truck comes.
It always comes. And that's most of the internet, a volunteer fire department, and a whole town that's never once asked who's driving the truck.
And then there's the big one, December 2021, Log4Shell. This is the one that ruined my December, and I'll be honest, it still kind of does. Every year when December comes around, I get a little wary for no reason. Nothing's wrong. Nobody paged me. It's just December, and some of my brain is still waiting for a logging library to ruin it.
Log4j is a logging library for Java. It's how a huge amount of Java software writes down what happened. User logged in, request failed, here's the error. It's everywhere. In enterprise software, cloud services, games, appliances, things you would never think as running Java. And Log4j had a feature called lookups.
If a log message contained a certain kind of specially formatted text, a lookup expression, Log4j would helpfully go fill it in, including through something called JNDI, by reaching out to another server on the network and loading what it said. Think about that for a second. The library that writes things down would, under the right conditions, read what it was writing down, go fetch code from wherever that text pointed, and run it.
And the thing being written down is very often text a stranger gave you, a username, a search box, a chat message. Some of the earliest public exploits were people typing the attack into Minecraft chat. You type the line into a video game chat box, the server logged it, and the logger went and ran your code. The director of CISA at the time called it one of the most serious vulnerabilities she'd seen in her entire career, if not the most serious. Two birthdays again. It was news in December 2021. The lookup feature that made it possible was added in 2013. Eight years in a logging library, the single most boring kind of software there is, the library you add on day one of a project and never think about again.
And Log4j was maintained by a small group of people at the Apache Software Foundation, a lot of it on their own time, people who spent their December being yelled at by the internet for a feature they'd built to be helpful. Afterward, the US Government Cyber Safety Review Board wrote a report about it, and the word they used was endemic.
Their point was that this wasn't going to be patched and done. Vulnerable versions would be out there for years, a decade or more, which is exactly the PHP five story. The fix exists. The old version is still out there anyway. Line them up and the pattern is hard to miss, okay?
Shellshock, a feature written around 1989, news in 2014.
Heartbleed, a feature written in 2011, news in 2014.
Log4Shell, a feature written in 2013, news in 2021.
PHP plumbing, code going back to 2012, news in 2026.
Not one of them is a dumb mistake by careless people. Three of them are features. Somebody was trying to be helpful. The fourth is the code that runs when things go wrong, which is the code that gets tested the least. And every one of them lived in software that everybody uses, and whose maintenance almost nobody thinks about paying for.
The bug is never the newest thing in the story. It's the oldest thing in the story. It's just the last thing we notice.
What's actually changing now? Because I don't think the answer is nothing, this is just how it is. The first thing that's changing is the speed of finding them.
The PHP bugs this fall were found partly by researchers at an AI company. That's not a coincidence. The tools for reading huge amounts of old code and asking, "What happens if this goes wrong halfway through?" are getting a lot better very fast. And one of the security firms on the finding end said something that I just keep rereading.
Their position was that the underlying flaws have been sitting in there for decades, that actually proving you could exploit them took a chain of steps right at the edge of what human researchers could realistically pull off, and that frontier AI models lowered the bar enough to turn a long-standing flaw into a working exploit.
Sit with that for a second. The vulnerable code was already there. It had been there for years. What was missing was somebody with enough patience, enough time to work out a path to actually exploit it. That constraint just got a lot cheaper, which is good news, and also the same tools are available to people who don't send you an advisory first.
The bugs aren't getting newer. The flashlights are just honestly getting better, and not everybody holding a flashlight is gonna be on your side. And I've got the clearest example of this I've ever had because it happened when I was finishing this episode. A few weeks after the October patch, PHP shipped another critical security release.
Depending on where you read about it, the reports say seven critical vulnerabilities or 18 CVEs. Those aren't counting the same thing, and one of them has a name. A research team called it PHP After Shell. Here's what PHP After Shell is. Back in October, the fix that went out closed off the route the attack was taking.
It restricted what could get loaded during the specific moment the bug lived in. What it did not do was change the underlying file right that made the whole thing possible in the first place. So a team went and looked at the part nobody fixed and found another way in, and the researchers' own line about this episode's thesis says it better than I can say it.
The original exploit stopped working, but the underlying problem was still there, which means if you patched PHP in October and you felt good about it, and you closed the ticket and went back to your actual job, you were patched against the exploit. You were not patched against the thing underneath the exploit, and you stayed that way for four weeks without any way of knowing. That's not a knock on anybody. Nobody in that sentence did anything wrong. It's just what it actually looks like when the light arrives in pieces.
So if the light is going to keep arriving from old stars faster than it used to and in pieces, here's what I actually want a team to do. And none of this is exotic. One, know what you're running, not what you think you're running, what's actually on the box. Which versions of which language, which libraries, which libraries those libraries pull in.
There are tools that will produce this list for you, and the industry calls it a software bill of materials. The name sounds like paperwork. It's the thing that turns, "Are we affected?" from a three-day panic into a ten-minute query. And know that the version number by itself doesn't always answer the question.
Some Linux distributions ship the fix without changing the number on the front of it. So the number can tell you less than you think in both directions.
Two, put end of life dates on a calendar, like a lease. Every language version you run has a date when it stops getting fixes. That date is public. It is known years ahead of time.
And almost every team I've worked with finds out about it the week something breaks. If PHP 8.1 is on your servers right now, the day it stopped getting fixes wasn't this October. It was already a year behind you. And here's the one I'd actually write down, PHP 8.2, the version that did get patched in October, the one you may have upgraded to in a hurry three weeks ago to get out from under this, its security support ends December 31st of this year.
That's about six weeks from now. That one isn't behind you. It's in front of you. It's on the calendar, and it's getting close. Three, budget upgrades like you budget anything else that's guaranteed to happen. Not as a heroic project somebody pitches when they're fed up. As routine work every quarter, the same way you change the oil, and it's the only thing that handles what just happened here. One patch in October, another four weeks later.
A heroic one-time patch is an event. What you actually need is a cadence. The team that upgrades a little all the time never has to upgrade a lot at once under a federal deadline.
Four, fund the fire department. If your company makes money on top of an open source project, find out who maintains it and find out if anybody pays them.
A lot of these projects take donations or sponsorships, and some companies now pledge a fixed amount per developer. Two thousand dollars a year was the number in twenty fourteen. Your company's coffee budget is probably bigger than that. And if you want to know how thin some of this actually is, here's a small one.
On the day the PHP fixes went out, PHP's own announcement had to tell people that one widely used volunteer-maintained package repository wouldn't have updates that day, and that they should get their packages from their distribution or build from source instead. I don't know why nobody published a reason.
I'm not gonna invent one for them. Nothing failed exactly. There were other ways to get it, and people got it. But the biggest PHP patch in years, one of those trucks just wasn't running, and the answer was, "Go build it yourself." And I wanna be fair here because this part matters. The maintainers aren't the villains in any of these stories.
They're the volunteers. They signed up to help, And the whole town quietly started assuming the truck would always come. The problem isn't that the volunteers write bugs. Everybody writes bugs. I have written bugs that I'd rather not describe on this podcast.
The problem is that we built an economy on top of their weekends, and then acted surprised when the truck showed up late.
Here's where I want this to land. Somewhere in the software you depend on right now, there's a bug that's already old. It was written years ago by somebody trying to be helpful in code that everybody uses and almost nobody reads. It doesn't have a name yet. It doesn't have a logo. Nobody's built a website for it.
Four weeks ago, the weakness behind PHP After Shell was one of those. It was just a file write that nobody had looked at in the right way yet. Now, it has a name, a number, and a release note. That's not a reason to panic. It's just true. It's been true the whole time. It was true in nineteen eighty-nine, in twenty eleven, in twenty twelve, and twenty thirteen.
What you get to decide is how you find out. You can find out from a headline with a three-day deadline, or a version that stopped getting releases two years ago. Or you could find out because you knew what you were running. You already moved off the old version, and when the advisory came out, you read it, checked your list, and went to lunch.
That's the whole goal. Not a world with no old bugs. A world where the light arrives, and you're already looking up. That's this episode of Chaotic Commits. I'm Joanne Skiles, and if somebody in your company still has a server on PHP 7.4, send this to them.
If nobody knows whose server it is, that's the episode. New episodes wherever you get podcasts. No highlight reel, just the commit history.