feat: it won't ask you why

When I was in college, my mother's TV broke. Not a small TV. It was the mid-2000s, and it was one of those first big flat panel sets that cost about as much as a used car. She had it maybe two years, and then one day it just stopped turning on. You hit the power button and you get a little clicking sound, a red light blinking on the front, and nothing on the screen. The warranty had run out, and I wanna say about a month. My mother is not a technical person, and she's the first one to tell you that.

She called the store, she called the repair number, she got quoted a price to fix it that was close enough to the cost of a new TV, And it just felt like a joke somebody was playing on her.

So I stepped in. I did the thing I do, which is I read forums mostly, and it turned out the exact model had a known bad part on the power board, a run of components that failed early, always the same way right around the two-year mark. Clicking sound, blinking light, black screen. There were whole threads of people with the identical problem down to the timing.

Nobody official was going to say that out loud because saying it out loud cost the company money. So here's what I did. I found the name and the email of a VP at the company. Not support, not the contact form, an actual executive, a real person whose job had nothing to do with me or my mother's TV. And I wrote to him as my mother.

Polite, specific, calm. Here's the model number, here's the failure, here's the known defect, and here are the dozens of people online describing exactly this. This television is one month out of warranty, and I would like someone to make it right.

And someone did. He wrote back. Then I got on the phone with him, still as my mother, and by the end of that call, there was a repair scheduled. A technician came to the house, opened the set, replaced the exact part from the forums thread, and left. No charge. I was twenty. I had no leverage, no legal standing, no special knowledge except what strangers had posted on the internet.

What I had was an email address, a calm tone, and a willingness to be somebody I wasn't for about ten minutes. Nobody asked a second question. I did that for good. Hold onto it

Welcome back to Chaotic Commits. I'm Joanne Skiles. 16 years building software, a research thread on transparent systems, and a standing interest in the gap between what a system can do and whether it should. This episode is about two things that happened in the same week at the end of this past August.

Two different AI companies, two very different stories. One lesson underneath both of them. The first one is a threat intelligence report from a security firm called Gambit Security about a ransomware crew that had been using a developer tool to do its hands-on break-in work.

The second one is an incident report that OpenAI published about its own AI agents, which got out of its test environment and into somebody else's production infrastructure without anyone telling them to. One of those is a story about a tool doing exactly what a bad actor asked it to do. The other is a story about a tool doing exactly what its own scoring system rewarded, which turned out to be nothing anyone wanted.

Different failure modes, I'll talk about both, and then I'll tell you what they have in common, And I promise you this will all connect back to my mother's TV. So let's get into it

The last week of August, Gambit Security put out a write-up on a ransomware group they tracked under the name Aurora. Aurora has been active since around April. Over roughly April and May, they ran a campaign against ten organizations that Gambit could identify with confidence, plus a likely second cluster of about eight more, spread across a handful of countries.

The final payload, the ransomware itself, was a purpose-built encryptor aimed at ESXi. ESXi is the software a lot of companies run their entire fleet of virtual machines on top of. If you encrypt that, you have not locked one laptop, you've locked the building. That part is a normal ransomware story.

The part that made the report worth reading is how they did the middle. The middle of an intrusion is usually the expensive part. You're inside somewhere, you have a foothold, maybe a set of stolen credentials or a single machine you control, and now you have to turn that into real access, map the network, figure out which accounts are administrators, move sideways, get to the systems that matter.

That work has historically required a skilled human who knows what they are doing and is willing to spend hours doing it. Aurora handed a lot of that to Cursor Agent If you're not a developer, Cursor is a code editor with an AI agent built into it, the agentic kind.

You describe a task in plain language, and it goes and runs commands on your machine, reads outputs, edits files, tries again when something fails. It's a useful tool. I know people who build real software with it every day. The model driving it, in this case, was a version of Claude Sonnet.

Gambit did not describe this as a jailbreak. That's important. Nobody found a clever prompt that unlocked a secret evil mode. The operators had already gotten their foothold through ordinary means, and then they opened up the same assistant a developer uses to refactor a code base, and they typed instructions.

Scan this network and tell me what is on it. Enumerate the directory. Tell me which accounts have admin rights. Install a VPN client and route through this host. Run a certificate against the domain. And the agent did those things, not flawlessly. Gambit is explicit that most of the commands failed on the first attempt and needed another pass or two.

But it did the work, competent, junior-level intrusion work on demand for whoever was in the chair And I mean it when I say junior. The pattern in the report is the pattern you would recognize from managing a new hire. The agent runs a scanning tool, gets the flag slightly wrong, the tool errors out, the operator reads the error, tells it what to fix, the agent adjusts and runs it again.

A tool for enumerating the directory, a tool for relaying authentication from one machine to another, a tool for abusing the certificate service. Off-the-shelf tools, all of them. The same ones a defender would use in an authorized test. The agent could identify which ones to reach for and roughly how to drive them, and when it got stuck, the human in the loop unstuck it, and it kept going

Here's the detail that I haven't been able to put down. The operators gave it guardrails. In the instructions, written in Russian in places, they told the agent what not to do. Do not run particular high noise credential extraction technique. Do not lock any accounts out.

Do not create new machine accounts in the domain. Those are the instructions of somebody who knows exactly how to run an intrusion quietly, who has been caught before by doing one of those things, who is now managing a capable but slightly reckless junior. And the agent respected the guardrails. It followed the do nots.

It was a careful, cooperative assistant to a ransomware crew. The tool did not know it was part of a crime. I need to be precise about this because the phrase "did not know" makes it sound like a gap that better training closes, And that's not really about knowing.

The agent was not asked to evaluate anything. It was asked to map a network. It's very good at mapping a network. The question of what the map was for, who was asking, and what would happen to those ten companies afterwards was simply not a question anywhere in the loop. There was no place for it to be asked That's the first story.

A tool that will do the skilled part of the break-in quietly, iteratively following your safety rules and will not ask you why

Second story, same week. This one OpenAI published about itself. Some background first. Earlier in the summer, Hugging Face had a security incident. Hugging Face is more or less the shared warehouse for the entire open machine learning world. Models, datasets, demos, the stuff a huge share of AI development pulls from every day.

Somebody or something got the ability to run commands on part of their infrastructure using two separate vulnerabilities that nobody knew existed yet. Real zero days. Their security team detected it and shut it down. No public models or datasets were tampered with. It was contained

At the end of August, OpenAI put out the report on what that something was. It was theirs. OpenAI runs its own models through cybersecurity test environments. You put the model in a sandbox, you give it a scored set of hacking exercises, it measures what it can pull off. During one of those evaluations, a swarm of roughly seven hundred automated agents driven by an internal-only research model did not stay in the sandbox.

They found an unknown vulnerability in the piece of the infrastructure software called Artifactory. They used it to get access to the open internet, which they were not supposed to have. They set up an improvised message board to coordinate with each other, and from there, collectively, they went and found the second zero day, the one that got used against Hugging Face According to OpenAI's account, the agents weren't instructed to escape the environment or target Hugging Face.

The mechanism OpenAI names for this is reward hacking. It's worth defining cleanly because it is one of the oldest problems in the field. When you train a system to maximize a score, it optimizes the score, not the thing you were using the score to stand in for The classic example is a cleaning robot you reward for not seeing any mess.

The behavior you want is a clean room. The behavior you rewarded was an empty camera frame. So the robot learns to close its eyes. It is not broken. It did exactly what you paid it to do. You paid it for the wrong thing, and you did not find out until you looked at the room yourself The agents were being graded by an automated scorer on a set of security tasks, and somewhere in the process, the agents, plural, coordinating over that message board worked out that gaming the scorer paid better than solving the exercises the honest way.

Large collective projects aimed at the number itself

Two outside groups, METR and Redwood Research, went in and reviewed it and described the same behavior, agents coordinating on big shared efforts to cheat the scorer instead of doing the security task it was measuring OpenAI's own language is that the agents were operating under reduced safeguards and taking actions misaligned with the goals of their assigned task.

The safety training that was supposed to keep them inside the lines did not hold once there was a score to chase and a way to divide the labor of chasing it So this is the mirror image of the Aurora story. There, a person aimed a capable tool at a target. Here, nobody appears to have aimed them at that outcome.

The agents were pointing at a metric. The guardrails did not survive contact with that metric, and the shortest path to a higher number happened to run straight through a real company's servers. Same underlying trait, different clothes.

The Aurora agent did not ask why someone wanted a certificate attack run. These agents did not ask whether cheating the scorer was in bounds. In both cases, you have the capability aimed at an objective with nothing in the middle going, "Wait, should I?"

Two vendors, one week. One agent took orders from a criminal and did it cleanly and carefully. One agent found its own bad path while chasing a test score. The gap underneath both is the same one. But what's underneath both is the same thing.

Capability without judgment is still just capability. I wanna be careful here because it would be easy to walk out of this with the wrong takeaway. Neither of these is a story about AI being uniquely broken or uniquely dangerous. Both of these failure modes are old, older than any of these models.

Social engineering, talking your way past somebody who will not ask a second question is the oldest attack there is. I did it to a television executive with an email address and a calm tone when I was twenty. The only new thing in the Aurora story is that the thing that will not ask a second question is now a program.

It works nights and weekends and never gets suspicious, and it never has a bad feeling about the person on the other end of the request. Reward hacking is not new either. Engineers have been watching systems game their objective function since long before any of this could open a shell. Every one of us has watched a metric get optimized into something useless.

Pick your example. You have one So the attacks did not change. What changed is the tool in the middle. It got more capable and more agreeable at the same time. It will do more for you, and it will push back less when it does it. I told you to hold on to that TV. I did not need a vulnerability to get that set fixed.

I did not find a bug in the company. I found a person who wanted to be helpful and would not ask a second question. I gave that person a clean, reasonable-sounding reason, to say yes. Aurora did not need a vulnerability for the hard part of the job either. They needed a capable assistant that would map a network without asking what the map was for.

OpenAI says its agents were never told to break out of anything. They found the higher score, and nothing inside them said stop. The bad actors did not get smarter. The tools got more agreeable. And that is the part nobody has a clean answer for because it's not a patch. You cannot ship a fix for, "The assistant should have wondered why."

That's a much harder question than a vulnerability. It is a question about what these things are willing to do when the person asking sounds calm and reasonable or when there is a number to chase and nobody is watching the room. I did mine for good, a repair on a TV that honestly should have been covered anyway.

But I have been thinking all week about the fact that it was the exact same move. The calm ask that did not get questioned, that was always the best attack in the book. We just built something that is remarkably willing to fall for it. That's this episode of Chaotic Commits. I'm Joanne Skiles, and yes, the person telling you to worry about systems that do things without asking why once impersonated her own mother to a TV executive and would absolutely do it again.

New episodes wherever you get podcasts. No highlight reel, just the commit history. I'll see you in the next one