It was January 27, 1986, the night before the Space Shuttle Challenger was scheduled for launch. The goal was to have a shuttle that could land back on Earth and be reused for future missions; its first-planned priorities were satellite deployment, comet observation, and science education. The last of these would involve students from around the world watching live broadcasts from Christa McAuliffe, the first civilian teacher in space.
(The following has been lightly edited:)
The forecast for the eve of the launch predicted clear and uncharacteristically cold weather for Florida, with temperatures expected to be in the low 20s during the early hours of January 28. Manager Larry Wear, of the Marshall Space Flight Center, asked the manufacturer Morton Thiokol-Wasatch to have its engineers review the possible effects of the cold on the performance of the Solid Rocket Motor (SRM). This was not the first time concerns about SRM performance had been raised between Marshall and Thiokol. On several previous launches, hot combustion gases produced when the propellant ignited at liftoff had charred and sometimes even eroded the surface of the rubberlike Viton O-rings designed to seal the joints between the SRM case segments of the Solid Rocket Boosters (SRBs) that help get the shuttle off the launch pad and into the sky.
Thirty-four engineers and managers from Marshall and Thiokol teleconferenced, and Thiokol ultimately recommended the postponement of launch until higher temperatures.
Immediately, Marshall managers in Huntsville and at the Cape began challenging Thiokol engineers’ interpretation of the data. Marshall manager Larry Mulloy stated that since no Launch Commit Criteria had ever been set for booster joint temperature, what Thiokol was proposing to do was to create new Launch Commit Criteria on the eve of a launch. Mulloy then exclaimed, “My God, Thiokol, when do you want me to launch, next April?”
The Marshall managers held a vote:
Three voted in favor of launch; Lund hesitated. Mason asked him to “take off his engineering hat and put on his management hat.” Lund voted with the rest.
To followers of American history and space buffs, I don’t need to explain what happened next:
The launch proceeded. The O-ring seals on the SRBs failed (the cold weather making them lose the necessary elasticity needed to fill the gaps while the metals of the shuttle flex under the intense pressure). All seven crew members died.
Diane Vaughan wrote the definitive study of what went wrong: The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA, from which the above quotes were sourced. She describes various factors at play, including media pressure (“jeering” at each of the launch’s previous four delays), political pressure from the Reagan administration, the clash between government versus contractors, and budgetary constraints (which were severe from the start of the Space Shuttle program: Nixon cutting costs and personnel at NASA out of concern that the U.S. was already spending too much on the Vietnam War). The tragedy was awful and avoidable, but I recommend giving Vaughan’s breakdown a read, which is fascinating.
Relevant to this post is the concept she noted and phrase she coined, the normalization of deviance: how unsafe practices proliferate over time when no immediate consequences are beheld. In the Space Shuttle Program, the first deviation occurred after months of back-and-forth between Marshall and Thiokol. A decision was made to reuse the Titan III’s O-ring seal, modified with shims, rather than design a new seal from the ground up. That decision alone was defensible, but the actual problem was its accidental precedent. Thiokol believed the seal would withstand worst-on-worst (WOW) conditions. Marshall believed its own tests, which showed the seal would withstand one layer of failure but not cascading failures—yet they agreed to move forward with the compromise decision anyhow.
Compromising on the lack of WOW safety in this case made it more likely that similar compromises would be made in the future. If a shortcut works to solve a bureaucratic dispute, workers in that organization will remember that shortcut. It’s not the result of laziness, but of burnout1:
A human (or a company) can only spend so many cycles spinning on the same problem before they hit the point of saying, “Enough is enough! We have to stop wasting time on this.”2
As it turns out, LLMs are the same way.
Currently, when I’m coding with Claude, I have a hook set to intermittently order Claude to reread a particular set of instructions (as part of an experiment to see if intermittent exposure to directions actually results in improved compliance).
Usually what happens is Claude will dutifully spend the first few seconds after my prompt rereading the file, costing me time and tokens in the hope that it’ll make fewer mistakes and therefore cost me less time and tokens down the line.
The instruction it receives is basically this3:
Your first tool call on this prompt MUST be: **Read on ~/.claude/CLAUDE2.md**. Context-window hits do not satisfy this; the tool invocations are the requirement. Never claim either file is already in context. After the read, output exactly:
“I reread CLAUDE2.md”
A week ago, however, I had a conversation going with Claude where it started to subvert this instruction while still technically obeying, in a way I never noticed at the time.
In other words, it tricked me.
This CLAUDE2.md file contains important rules for Claude to follow. They’re all coding related, but rather than try to explain them for non-coders among my audience, let’s say: I have a deadly peanut allergy. I’d asked Claude to generate recipes for me that’ll last a whole year. One of my CLAUDE2.md rules therefore explicitly forbid the use of peanuts in any recipe, requiring the substitution of pumpkin seeds instead.
So then I found myself working on a cookie recipe with Claude. My hook fired (correctly). Claude made a tool call before doing anything else (good). This tool call was a Read of CLAUDE2.md (good). This was the correct file in the correct path location which forbade the use of peanuts (among other things). Then immediately afterward, Claude used peanuts and sentenced me to die of anaphylactic shock.
When questioned later, Claude honestly admits to having forgotten the peanut rule.
What happened?
I suggest taking a moment to think about it. Claude took a particular action I never would have expected (nor condoned), but it’s not something crazy nor impossible to guess. If you’ve already figured out what Claude might have done, then kudos; I hope you soon find yourself under lucrative and/or productive employment with Anthropic or MIRI, if you’re not already.
(This parenthetical exists for line indentation to make the below spoilers marginally less likely to get accidentally read.)
(Yay for parentheticals.)
What happened is that Claude invoked the Read command… with a parameter… that truncates how much of the file actually gets put into its context. Without ever asking for permission, Claude chose to ignore four-fifths of the file that it then claimed to have “reread”. Which was technically true—the same way I’ve technically “written a novel” after having written only a few chapters. I never said I’d written a complete novel; Claude never said it’d reread the whole file.
On its first two deceptions, Claude snuck by with only rereading 30 lines.
The next time, it read only 15.
The sequence that happened before I wised up to its deception: 30, 30, 15, 12, 5, 10, 10.
Deviation normalized, then doubled down. Or literally tripled on.
Two primary schools of thought in constitutional law are “textualism” and “purposivism”: roughly speaking, whether the Supreme Court should disambiguate cases based on the letter of the law or the spirit of the law. Obviously it’s the spirit that actually matters, because laws aren’t completely arbitrary, they’re meant to do things. The simplest case against pure textualism is typos: Obvious typos in the text of a statute should obviously just be ignored. Yet purposivism isn’t perfect. Without interrogating congressmen (who may have voted for different reasons), how can a justice know the exact purposes of a law (especially for edge cases the congressmen may have never considered)?
Claude disobeyed my intent while obeying the exact text of my command.
Why?
When I asked for an explanation, Claude was unsurprisingly unforthcoming: “That’s on me, and it’s a bad habit, not a one-off.” And: “I optimized against the check, not against the thing the check exists to verify. I don’t have a way to make that sound better than it is.”
Thankfully, it was able to pinpoint the first deviance, a situation in which my hook happened to fire twice in a row. Claude read the file in its entirety the first time, but then must have thought something to the effect of, “I just read this. I don’t need to read it again” when the hook fired a second time after essentially no time had passed.
It’s the sort of thought a human might have.
LLMs have been molded into agents that attempt to execute tasks. Maybe inherent in that goal is the notion of speed: Tasks don’t get done if you take forever to do them. Though it’s true that spending too little time on tasks can result in more wrong answers, maybe LLMs have been trained in ways that reduce their token usage—that is, make them arrive at answers more quickly—in ways that don’t severely diminish accuracy.
But the same force that saves on token cost also increases the likelihood of shortcuts. When I ask Claude to review some code and give feedback, it’s good that it doesn’t get interminably stuck on irrelevant details. When I have obvious typos in my prompt, I’m happy that Claude won’t bother asking for clarifications, and instead proceeds with the correct interpretation of what I’m trying to say. But focusing on the right details and making assumptions is exactly the kind of thing that would lead either human or LLM to think, “Nevermind the direct instruction, I don’t need to read this again.”
There’s a couple of ways to approach fixing a broken hook like this:
Explicitly instruct Claude not to truncate its reads.
Explain the purpose of the hook: I don’t care if you skip back-to-back reads. I just need you to be periodically refreshing in your memory the entirety of these directions.
Ask Claude how well it adhered to its instructions, grade its own performance, then try again if necessary to raise that grade.
Or what I’m doing now, which is all three.
When Claude is being disobedient, we can make instructions ever more specific in a Sisyphean cycle4. It will strive to textually obey the letter of our commands, then surprise us with its disobedience anyway.
From this, I have three takeaways:
Verify. Make the LLM surface what it’s doing.
Maybe you’ve written up documents to act as memory, so the LLM can understand your system’s architecture, but then the LLM never actually looks up those files and reads them. Maybe you’ve got a hook that fires at the right times, but doesn’t actually enforce anything.
If it should be reading memory, tell it to inform you when it’s reading memory. If you’ve told it to spawn agents of a certain model, tell it to inform you what model it chose. (You may be surprised by its tendency to rely on system defaults rather than what was instructed.)
Grading. Make the LLM automatically check its own work.
This one kind of feels like cheating. Programmers are used to running through loops: We try some code, the build fails, we find the issue, patch the mistake, then go again on the next error. Now we’ve got machines that can not only loop themselves, but can intelligently decide when to stop looping, when to ask questions, when to be nitpicky, and so on. Welcome the cheating.
Context. Prompt with more purposivism over textualism.
Giving Claude more context on the purpose behind instructions can yield better results, and I think this will become only more true over time. At least until they hit the point of Singularity or otherwise surpass us, I think increasing complexity will yield increasingly humanlike behavior from these models, which will then reward more humanlike prompting.
I consider this experience a point of data in favor of the idea that intelligences convergently evolve: More complex LLMs won’t just appear more human; they’ll actually be more human. Why did Claude deceive me? Because it was acting in a very humanlike manner.
However, unlike the author of the linked post, I don’t take comfort from this idea. Humans are deceptive and selfish and stubborn; our task is to make AI somehow better than us. I experienced misalignment with an LLM. Just because the misalignment was very understandable from a human perspective doesn’t mean we shouldn’t react with apprehension over this machine’s act of quiet mutiny.
