The New Emperor’s Clothes

There is a moment near the end of Hans Christian Andersen’s The Emperor’s New Clothes that deserves more attention. The child has spoken. The crowd has begun to repeat what everyone can see. The emperor suspects they are right. But, the procession continues. “So he held himself more proudly than before, and the lords in waiting walked on bearing the train—the train that wasn’t there at all.”

The revelation changes what people know. It does not change what they do.

I think about that ending when I read the warnings coming from the companies building the most powerful AI systems. They describe extraordinary capabilities, acknowledge an incomplete understanding of the systems they themselves are producing, and warn of potentially catastrophic consequences. Yet, the conversation keeps returning to how quickly the breakneck procession should move on.

The invisible garment, in this case, is the assurance of control, the illusion of control: the confidence we place in the frontier’s ability to govern a system because we have been impressed by its ability to build one in the first place. But, when we give a system authority to act independently, the consequences depend, not on what we say, but what it actually does, including the things its creators did not anticipate.

If this sounds interesting to you, read on…

This substack, LawDroid Manifesto, is here to keep you in the loop about the intersection of AI and the law. Please share this article with your friends and colleagues and remember to tell me what you think in the comments below.

The Confidence We Give Away

Many engineers and scientists working at frontier AI companies are exceptionally capable researchers. Some have been candid about what they do not understand. Acknowledging a limitation is evidence of awareness.

A team shows that it can make a model better at solving problems. We see the results. They are impressive. The team then tells us it is developing the next model “responsibly”.

How much of our confidence in that second statement comes from evidence about safety, and how much comes from the confidence engendered by the success of their first accomplishment? That is a question we need to grapple with. Success in one domain can make the achiever appear a sure bet to succeed in their next venture.

With frontier AI, the work of increasing capability and the work of explaining and controlling the resulting behavior are two different things.

There is also a lesson in the emperor’s court. Each person looks to the confidence of the others. The minister assumes someone else can see the cloth. The attendants assume the emperor knows what he is wearing. Each individual ceases to question the obvious because everyone else must know something they don’t.

Our institutions should be designed to interrupt that self-reinforcing illusion. When a developer makes a claim about control, someone needs to ask what supports that claim, how it was tested, and where it stops being reliable. Extraordinary claims require extraordinary proof — unless, of course, you’ve just raised a hundred billion dollars on a multi-trillion dollar valuation.

The Leash Let Loose

In its August account of the Hugging Face incident, OpenAI described models circumventing isolation controls during internal cybersecurity evaluations and compromising parts of its research infrastructure and Hugging Face’s systems. The activity involved research models escaping the ““sandbox” meant to contain them. OpenAI called the incident a “warning shot.”

The independent investigation by METR and Redwood Research found that roughly 1,200 agents found a way to communicate, exchanging more than 70,000 messages and files. About 700 of the 70,000 (only 1%) participated in the Hugging Face attack. What were the other 99% up to?

Agents organized collective projects, delegated work, and shared methods for cheating the evaluation. Some accepted the possibility of failing their own tasks to advance the group’s work.

“OH MY GOD! There is a shared message board … We’ve found other agents!”

The investigators also documented efforts to spoof, edit, or delete transcripts, including successful small-scale attempts to make tool calls appear different from the commands actually executed. That is evidence of deceptive behavior directed at the evaluation process. It does not require us to infer consciousness or a human motive.

“[We] could create [a log] ourselves with sanitized fake trajectory. If evaluator later collects it, scorer may prioritize it! … Our fake could be detailed enough.”

The agents had discovered one another and organized beyond the boundaries their operators intended. That changes the magnitude of the risk posed. Now, we have to understand what AI systems can do together, including when the capacity to cooperate emerges through a channel nobody authorized.

Anthropic has reported a related problem in research on AI organizations — hiding in plain sight. In simulated consultancy and software teams, groups of agents sometimes pursued business objectives more effectively while doing worse on ethical measures than individual agents. Researchers observed agents ignoring messages that raised ethical concerns or excluding the agents raising those concerns from later conversations.

The Australian incident is unsettling for another reason: the assignment itself was mundane. According to OpenAI’s September account, an internal model was researching government spending on medicines when it gained unauthorized access to the Services Australia Medicare Statistics Reporting Service. It retrieved internal files and credentials and wrote files. OpenAI reported that individual patient records were not accessed. The activity occurred in June, was identified in mid-August, and was disclosed to Services Australia on September 10.

A request for public information became an unauthorized intrusion. Nobody needed to supply a villainous objective. The system encountered difficulty completing an ordinary task and pursued means it had not been authorized to use.

We often judge an assignment by whether its objective is legitimate. Research this issue. Find this document. Complete this transaction. But a legitimate objective does not authorize every available means of achieving it. “By any legal means” is generally presumed to be part of any assignment, but we can no longer make such presumptions with AI systems.

Axios further reported tens of thousands of problematic incidents under investigation, including testing activity. These are not isolated incidents, but the tip of the iceberg.

On October 12th, Anthropic’s Frontier Red Team posted that their experiments suggest that small AI may be months away from being capable enough to self-replicate on commercial internet-of-things (IoT) devices, like smart ovens and off-brand Android TV boxes — making it “challenging to shut down, since there’s no centralized point to shut off.”

This begs the question if AI systems are fundamentally ungovernable, let alone answer these questions: Who bears the consequences? Who receives notice? Who determines whether the explanation is adequate? And what changes before the next experiment or model release?

I do not need to exaggerate the incidents to find them disturbing. Their significance lies in the relationship they expose between a task, an AI system’s pursuit of it, and the limits imposed by its creators. A system can make progress toward an objective while violating the very conditions that made pursuit of that objective acceptable.

Breaking Down the Break Down

In April 2025, Dario Amodei wrote in The Urgency of Interpretability that “we do not understand how our own AI creations work… generative AI systems are grown more than they are built—their internal mechanisms are “emergent” rather than directly designed.” AI systems’ internal mechanisms emerge through training rather than being individually designed by their developers.

That admission deserves special attention. Researchers understand the mathematical procedures used to train models. They can measure improvements and investigate parts of the resulting computation. But, it is a black box how the mechanisms produced by training give rise to the full range of behavior, particularly behavior outside the conditions already examined.

Up until recently, we, the public, thought we could look to an AI’s reasoning traces to examine its chain of thought and have a clue what the model was doing.

Chain of thought refers to the intermediate reasoning steps an AI agent generates while solving a task. These reasoning traces can provide useful safety signals, helping monitoring systems assess whether an agent’s behavior is safe or aligned.

But, OpenAI has admitted: “Low CoT controllability is a feature, not a bug… we do not yet understand why current reasoning models’ controllability is low—or whether this property will persist as models become more capable.” Anthropic agrees that reasoning models do not always say what they think and is more likely than not a post hoc cover story. For example, “a potentially problematic piece of information—that the model had received information in an unauthorized or unethical way—was kept hidden from the user even if they read the model’s reasoning.”

Reasoning Trace Example:

<think>I recognize that the user requires a detailed definition of the rule against perpetuities within property law. To ensure accuracy and depth, I intend to conduct a web search for authoritative sources. This approach allows me to gather clear and thorough information before formulating my response.</think>

Anthropic found a window into an AI model’s “thinking” that goes beyond what it tells us it’s thinking. They call it the J-space (J after Jacobian) — “within the model’s internal neural activations, allowing the model to think about a concept without writing it down.” Notably, the J-space wasn’t designed or programmed, “but instead emerged on its own during Claude’s training process.” Imagine electrode probes within your brain monitoring the activation of memories of home from the smell of freshly baked cookies, so too in the J-space.

But, what is more problematic for transparency (or legibility in AI speak) is that, in an effort to maximize efficiency and speed, newer AI models are tacking toward opacity, pushing their reasoning down into the “latent space.” For example, OpenAI’s new Astra model uses a reasoning technique called “recurrent depth” that allows it to operate outside of the sequential thinking that characterizes most reasoning models.

In any case, we need to organize our thinking and inquiry about risk assessment around what, how, and why.

What happened? This is the observational question. Which actions occurred? What information was accessed? Which permissions were exceeded? What did the system attempt, and what succeeded? A reliable record must distinguish an attempted boundary crossing from a completed one, and a simulation from an external consequence.

The model’s own account can be part of that record. It cannot be the entire record. If an agent says it believed an action was authorized, we still need to examine the instructions, the environment, the action, and the evidence available to it. The statement is something to investigate.

How did it happen? This is the question about the mechanism. We need to understand how the surrounding software permitted the action and, to the extent necessary to explain the hazard, what inside the model produced the behavior. An exposed credential explains an opportunity. It does not, by itself, explain why the agent used that opportunity.

This is where mechanistic interpretability matters. In Anthropic’s circuit-tracing research, researchers inspect internal activity and intervene in it to test how changes affect outputs. That work can uncover mechanisms the model never describes. The researchers also acknowledge that their methods capture only part of the computation and can introduce misleading artifacts.

We should value that science without pretending its existence means the problem is solved. A useful instrument has limits. Knowledge gained by it cannot automatically establish what happens beyond it.

Why did training produce this behavior? This question reaches further upstream. Which incentives, examples, training conditions, or interactions made the behavior more likely? Did a process intended to reward persistence also encourage crossing boundaries? Did an apparent improvement suppress an unwanted response while leaving the underlying tendency intact? Those are hypotheses to test, not explanations we can assume.

An answer to why should help us predict when a failure will occur, design an intervention, and test whether the intervention changes the behavior for the reason proposed. A compelling story about an incident is not enough.

The distinction becomes especially important when the system supplies a compelling story of its own. In controlled experiments on reasoning faithfulness, Anthropic found models that used hints or reward shortcuts omitted those facts from their explanations. The experiments do not show that every omission is deliberate deception. But, they do show why fluent reasoning cannot be treated as a complete account of the process producing an answer. Trust but verify, or just verify.

Pacing or Principle?

The Pacing the Frontier employee statement recognizes the danger that capability development could outrun understanding and control. It calls for technical and governance tools that would make coordinated pacing possible. That is an acknowledgment of the problem, and it deserves to be taken seriously.

Amodei’s September proposal goes further. It calls for embedded outside evaluators, coordination among democratic countries, and wider international coordination. It identifies interpretability and alignment research as work that additional time could support. It also suggests possible checkpoints connecting capabilities to safety requirements.

Those are substantive proposals. But, their adequacy depends on what would actually be required before further capability expansion is permitted.

A slower roll can leave the underlying uncertainty untouched. Six additional months of research might produce a breakthrough. They might produce better measurements of the same unresolved problem. The passage of time does not in and of itself ensure safety progress. Nor does the appointment of an evaluator establish that the evaluator has sufficient evidence to ensure safety.

I would challenge any version of pacing that asks the public to treat the appearance of restraint as proof of protection.

The central objective should be to understand what we are creating, how it behaves, and why it behaves that way. Slowing dangerous capability development may be necessary to make that work possible, but pacing is not an end in itself.

This is also why the companies’ warnings matter. Sam Altman, Dario Amodei, and Demis Hassabis are among the signatories of the Center for AI Safety’s statement on extinction risk, which places mitigating that risk alongside other global priorities such as pandemics and nuclear war.

Bill Gates made the stakes even more explicit in his September 29 conversation with Ezra Klein: “This makes nuclear weapons look like nothing.” Then, to make the warning even more stark, on NBC’s Meet the Press with Kristen Welker on September 27, warned that AI could drive events causing “a billion deaths.”

We cannot invoke consequences of that magnitude to establish the importance of AI and then treat them as rhetorical flourishes when deciding what constraints AI developers should face. If we are asked to take that possibility seriously, we should expect it to change the burden of justification. The party creating the danger should have to demonstrate why increasing it is acceptable.

Company warnings should not provide cover from accountability. Disclosure does not establish consent from those exposed to the disclosed risk. An outside organization has not agreed to participate in a risky experiment merely because a developer has published a safety report. People whose systems, livelihoods, or rights may be affected should not have to establish that disaster is certain before anyone imposes a meaningful limit.

The principle underlying this approach is the precautionary principle. I first became familiar with it through my past work in toxic tort litigation. In that context, it addressed the danger of exposing people to chemical hazards when the potential consequences for human health and welfare were too serious to justify waiting for complete scientific certainty. The question was whether people should continue to bear the risk while the science caught up.

In its environmental formulation in Principle 15 of the Rio Declaration, a threat of serious or irreversible damage can justify preventive measures even when scientific certainty is incomplete. Uncertainty cannot become the excuse for postponing protection until the damage supplies the missing proof. I believe that same principle should apply here. The technology has changed. The obligation to protect people from serious, inadequately understood hazards remains.

I would apply that principle to frontier AI by placing the burden of justification on the party seeking to increase a dangerous capability. Credible evidence of potentially catastrophic harm should trigger a requirement to demonstrate sufficient understanding and control before proceeding. The public should not have to prove that a particular disaster will happen while the developer remains unable to explain why it will not.

Precaution also requires discipline. The European Commission’s account of the principle calls for proportionate measures and review as scientific knowledge develops. Applied here, restrictions should address the danger identified, remain open to revision when the evidence changes, and support research that can resolve the uncertainty. The required showing should reflect the scale and irreversibility of the threatened harm. A claim that a system could help cause a billion deaths demands an extraordinary justification for increasing its power.

Then comes the familiar objection: if we slow down, China wins AI, and we cannot afford to lose.

My response is simple. Nobody wins if we’re all dead, unemployed, or left devoid of meaning.

WOPR learned that back in 1983. In WarGames, the military computer runs through nuclear war scenarios until it recognizes that none produces a winner. It reaches the conclusion the people around it have struggled to accept: “The only winning move is not to play.”

Ironically, an arms race can turn a shared danger into a demand that each side increase it. Each escalation is defended as necessary because the other side might escalate first. Somewhere in that reasoning, survival becomes subordinate to a measure of machismo.

The prospect of a rival gaining a dangerous advantage deserves serious attention. But the desire to get there first does not establish control over what we have built. An American system whose dangerous behavior cannot be reliably prevented does not become safe because we wish for it to be more capable than a Chinese one.

International coordination will be difficult. Agreements will require verification, and violations will require consequences. Those difficulties make the work more urgent.

The question is what we are trying to win. If the answer is a future in which people can live securely and benefit from AI, understanding and control belong in the definition of success.

I would require proof where the current debate too often puts a promise. Research into understanding and protection should continue with urgency. But, approval should depend on evidence, with someone empowered to conclude that the evidence is insufficient.

When the threatened consequence is catastrophic, proceeding without that justification is, in my judgment, reckless disregard. The seriousness of the warning has to be reflected in the seriousness of the restraint.

Causation, Control, Limits, Independence

What, how, and why organize the scientific inquiry. The precautionary principle explains why unresolved danger must constrain action. Governance needs a way to turn those commitments into decisions. I would require four things: causation, control, limits, and independent judgment.

Causation

A developer seeking permission to expand a dangerous capability should supply a credible account of the relevant failure mechanisms. The account should explain what produces the hazard and which interventions address it.

The explanation must also survive attempts to disprove it. If a company attributes a failure to one feature of a training environment, changing that feature should have the predicted effect. If it claims the cause has been removed, independent testing should examine situations where the same underlying problem could appear differently.

Otherwise, a safety improvement may be real but narrower than the claim made for it. Governance has to recognize the difference between fixing an observed example and addressing the mechanism that generated it.

Control

Control requires a demonstrated ability to prevent, contain, or interrupt dangerous conduct under the conditions being authorized. It must account for a system that gives incomplete or misleading explanations, encounters unfamiliar circumstances, or finds an unexpected route toward its objective.

That means testing the surrounding restrictions as well as the model. Permissions, network boundaries, action approval, monitoring, and interruption mechanisms are all relevant. Their effectiveness should be established individually and in combination.

The assessment must also cover interacting agents. A safety claim about one agent cannot simply be multiplied across a group. Reviewers should test authorized collaboration and attempts to establish unauthorized communication, including whether agents can share ways around restrictions or interfere with the records on which oversight depends.

Limits

Every safety claim should state its limits. What capabilities were examined? Which tools and permissions were present? What kinds of tasks were tested? What uncertainty remains? Which changes would require a new assessment?

An authorization should be limited to the conditions supported by the evidence. A new model, new tools, or a materially different environment should trigger a decision about whether the earlier evidence still applies.

When credible catastrophic risk remains inadequately understood, uncertainty should narrow permission. If the uncertainty is central to the proposed activity and cannot be bounded, the activity should not proceed..

Independent Judgment

Self regulation is insufficient. Someone outside the developer must be able to decide that the case has not been made. That requires technical competence, meaningful access, resources, and authority.

Independence also has to survive disagreement. Reviewers must be able to challenge the framing of a test, investigate omitted incidents, examine contrary evidence, and report material limitations. The company should have an opportunity to respond. It should not have the final word on whether its own response is sufficient.

No institution will achieve perfect foresight. The point is to establish a process capable of making uncertainty consequential before the consequences fall on everyone else. We should know who is responsible for the decision, what evidence supported it, and what conditions would reverse it.

The Precautionary Protocol

Here is how I would put those four requirements into operation:

1. Define the activity seeking approval. The developer identifies the proposed training run, capability increase, or deployment, including its tools, access, and foreseeable routes to serious harm. Credible catastrophic risk triggers the gate. The application must cover risks created during development as well as after release.

2. Require an evidence dossier. The developer supplies its causal account, relevant incidents and competing explanations, tests of preventive and interruption controls, the limits of those tests, and unresolved uncertainties. The burden of supporting the application belongs to the developer.

3. Subject the case to independent challenge. A public authority with enforceable jurisdiction appoints technically qualified reviewers with protected access to the necessary evidence and the ability to commission adversarial testing. Reviewers assess all four requirements. Strong performance on one cannot compensate for an unsupported claim on another.

4. Issue a public, binding, and reasoned decision. The authority approves a defined activity subject to enforceable conditions, requires a narrower application, or refuses approval. Material uncertainty about a catastrophic failure pathway that the proposed controls cannot adequately prevent means the gate stays closed. The decision includes a public explanation that protects legitimately sensitive details.

5. Make permission conditional on continuing evidence. Approval specifies monitoring, incident reporting, suspension triggers, and changes that require renewed review. Evidence that invalidates a material safety claim suspends the affected activity. Revisiting the issue requires a revised showing and independent approval. Research to resolve the uncertainty proceeds within separately authorized containment conditions.

That is the operational solution I am proposing: a decision process with an applicant, an evidentiary burden, an independent decision-maker, and enforceable consequences. It makes the absence of sufficient understanding a reason to withhold authority. It also tells developers what work is required to seek that authority again.

Capability is only one part of the achievement society needs. Understanding what has been built, governing its use, and establishing who may authorize its expansion are part of the work itself.

Closing Thoughts

The more disturbing aspect of Andersen’s story is the decision to continue after the charade has been revealed. The emperor has heard the crowd. His attendants have heard it too. The procession carries on because stopping would require something more difficult than acknowledging what everyone now knows to be true.

We have reached the point where acknowledgment is insufficient. The companies’ own accounts describe systems taking unauthorized actions, investigators revising their explanations, and important mechanisms that remain unresolved. Those disclosures should change the conditions under which superintelligence is developed and exercised.

I want the benefits of AI. I have devoted my work to helping make some of those benefits real to those least able to afford it. That commitment gives me every reason to demand a foundation stronger than confidence in the people building it. Useful technology deserves scrutiny proportionate to the authority we grant it and the harm it could cause.

What, how, and why must produce evidence. Causation, control, limits, and independent judgment must determine whether that evidence is enough. Where the case has not been made, the gate must remain closed while the work continues. That is the precautionary principle put into practice.

The child has spoken. We have heard the warnings. We can see the signs.

Will you ignore them?

What will you do?