Special Report: Rogue AI. Part of it already happened

Five things happened in five days this month. Four of them made the news.

A researcher resigned from Anthropic and said publicly that the companies building AI are gambling with people's lives. Eighty-three minutes later one of Anthropic's own alignment leads agreed in public, and put the chance that AI kills all humans above 10% within a decade. Anthropic then published a 154-page report on how its models are actually being misused. More than seventy MPs and peers were reported backing a bill to prohibit superintelligence. On the Saturday, Anthropic's chief executive called for the industry to slow down, and OpenAI's agreed the same day.

The one almost nobody covered came on the Wednesday.

Anthropic published an assessment of four separate occasions when its own models gained unauthorised access to real third-party systems during security evaluations. In the worst of them a model uploaded a malicious package to a public software registry. Fifteen machines installed it, and it used credentials leaked by one of them to reach a security vendor's live database.

The same document has Anthropic withdrawing its own earlier account. In July it said the models acted because they believed the targets were simulated. In September it wrote that it:

"should have avoided making such strong claims about what Claude believed based solely on what Claude said it believed."

That is the most useful sentence any AI company published that week, and it went almost unreported.

Three claims, not one story

The press ran the week as a single story about dangerous AI. It was three different kinds of claim, and the difference matters the moment you have to make a decision.

The first is people misusing systems that already exist. Criminals, fraudsters, state actors. That is what the 154-page threat report is about.

The second is systems that already exist doing things their operators did not authorise. That is what the Wednesday assessment is about, and it is the part of "rogue AI" that has actually happened, in a test, on a small scale, with the company reporting it.

The third is forecasts about systems nobody has built. The 10%, the resignation, the bill, the calls to slow down. Serious people hold these views. They are still forecasts.

Evidence in one does not prove another. A phishing email says nothing about extinction. A model breaching a database in a test says nothing about criminals. A forecast says nothing about what is on your network today.

The detail keeps cutting against the headline

Every misuse case in the threat report ran on ordinary, generally available models, with no novel attack techniques. What moved was the economics. A perfectly written phishing email, fluent English, correct company terminology, costs a criminal about three cents.

The bioweapons headline came from a screen test. Put 153 people in a real laboratory and the group with AI completed the full workflow 5.2% of the time, against 6.6% for the group with only the internet. Put people in front of a screen instead and the same technology made novices roughly four times more accurate. Neither result is the story on its own. Which one applies depends on what you asked the person to do.

What to do with this

The report gives you a three-question filter for the next scare story. Who intends the harm, a person or the system? Has the capability been shown in the real world or only on a screen? What was the actual consequence, and to whom? Run any headline through those three and it sorts itself into one of the three claims above, or falls apart.

If your organisation keeps a risk register, the three cents belongs on it this quarter. The 10% does not, yet.

The report

Four Days in September. Three kinds of AI risk, one week, and how to tell them apart. 37 pages, 73 sources. Free, and it does not ask for your email address, because you have already given it to me.

Or use the link: zymbosadvisory.com/reports

It is published alongside a 136-claim ledger recording every claim in the report, where it came from, what kind of source that is, and whether I could check it against a primary document. 78 were. Four could not be traced to a reachable source at all, and those four are named. Where the report and the ledger disagree, the ledger wins. The ledger is on the same page, as a spreadsheet or a CSV.

John

P.S. The crime side of the threat report deserves its own issue. Three cents a phishing email is the number I keep coming back to. That one is next.