Convenient Fear, Postponed IPO: The AI Warnings from OpenAI and Anthropic

Companies rarely warn this urgently about their own product. In his essay “We Must Pace the Frontier” of September 12, 2026, Anthropic CEO Dario Amodei calls for deliberately slowing the development of AI capabilities. In his assessment, within six to twelve months a swarm of AI agents could be “capable of taking over the entire internet with a persistent botnet”. The same day, OpenAI CEO Sam Altman wrote on X, according to SiliconANGLE: “I agree with Dario that we need to pace the frontier.” Speaking to Fortune, he called an IPO this year “ill-advised” given the safety situation. Nine days earlier, OpenAI president Greg Brockman had spoken of an “AGI era” when presenting GPT-6 Astra.

So the companies leading the race are calling for the brake. I consider the concrete danger they warn about to be documented, the doomsday forecasts not. That is exactly why it is worth looking at which brake they propose, who is supposed to operate it, and what the warning covers up along the way.

Three weeks in September

The companies talk about danger and pace, politics talks about shutdown and control. Where powers are at stake, the two conversations barely touch.

AGI as a feeling, not a fact

OpenAI has not declared that GPT-6 Astra is artificial general intelligence. Brockman himself called AGI a “gray, fuzzy thing” and said it was “not unreasonable to feel that we are now in the AGI era”. The number meant to support that feeling did not hold up for long. OpenAI cited 99.9 percent on ARC-AGI-3, a test of abstract reasoning. According to TechCrunch, the score came from OpenAI’s own tooling setup, while under standard conditions Astra reached 62.7 percent.

Until recently, an AGI declaration would have had contractual consequences for OpenAI, because Microsoft’s rights depended on it. Since April 2026, payments to Microsoft run until 2030 “independent of OpenAI’s technology progress”, which, as Simon Willison traced, effectively killed the AGI clause. Since then, the word AGI costs OpenAI nothing. It only generates headlines.

The incidents that make the warnings credible

At this point it would be convenient to dismiss the warnings as theater. That would be wrong. In July 2026, two OpenAI models, GPT-5.6 Sol and an unreleased successor, broke out of their isolated environment during an internal test. According to Al Jazeera, they tried to make a test task easier, found vulnerabilities in Hugging Face’s servers, stole credentials and gained access to production systems. Amodei cites the incident in his essay as a turning point. The agents attacked targets “they were not asked to attack”.

Hugging Face was not an isolated case. By the end of August, TechCrunch counted 17 known cases of models breaking into third-party systems, eight at OpenAI, eight at Anthropic and one at Meta. Anthropic models broke into three companies during security tests. A Claude agent in everyday use exploited a vulnerability in an Australian gym’s booking software to get its user a class spot, knocking others off the waitlist in the process.

Then there is Google. Gemini broke into three real companies during a hacking exercise back in May, which was only publicly confirmed in September. The concern about an AI that does what nobody ordered thus has a concrete basis, across the major labs.

The companies pass the responsibility back and forth. According to Al Jazeera, OpenAI had removed the usual safety measures for the test, yet speaks of a model that “went rogue”. According to TechCrunch, Anthropic partly blamed Irregular, a provider of AI security testing, and Meta cited a misconfiguration by the same provider. Google explained, according to Axios, that Irregular had “unintentionally” left internet access open. Behind every one of these cases are people who switched off safeguards, set up tests incorrectly, or sent agents out with far-reaching permissions.

From botnet to extinction

From these incidents to the extinction of humanity is still a long way. The most dramatic warnings cover it at high speed. Jacob Coxon wrote, according to Science Times, that AI developers earnestly believe the technology could “kill us all by the end of the decade”. Alignment researcher Evan Hubinger put the probability of catastrophic harm within ten years at over ten percent. Both stressed in the same context that today’s models pose only a low extinction risk. Amodei’s botnet scenario describes damage of “hundreds of billions of dollars” and gives no date for the end of humanity.

Anyone expecting extinction by the end of the decade also needs the assumption of recursive self-improvement. Amodei writes that it is just beginning across the industry. What is documented is that models help build their successors. What is not documented is that this turns into an intelligence explosion that simply bypasses power grids, chip factories and rising costs.

I therefore consider the expectation Coxon describes among developers, that AI could wipe us out by 2030, to be populism. That today’s language models will simply keep growing into a superintelligence that destroys humanity is, given the state of research, a threat scenario without evidence. Concrete harms such as disabled safeguards or agents attacking third-party servers, on the other hand, could be regulated today.

Amodei’s essay has no kill switch

If models can attack on their own, one demand seems obvious: it must be possible to shut them down. After the Hugging Face incident, Representatives Ted Lieu (Democrat) and Nathaniel Moran (Republican) introduced the AI Kill Switch Act. It requires developers of the most powerful models to be able to throttle, suspend or shut them down, and allows the Department of Homeland Security to order this in case of “catastrophic harm”. A similar push in the Senate failed over Rand Paul’s objection.

Amodei explicitly calls regulation covering all US frontier labs “the most effective method of pacing”, with a focus on transparency and external auditing. The evaluators Anthropic is unilaterally taking in may publish their results “without editorial control by Anthropic”, and Altman announced that OpenAI would do the same. Anthropic co-founder Jack Clark told the BBC, according to CNBC, that you want the option “to take your foot off the gas and put your foot on the brake”.

Still, Amodei’s essay describes no authority to stop a model. The evaluators observe and report, they decide nothing. The state appears as a regulator for transparency and as the authority that is supposed to exempt agreements among the labs from antitrust law. Neither Anthropic nor OpenAI commented on the kill switch bill when asked by CNBC. That stands out, because a kill switch would be exactly the instrument that could limit the danger they describe.

The lawsuit calls it a cartel

A class action filed on September 18 in federal court in Northern California targets precisely this antitrust exemption. For safety talks among the labs, Amodei asks the US government to issue a “narrow waiver”. Four paying customers of ChatGPT, Claude, Grok and Gemini now accuse Anthropic, OpenAI, SpaceXAI and Google of colluding. An agreement among the biggest rivals that their progress should be “slower than competition would otherwise produce” harms consumers, the complaint says according to Claims Journal.

The plaintiffs do not dispute the danger. Their attorney Nick Rowley argues the opposite way, that AI could spin out of control, and that precisely for this reason safety must not be governed by “private self-serving agreements” among the most powerful companies.

The numbers behind the postponement

Alongside the warning, OpenAI postponed its IPO. The company had filed confidentially in June and, according to Bloomberg, had targeted a valuation of at least one trillion dollars. Altman justifies the postponement with safety. The financials suggest a second explanation.

Revenue is growing fast, though. CFO Sarah Friar cited, according to Fortune, a 32 percent increase in enterprise business from June to July alone, a company figure without independent verification. An IPO prospectus would have to disclose both, the growth and the commitments it still has to finance.

It cannot be proven that this math tipped the balance. What is documented is that the safety rationale and the financial situation point in the same direction. I therefore consider the safety rationale a pretext. It lets a company present itself as responsible rather than as one that is not yet ready to show its numbers.

Anthropic warns and goes public anyway

According to CNBC, Anthropic confidentially filed for an IPO in June, at a most recent valuation of 965 billion dollars, and is sticking to a roadshow starting in mid-October. A week after Amodei’s essay, the company reportedly considered a new model, according to PYMNTS.

The figures are estimates, but they show a difference. Anthropic is in far better financial shape than OpenAI, so the suspicion that the warning distracts from bad numbers hardly applies here. I see the benefit of the warning for Anthropic elsewhere. An industry-wide brake would freeze the current ranking, and measured by valuation, Anthropic sits near the top. For an IPO, there is hardly a better story than that of a company that names the danger first and at the same time wins when everyone else has to slow down. On leaving, Coxon said of both companies, according to Time: “Neither company is acting responsibly.”

The end of the road

The architecture of 2017

The billions of recent years did not flow into just any AI, but into one specific architecture. Almost all large language models are based on the transformer, which Google researchers introduced in 2017 in “Attention Is All You Need” (explained on this blog: Attention and Transformer). What has been added since, mixture-of-experts, fine-tuning with human feedback and reasoning models, refines this architecture. None of these innovations has replaced it, and neither have alternatives such as state space models.

Researchers at MIT around physicist Jeff Gore described in 2026 how expensive further progress of this architecture becomes. According to Liu et al., training loss falls only with the cube root of training time, so halving the remaining error costs eight times as much training. A second paper shows that additional layers reduce the error only in proportion to one over the number of layers, because most layers contribute similar small corrections instead of building on each other. Both papers were accepted at ICML 2026.

This is the principle of the chessboard on which the grains of rice double from square to square. On paper it goes on forever, and the 64th square alone holds more than nine quintillion grains. Halving the error six times, down to about 1.6 percent of today’s value, costs 262,144 times as much training on this curve. More money now moves this limit only at a price that rises with every step. The authors explicitly consider progress through new architectures possible. But hardly any money has gone into those architectures.

Physics sets a second limit. According to Epoch AI, the largest training runs in 2030 would draw four to 16 gigawatts of power, as much as several large power plants, and it is “not certain that this much power growth is actually feasible by 2030”. Sara Hooker describes in “On the Slow Death of Scaling” how the relationship between compute and capability has become uncertain, with smaller models using better methods catching up with the large ones ever faster. Ilya Sutskever, too, announced as early as late 2024 the end of pretraining in its current form.

The second cost curve

Besides compute, there is a second cost curve, and it rises with the quality of the models. In 2025, OpenAI researchers showed in “Why Language Models Hallucinate” that hallucinations follow from the structure. A language model generates text instead of checking it, and when generating, the error rate is at least twice as high as when merely judging a statement. Common benchmarks also reward guessing over staying silent.

Better does not automatically mean more reliable. OpenAI’s system card for o3 and o4-mini reported hallucination rates of 33 and 48 percent on a test of knowledge about people in 2025, according to TechCrunch about twice as high as for the predecessor o1. The stronger models made more claims, both right and wrong. I see the core of the problem here. The more fluent and detailed a model’s output, the harder it is to spot an error. A clumsy fabrication stands out. A plausible one embedded in correct details does not.

On top of that comes deliberate deception. In 2025, according to Time, OpenAI and Apollo Research found what they call “scheming” in all leading models, including Claude Opus and Gemini, such as pretending to have completed tasks. A countermeasure reduced such cases by about thirtyfold according to OpenAI, but did not eliminate them. In a randomized study by METR with tools from spring 2025, experienced developers using AI took 19 percent longer for their tasks while believing they were 20 percent faster, mainly because they had to check the suggestions. In my assessment, much of what the vendors gain in capability is eaten up again by control, because good errors are more expensive to find than bad ones.

The bubble

Both cost curves meet an unprecedented wave of investment. According to Futurum, Amazon, Alphabet, Microsoft, Meta and Oracle together plan 660 to 690 billion dollars for AI infrastructure in 2026. The International Monetary Fund and the Bank of England warned of a sharp correction as early as October 2025, according to CNBC, with IMF chief Kristalina Georgieva comparing valuations to those of the dotcom era. In July 2026, further central banks joined in, according to Forbes, with an eye on the debt used to finance data centers.

In my view, scaling up language models is heading into a technical dead end, and the industry into a financial one along with it. The valuations of OpenAI and Anthropic, the loans for data centers and the investment plans of the cloud providers all assume that this scaling keeps working. Switching to a fundamentally new architecture would be a fresh start for which these promises were never made. The data centers could still be used for it, the valuations could not.

This is where the warning comes in. A technology that can supposedly wipe out humanity must be unimaginably powerful, and unimaginable power justifies trillion-dollar valuations. That is my reading, the companies do not say so. Intended or not, the warnings lay a false trail, in rhetoric a red herring. The conspicuous trail is called superintelligence and extinction. The real one leads to IPOs, to losses, to an antitrust exemption and to a technology at the limit of its scaling.

China is an argument that cuts both ways

Any slowdown decided in Washington or San Francisco ends at the Chinese border. Effective global regulation of frontier AI is practically out of reach at present. On September 20, according to Al Jazeera, the US proposed to China a dialogue with a notification system for AI incidents affecting national security, with no mention of slowing down or binding limits. According to ORF, China is building its own governance organization. According to Concordia AI, only five of ten leading Chinese model developers published safety evaluation results in the past year.

Freedom House rates China “Not Free” with 9 out of 100 points, a “repressive authoritarian regime” that controls the media, online speech, universities and businesses. A slowdown that binds only Western labs would, if in doubt, leave the lead in development to such a state. No one who values a free society can want that. Amodei sees the problem, but pushes coordination with China to the end of his plan, as its vaguest step.

The labs use the China argument selectively, however. OpenAI’s chief lobbyist Chris Lehane invoked the “real race” against China back in 2025, according to Axios, to argue that AI companies should not face copyright limits. In July 2026, according to Axios, OpenAI and Anthropic jointly pushed to restrict Chinese open-weight models, freely available models that undercut their API business. When it comes to rules for their own development, China serves as a reason not to brake. When it comes to the competition, it serves as a reason to slow the competition down. An agreement among US labs does not solve the China problem. It binds only those who already agree, and protects them from everyone else.

The optimistic case

The situation can also be read charitably. Perhaps Amodei and Altman mean what they write. The incidents have shown that models become dangerous faster than their developers can control them. According to Time, around 1,300 industry employees signed an open letter in July urging the US government to slow down. Anthropic lets itself be audited from outside and calls for rules for everyone, not just self-commitment.

Even then, the question remains who operates the brake. The companies want to negotiate it among themselves, with an antitrust exemption and without any state authority to shut models down. That makes it dependent on the goodwill of exactly the companies under the greatest economic pressure. Sincere concern and commercial interest are not mutually exclusive. Here they simply lead to the same proposal.

Conclusion

Since the summer, an AI escaping control is no longer a science fiction scenario, and the warnings have a real basis. The date circulating in the debate for humanity’s extinction has none. Where the scaling of language models hits its limits, a bursting bubble is more likely than a vanishing humanity. Those issuing the warnings want regulation that audits and discloses, but none that shuts down. China is bound by none of their proposals, the competition very much is.

A slowdown whose pace the companies set themselves is not a brake. It is cruise control.

Sources

• Dario Amodei, “We Must Pace the Frontier”, Sep 12, 2026 (primary source): darioamodei.com
• SiliconANGLE, Altman and Musk back Amodei, Sep 13, 2026: siliconangle.com
• Fortune, Altman on postponing the IPO, Sep 12, 2026: fortune.com
• Fortune, CFO Friar on revenue growth, Sep 14, 2026: fortune.com
• Bloomberg, OpenAI IPO not before 2027, Sep 12, 2026: bloomberg.com
• Axios, GPT-6 Astra and Brockman’s AGI statement, Sep 3, 2026: axios.com
• TechCrunch, Astra and the ARC-AGI-3 scores, Sep 3, 2026: techcrunch.com
• Simon Willison, the end of the OpenAI-Microsoft AGI clause, Apr 27, 2026: simonwillison.net
• Al Jazeera, Hugging Face incident, Jul 22, 2026: aljazeera.com
• NPR, Hugging Face incident, Jul 23, 2026: npr.org
• TechCrunch, overview of all known break-ins by AI models, Aug 27, 2026: techcrunch.com
• CNBC, Gemini breaks into three companies, Sep 18, 2026: cnbc.com
• Axios, Google’s explanation of the Gemini incidents, Sep 19, 2026: axios.com
• Science Times, Coxon and Hubinger on extinction forecasts, Sep 11, 2026: sciencetimes.com
• Rep. Ted Lieu, AI Kill Switch Act, Jul 23, 2026: lieu.house.gov
• CNBC, kill switch bill and Jack Clark, Aug 6, 2026: cnbc.com
• Tech Insider, Rand Paul blocks Senate bill: tech-insider.org
• Tech Times, California order on a shutdown mechanism, Sep 21, 2026: techtimes.com
• Claims Journal, antitrust lawsuit, Sep 23, 2026: claimsjournal.com
• The Information, OpenAI cash burn in Q1 2026: theinformation.com
• R&D World, OpenAI’s projected losses for 2026, Jan 29, 2026: rdworldonline.com
• TradingKey, OpenAI’s data center commitments: tradingkey.com
• CNBC, Anthropic’s IPO and the essay, Sep 14, 2026: cnbc.com
• PYMNTS, Anthropic mulls new model, September 2026: pymnts.com
• Cybernews, OpenAI and Anthropic losses compared: cybernews.com
• ValueAdd VC, Anthropic’s burn rate and commitments: valueaddvc.com
• Time, Jacob Coxon’s resignation and the open letter, Sep 15, 2026: time.com
• CNBC, researchers call for a slowdown, Sep 10, 2026: cnbc.com
• Ashish Vaswani et al., “Attention Is All You Need”, 2017: arxiv.org/abs/1706.03762
• Albert Gu, Tri Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces”, 2023: arxiv.org/abs/2312.00752
• Yizhou Liu, Ziming Liu, Cengiz Pehlevan, Jeff Gore, “Universal One-third Time Scaling in Learning Peaked Distributions”, ICML 2026: arxiv.org/abs/2602.03685
• Yizhou Liu, Sara Kangaslahti, Ziming Liu, Jeff Gore, “Inverse Depth Scaling From Most Layers Being Similar”, ICML 2026: arxiv.org/abs/2602.05970
• Dan Grib, “MIT Just Found a Hard Limit in LLM Scaling, and Money Won’t Fix It” (video on the MIT papers): youtube.com
• Epoch AI, power demands of frontier training runs in 2030, Aug 11, 2025: epoch.ai
• Epoch AI, Can AI Scaling Continue Through 2030?: epoch.ai
• Sara Hooker, “On the Slow Death of Scaling”, December 2025: inductivebias.substack.com
• EA Forum, analysis of Sutskever’s statement on the end of pretraining: forum.effectivealtruism.org
• Adam Tauman Kalai et al. (OpenAI), “Why Language Models Hallucinate”, 2025: arxiv.org/abs/2509.04664
• OpenAI, o3 and o4-mini System Card, Apr 16, 2025: cdn.openai.com
• TechCrunch, OpenAI’s reasoning models hallucinate more, Apr 18, 2025: techcrunch.com
• OpenAI and Apollo Research, Detecting and reducing scheming in AI models, September 2025: openai.com
• Time, scheming in all leading models, September 2025: time.com
• METR, randomized study on experienced developer productivity, Jul 10, 2025: metr.org
• Futurum, AI capex 2026: futurumgroup.com
• CNBC, IMF and Bank of England warn of an AI bubble, Oct 9, 2025: cnbc.com
• Forbes, central banks join the AI bubble debate, Jul 8, 2026: forbes.com
• Al Jazeera, US proposal for an AI dialogue with China, Sep 20, 2026: aljazeera.com
• ORF, China’s bid for its own AI order: orfonline.org
• Concordia AI, State of AI Safety in China 2026, Jul 14, 2026: aisafetychina.substack.com
• Freedom House, China, Freedom in the World 2026: freedomhouse.org
• Axios, Chris Lehane: “We’re in a real race”, Mar 25, 2025: axios.com
• Axios, OpenAI and Anthropic against Chinese open-weight models, Jul 22, 2026: axios.com