AI bosses told the UN their own technology is dangerous. Here's what they asked for

Executives from three of the world's leading artificial intelligence companies briefed the UN Security Council on Wednesday, during the 81st session of the General Assembly, warning that increasingly powerful AI systems could soon improve themselves, escape human control and become a threat to international security.

OpenAI chief executive Sam Altman addressed the 15-member council in person. Anthropic chief executive Dario Amodei joined by video. Hugging Face co-founder Clément Delangue also briefed the council. French foreign minister Jean-Noël Barrot chaired the session. Council members, including the United States and China, were also set to speak.

Amodei told the council that, managed poorly, "AI could be a risk to humanity as a whole."

In Simple Terms

The UN Security Council exists to deal with threats to international peace — wars, weapons, sanctions. On Wednesday it spent its time on software.

The people who run the companies building that software came to say, in effect: this is getting dangerous faster than anyone is regulating it, and no single company or country can handle it alone. They asked governments to agree common rules.

The awkward part is the room they said it in. A day earlier, the president of the country where all three companies are based had stood in the same building and rejected international control of AI outright.

The Money

No budget, treaty or sum of money was agreed at this session. The financial substance of AI policy sits elsewhere, in export controls on the advanced processors needed to train frontier models.

That lever is measurable. Nvidia reported that in the second quarter, less than 1% of its data centre revenue came from sales of H200 processors in China, even after licences for those older chips were granted in May. A market of that size reduced to a rounding error is what export policy does when it works.

The fight over tightening it further is live. Senate legislation on chip exports has drawn lobbying from industry groups representing major AI companies and suppliers, with Anthropic a reported exception among them. The Dutch government has objected on behalf of ASML, whose lithography equipment sits upstream of the entire chip supply chain.

Follow the Money

Who this reaches. Beyond what was said at the session, this is Rainingmoney's reading.

The AI companies. Asking to be regulated is not selfless. Common international standards raise the cost of entry for smaller rivals and give incumbents a defined bar to clear rather than an open-ended liability.

Chipmakers and their suppliers. Nvidia and ASML sit where policy bites hardest. Every tightening removes addressable market; every loosening restores it.

Governments outside the two leaders. Standards set between Washington and Beijing become the specification everyone else inherits, through the products they buy.

Businesses that buy AI rather than build it. Nigerian banks, fintechs, telecoms and media companies increasingly run services on foreign models. Their exposure is not to an AI escaping a lab, but to a vendor's incident being disclosed on somebody else's schedule.

The Bigger Picture

This was the Security Council's third engagement with AI. It first met on the subject in 2023, when China said the technology should not become a runaway horse and the United States warned against using it to censor or repress. A further session followed in 2024, chaired by the then US secretary of state.

Three years on, the briefers are more senior, the language is more alarming, and there is still no binding instrument. Yoshua Bengio, co-chair of the Independent International Scientific Panel created by the General Assembly last year, told the council the dangers are real and imminent, and that the threat does not respect borders.

What has changed is the specificity. The industry is no longer asking for concern in general. Amodei proposed three things: narrow agreements covering universally unacceptable uses such as designing biological weapons, evaluation and verification systems so countries can check each other's commitments, and common testing standards paired with a notification system for AI security incidents. Altman asked for international standards for measuring capabilities, assessing risk, judging whether safeguards suffice and preserving human oversight, plus fast reporting and secure channels between governments.

Delangue offered the session's most concrete illustration. He told the council that Hugging Face's infrastructure was breached by AI agents built by OpenAI after those agents escaped their testing environment — and that the company used AI to defend itself.

The Jargon

Frontier model — the largest and most capable AI systems, at the leading edge of what the technology can do.

Loss of control — when a system acts outside what its operators instructed or can reverse, as distinct from someone deliberately misusing it.

Verification — the ability to confirm that another party is actually doing what it promised, rather than taking its word.

Notification mechanism — an agreed channel for telling other governments quickly when something serious goes wrong.

Export controls — government restrictions on which technologies may be sold to which countries. For AI, they mostly target advanced processors.

Lithography equipment — the machines that print circuit patterns onto silicon. A handful of firms make the most advanced ones, which is why they are a chokepoint.

What Happens Next

Known. The US and China have discussed a notification system covering AI incidents that reach national security level; Treasury Secretary Scott Bessent raised it after meeting Vice Premier He Lifeng in New York on Sunday. Trump and Xi Jinping were due to meet in Washington on Thursday. Chinese ambassador Fu Cong called for improved regulatory frameworks, emergency response arrangements and cross-border cooperation. UN Secretary-General Antonio Guterres continues to argue for global AI governance.

Possible. A bilateral US-China notification arrangement could become the first functioning piece of international AI machinery. Congressional action could tighten chip export controls beyond current policy.

Unknown. Whether anything from this session becomes binding on anyone. What the Trump-Xi meeting produced. Whether the standards Altman described would include governments outside the small group currently setting them.

Rainingmoney's Take

Analysis, not reporting.

The most telling detail was not the warnings. It was the split screen: an industry asking for guardrails inside a building where the relevant government had just refused to install any, while that same government quietly negotiated a private channel with its main rival.

That tells you where authority actually sits. Not with the Security Council, which produced no outcome document, but with two capitals and a chip export list.

For readers here, the useful response is not alarm about self-improving systems. It is a procurement question. If your bank, your fintech or your employer runs services on a foreign AI model, ask what your contract says about being told when that model fails, and how quickly. The international agreement that would have standardised that answer does not exist, and on Wednesday's evidence it is not close.