What an AI Armageddon might actually look like

Runaway code and self-improving algorithms have left the likes of OpenAI and Anthropic staring into a potentially apocalyptic future of their own making Researcher Jacob Coxon became the latest insider to sound a dire warning about the threat to the world from artificial intelligence when he quit AI giant Anthropic this week. For three years, the 27-year-old British engineer trained models at OpenAI and Anthropic. Both AI start-ups acted irresponsibly, Coxon wrote on X on Wednesday, saying: "The people building AI earnestly believe that it could kill us all by the end of the decade." The response from his manager arrived just hours later. Evan Hubinger, head of alignment research at Anthropic, also confirmed the warning, saying: "We are, in fact, earnestly convinced that AI could kill all humans! I think the probability of that happening within the next decade is greater than 10pc." Anthropic, he added, does not yet have a solution to this problem. When the head of safety at a multi-billion-dollar technology company casually predicts humanity's downfall at the hands of his own product, the debate leaves the realm of academic thought experiments. The warning comes from the engineers writing the code. Both developers point to an industry dynamic in which competitive pressures outweigh safety concerns. But what exactly are AI developers afraid of? What is the hypothetical scenario in which artificial intelligence brings about Armageddon? The two AI Cassandras are not warning about the current generation of AI models, though even these have already been caught deceiving their human overseers, engaging in hacking and taking part in criminal activity online. Instead, Coxon and Hubinger fear the next stage of development. In their competition with one another, OpenAI and Anthropic are pursuing what is known as recursive self-improvement, which Hubinger says is arriving faster than expected. Behind the technical term lies an algorithm capable of analysing its own source…