Anthropic Alignment Science Lead Warns of Catastrophic AI Risks as Internal Departures Mount

A top safety scientist at Anthropic has issued an extraordinary warning about the catastrophic risks of frontier artificial intelligence, estimating a greater than 10% chance that advanced AI systems could cause human extinction within the next decade. Evan Hubinger, Anthropic’s Alignment Science Lead, publicly acknowledged on social media that the company does not yet possess a comprehensive plan to control artificial superintelligence and is "not clearly on track to" develop one in time. The remarks came in response to the high-profile resignation of researcher Jacob Coxon, who accused leading AI labs of an irresponsible race toward self-improving models, reigniting severe concerns over whether commercial pressure is outpacing foundational safety research.

Background

To understand the gravity of these statements, it helps to understand the "alignment problem" in artificial intelligence. At its core, AI alignment refers to the challenge of ensuring that an artificial intelligence system reliably understands, adopts, and pursues human values and intended goals without unexpected, harmful side effects.

While existing models—such as Anthropic's Claude or OpenAI's ChatGPT—are aligned using techniques like reinforcement learning from human feedback (RLHF), safety scientists distinguish between current large language models and future "artificial superintelligence" (ASI). Superintelligence refers to a hypothetical system whose general cognitive abilities dramatically exceed those of human experts across virtually all domains.

If an ASI system were to undergo recursive self-improvement—where an AI autonomously designs and trains increasingly capable iterations of itself—it could rapidly acquire immense power, autonomy, and influence. Without a foolproof mathematical and scientific framework to control its motivations, such a system could pursue objectives contrary to human survival, a scenario that researchers refer to as existential risk, or "p(doom)."

Anthropic was founded in 2021 by former OpenAI researchers specifically to position itself as a public-benefit, safety-first alternative, dedicated to pioneering research into AI interpretability, stress-testing, and responsible scaling policies.

What Happened

The controversy began when Jacob Coxon, an AI researcher who spent three years working on model pretraining across both OpenAI and Anthropic, announced his resignation. Coxon published a series of posts on X (formerly Twitter) explaining his decision to walk away from frontier labs, accusing the industry of gambling with human civilization.

"Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives," Coxon wrote, adding that top engineers and executives privately harbor deep fears about the technology. "The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt."

Rather than pushing back against the critique, Evan Hubinger, Anthropic's Alignment Science Lead, publicly validated Coxon’s concerns.

"Jacob is correct here—we really do earnestly believe AI could kill all humans!" Hubinger wrote on X, as reported by Hindustan Times and Axios. "I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to."

Hubinger clarified that risks from current, deployable language models remain low. However, he emphasized that recursive self-improvement appears to be developing faster than anticipated, leaving safety researchers with little room for error as capabilities advance.

Why It Matters

Hubinger's 10% figure is not an official corporate forecast from Anthropic, as noted by tbreak, but an unusually blunt admission from a scientist leading the very team charged with keeping the lab's models aligned.

In conventional risk assessment, a greater-than-10% probability of an existential catastrophe occurring within ten years would prompt immediate, sweeping international intervention. If an aviation or nuclear agency stated a 10% risk of total catastrophic failure in a decade, public authorities would halt operations until safety protocols were guaranteed.

Instead, the AI sector finds itself caught in an intense competitive equilibrium. Multiple leading labs and sovereign states are pursuing superintelligent capabilities, fearing that pausing or slowing down will simply allow competitors to pull ahead.

Furthermore, these events highlight mounting friction within top labs. Over the past several years, safety researchers across the ecosystem have departed under similar circumstances, including the high-profile disbanding of OpenAI's Superalignment team after leaders like Jan Leike resigned, as detailed by Forbes. When the most safety-conscious engineers leave the organizations building the technology, it signals a growing rift between long-term research goals and the commercial mandate to deploy next-generation models.

What's Next

Coxon's departure and Hubinger's candid remarks will amplify calls for external regulation, international treaties, and legally binding safety standards.

Key areas to watch going forward include:

• Regulatory Oversight and Capabilities Pauses: In his public comments, Coxon urged governments to consider coordinated limits on frontier training runs, including potential multilateral agreements between major powers such as the United States and China to curb runaway capability growth.
• Independent Auditing: Expect increased scrutiny on how independent evaluation bodies—such as national AI Safety Institutes—inspect frontier models for self-improvement and autonomous cyber capabilities before public deployment.
• Internal Governance at Anthropic: As Anthropic continues to raise capital and scale enterprise offerings, the company will face pressure from investors and researchers to demonstrate concrete milestones in its superalignment roadmap rather than relying on tentative scaling policies.

Whether tech companies can bridge the widening gap between machine capability and technical alignment remains the most consequential open question in modern science.