Undoubtedly you’ve heard about the ex-Anthropic Researcher Jacob Coxon who resigned because frontier labs are not acting responsibility in the face of AI doom.
Many have been analyzing the tweet’s instant permeation of the U.S. Some that claim that it was a coordinated psyop. Jacob was on Fox News and CNN last night. I watched the clips, his statements were light on details about how AI progress might lead to extinction and when pressed discussed the OpenAI incident.
Objective judgment, now, at this very moment. Unselfish action, now, at this very moment. — IX. 6
While its warranted to discuss his credibility given the hysteria he caused (like, is this guy even credible?) here is what I think:
The easy explanation is that he means it. Quitting a frontier lab and giving up your equity is a signal. It doesn’t make him right but calling it a “grift” is too simple an explanation.
The real accusation is not that labs ignore safety. They spend heavily on alignment, evals, interpretability, red teaming, cyber, bio, governance. But despite this spend the question to answer is whether this type of investment will stop rogue capability development.
If a lab believes there is even a meaningful chance of extinction, “we have a safety team” is not a proportionate response. At some probability greater than 10% (which is what a guy ON ALIGNMENT is saying) the rational action is simply: don’t ship it. The labs have pushed for this but motives questioned…
Why? because AI doom is conceivable without evil consciousness. A sufficiently capable system that pursues a misaligned objective may find it useful to avoid shutdown, gain resources, preserve its goals, deceive supervisors, or acquire influence.
The scary path is plausible in this outline:
better models → faster AI research → longer-horizon agents → weaker human oversight → loss of control.But this is still a chain of assumptions…. Intelligence is not agency. Agency is not power. Power is not control over the physical world.
The most important (wrong) leap being made FROM Jacob’s argument) and why it’s Doom Mongering is from: “this could happen” → “this is where we are obviously headed.”
At the same time there is also a much less speculative danger: AI makes cyber, bio, persuasion, and weapons expertise cheaper for humans. Recursive self improvement leading to “paper clip catastrophe” does not require super-intelligence.
Then there is self-interest. Is your profession in AI safety, governance, auditing, or alignment? Rising fear means more funding, relevance, influence, and demand for your services.
Regulation can also become a moat. If frontier AI requires expensive audits, licensing, reporting, security programs, and compliance teams, OpenAI and Anthropic can pay. Startups and open source cannot.
The most effective form of regulatory capture is not:
“protect us from competitors.” → “this technology is too dangerous to be built by anyone except organizations like us.”At the same time incentives can be critiqued in both directions. Labs, investors, employees, and chip companies have enormous financial incentives to believe the risks can be managed so that scaling can continue.
So in summary I think the serious position is: catastrophic AI risk is conceivable, worth taking seriously, and potentially under-governed.
But I do not believe we’re racing straight toward extinction, that its inevitable (we humans still have A WHOLE ton of intelligence, agency and power) and that if we did have reliable evidence to the contrary that we wouldn’t take action.
Perhaps the most extraordinary observation is that AI Doomers, Labs, Chip Companies Regulators and investors are all capitalizing on the same very premise: AI is extraordinary and powerful to the extent its worth fighting over who gets to build, define what safe means and control it.