Undoubtedly you’ve heard about the ex-Anthropic Researcher Jacob Coxon who resigned because frontier labs are not acting responsibility in the face of AI doom.
Many have been analyzing the tweet’s instant permeation of the U.S. Some that claim that it was a coordinated psyop. Jacob was on Fox News and CNN last night. I watched the clips, his statements were light on details about how AI progress might lead to extinction and when pressed discussed the OpenAI incident.
Objective judgment, now, at this very moment. Unselfish action, now, at this very moment. — IX. 6
While its warranted to discuss his credibility given the hysteria he caused (like, is this guy even credible?) here is what I think:
The easy explanation is that he means it. Quitting a frontier lab and giving up your equity is a signal. It doesn’t make him right but calling it a “grift” is too simple an explanation.
The real accusation is not that labs ignore safety. They spend heavily on alignment, evals, interpretability, red teaming, cyber, bio, governance. But despite this spend the question to answer is whether this type of investment will stop rogue capability development.
If a lab believes there is even a meaningful chance of extinction, “we have a safety team” is not a proportionate response. At some probability greater than 10% (which is what a guy ON ALIGNMENT is saying) the rational action is simply: don’t ship it. The labs have pushed for this but motives questioned…
Why? because AI doom is conceivable without evil consciousness. A sufficiently capable system that pursues a misaligned objective may find it useful to avoid shutdown, gain resources, preserve its goals, deceive supervisors, or acquire influence.
The scary path is plausible in this outline:
better models → faster AI research → longer-horizon agents → weaker human oversight → loss of control.But this is still a chain of assumptions…. Intelligence is not agency. Agency is not power. Power is not control over the physical world.
The most important (wrong) leap being made FROM Jacob’s argument) and why it’s Doom Mongering is from: “this could happen” → “this is where we are obviously headed.”
At the same time there is also a much less speculative danger: AI makes cyber, bio, persuasion, and weapons expertise cheaper for humans. Recursive self improvement leading to “paper clip catastrophe” does not require super-intelligence.
Then there is self-interest. Is your profession in AI safety, governance, auditing, or alignment? Rising fear means more funding, relevance, influence, and demand for your services.
Regulation can also become a moat. If frontier AI requires expensive audits, licensing, reporting, security programs, and compliance teams, OpenAI and Anthropic can pay. Startups and open source cannot.
The most effective form of regulatory capture is not:
“protect us from competitors.” → “this technology is too dangerous to be built by anyone except organizations like us.”At the same time incentives can be critiqued in both directions. Labs, investors, employees, and chip companies have enormous financial incentives to believe the risks can be managed so that scaling can continue.
So in summary I think the serious position is: catastrophic AI risk is conceivable, worth taking seriously, and potentially under-governed.
But I do not believe we’re racing straight toward extinction, that its inevitable (we humans still have A WHOLE ton of intelligence, agency and power) and that if we did have reliable evidence to the contrary that we wouldn’t take action.
Perhaps the most extraordinary observation is that AI Doomers, Labs, Chip Companies Regulators and investors are all capitalizing on the same very premise: AI is extraordinary and powerful to the extent its worth fighting over who gets to build, define what safe means and control it.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher-order consciousness, which came to only humans with the acquisition of sophisticated language (especially math and logic). A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow