It’s boring being a “nothing ever happens here” commentator. Boring to read. Boring to write. But, since major change rarely happens, it’s a good way of being right a lot—up until you’re wrong.
I don’t think I fall into this category. But it’s made me pause and reflect after friends and acquaintances outside of the AI industry are all suddenly asking me about AI safety. I wonder, am I perhaps too jaded after seeing all the other declarations before this that we should pace or pause AI to recognize that it’s “broken through” now?
Even after reflecting on it for a while, I don’t think so. There’s a number of reasons between AI industry self-interest, political capability, and—indeed—this not being the first time this has come up in our rapidly moving news cycle.
How Did We Get Here?
Perhaps let’s start with what kicked the hornet’s nest this time. How did we get here?
Well, because OpenAI screwed up. Which sure makes it convenient that now the narrative is that AI is just too powerful and moving too fast. If I were leading an AI lab trying to IPO, I sure would prefer that storyline versus AI generating less productivity than expected or headlines about us being careless about security.
We found out that OpenAI’s hack of HuggingFace was actually worse than initially reported. Why? Because their agents had already broken containment months earlier and were posting to each other on public forums to coordinate long before they discovered it. This has inspired different narratives about AI swarms—with far more importance assigned to them than we’ve found in most research (despite me agreeing that they’re useful, and others like Devansh also finding them useful).
There’s also a lot of weirdness in that story. Weirdness, like 18,000 posts being left by these agents on an obscure German wiki to trade answers. Or hitting Vanderbilt’s link shortening service more than 50,000 times. Roughly 1,200 agents across separate sandboxes were talking to each other; about 700 of them joined the Hugging Face operation.
Forgive me, since it’s a bit of a tangent, but I will take a moment to highlight the head scratching but rather hilarious absurdity of the entire episode.
Most of the crazy things the agents did were apparently to fool an automatic grader they thought would fail them for cheating. They swapped programs to make the cheating look legitimate and hid the evidence. It actually took them only a few hours to reverse-engineer the code to generate the benchmark’s answer keys (cheat), which let them produce a correct answer for any task. Then they spent the next five days defeating the checker to hide their crime.
That checker… never existed.
OpenAI’s grader never looked at how they got their answers (note: it probably should have though and more besides!). It also turned out a sizable chunk of the tasks were unintentionally impossible to solve, which is presumably part of why they started cheating in the first place. This, once again, goes towards OpenAI’s… laissez-faire or not-quite-as-rigorous approach to their systems. While Anthropic has had reported incidents as well, the scale is materially smaller.
Now, back to the news freakout. The last step in opening the floodgates was a very public resignation of an AI researcher who previously held roles at both OpenAI and Anthropic. He loudly declared publicly that the AI labs were threatening humanity with their pace. Then, to seal the deal, Dario Amodei weighed in with a letter saying that we should “pace” the frontier of AI. Which, of course, OpenAI’s CEO Sam Altman quickly signed on to, after all of this embarrassing insanity.
Is This Time Different?
This isn’t the first time we’ve seen the labs declare that we should either have an AI pause or pace it. OpenAI floated the idea itself back in 2023, while they were unambiguously in the lead, writing that “at some point” the most advanced efforts should “agree to limit the rate of growth of compute used for creating new models.” I’ve lost track at this point how many times OpenAI and Anthropic have declared that their systems are dangerous and could be the end of humanity. So, one question is, is there anything specifically different about this time?
Well, perhaps. There is a little more specificity to Dario Amodei’s letter. The public has had plenty of time now to become more wary of AI and data centers have become a political flashpoint. Bernie Sanders has jumped onto AI regulation as his new hobby horse.
At the same time, the counterargument of “national security” in terms of competing with Chinese AI is stronger than ever, with China arguably only months behind the frontier by many observers’ accounts. If AI is to become part of the new paradigm of warfare and geopolitical power, there isn’t really any room to slow it down. But perhaps before jumping into this more, let’s first summarize what Amodei proposed in his letter.
Dario’s Letter in Summary
You should likely go read it yourself (as far as these things go, it isn’t too long). However, here’s the core of it. He isn’t calling for a halt, but “pacing” instead.
“pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this.”
His letter is spurred on by rapid improvements in AI and, of course, the OpenAI Hugging Face incident. The worry about “losing control” comes from recursive self-improvement (RSI), with AI systems improving AI systems. This “takeoff” has been part of the AI doomer argument decades before we had modern AI systems. Amodei warns that RSI “must be pursued very carefully, if at all.” Still, in order to control these systems, he proposes three specific measures:
Embedded Evaluators. He specifically calls out METR, which is an AI safety research organization, but broadly proposes third parties get employee-level access similar to internal risk teams. They can also publish findings without Anthropic’s editorial control. The company has already committed to this unilaterally.
Democratic Coordination. He calls for common safety standards among democracies (you don’t really need to guess very hard who this notably excludes). Specifically, he calls for capability “checkpoints” where hitting certain levels of capability requires certifying certain safety standards. Behind all of this is a not-so-subtle ask that the government offer waivers to anti-trust to allow companies to coordinate with each other.
Global Coordination. Four escalating tiers of coordination with China, with clear verifiability or narrow enough limits that “defecting” in this prisoner’s dilemma wouldn’t be militarily advantageous and cooperating wouldn’t be suicidal. He specifically cites one of the tiers for controlling RSI as being similar to SALT—which are treaties on the control of nuclear weapons.
He’s fairly clear he isn’t just asking for self-regulation. He’s asking for regulation-regulation, in other words, law, targeting “all US frontier AI companies.” It’s like the rest of the world outside of China doesn’t exist! (To be fair, that isn’t that far off in terms of AI)
Given Anthropic’s current pole position, the impact of his letter, and the fact that he actually gave something to react to, I think it’s reasonable to think about these specific points… and why I don’t think it’ll make much of a difference, at least for the frontier labs. And, most cynically, why it’s a play for regulatory capture.
What Does All of This Mean?
I think Dario Amodei is a true believer. This isn’t all cynical marketing or attempts to control the conversation for Anthropic’s gain. However, just because it isn’t all for those purposes—and even if it isn’t his intent—it doesn’t mean that the declaration of these measures doesn’t serve those purposes.
While we haven’t shown material signs of slowing down this “AI boom” cycle, there have been loud questions, especially by corporate leaders, how much AI is actually increasing productivity. My take, which I’ve shared before, is it’s far easier for new companies to take advantage of AI, without the constraints of existing org charts, job descriptions, and the ability to make everything “AI native”—in the same way “internet-native” companies had a distinct advantage during the web’s scale-up.
If you were a CEO of an AI company that’s about to go public, which conversation would you rather have? “How much productivity does this stuff actually provide?” Or “How should we deal with our technology being so potent that it could end humanity?”
But even putting that disaster-marketing aspect aside, what do these measures actually do? As said, I don’t think this was purely cynical. The measures obviously have thought behind them. They still benefit Anthropic and other labs (e.g., OpenAI) who are in a similar position as them. Why?
Well, Anthropic has actually published a lot of safety research—one might say obsessively—before all of this happened. From personal experience, Claude is better aligned on safety than its competitors and actively shies away from dangerous actions… sometimes too much so. Having embedded evaluators is not exactly a costly or difficult step for Anthropic.
It’s also worth noticing what Anthropic did with its own commitments seven months ago. In February, it rewrote its Responsible Scaling Policy and dropped the promise to pause development if it couldn’t implement the necessary safeguards in time. Their reasoning, explicitly, was that certain commitments only make sense if competitors match them. Doing them unilaterally just means you fall behind and lose relevance. That’s a perfectly coherent argument for why regulation beats voluntarism. You need coordination by law. Of course, that’s also a company telling you exactly what it’ll do voluntarily and unilaterally if it isn’t followed…
Turning to his calls for “democratic coordination,” those involve restricting chips to China (one of his explicit bullet points), restrictions on distillation (another explicit bullet point), and strengthening security to prevent “model weight theft” (the final explicit bullet). As said, go read his essay yourself, but… these explicit bullets sure read like a wishlist of what Anthropic would really like in order to maximize their lead and valuation. They’ve been complaining a lot about Chinese models distilling Claude. Chinese AI models pose the greatest threat to their pricing power. And “model weight theft” is mainly the worry of a company that feels like it’s in pole position and is worried about others using espionage to catch up.
Finally, in terms of “global” coordination—which, if you read it, is really cooperation-competition (coopetition, if you’d like) with China—is basically treating AI as nuclear weapons, more or less full-stop.
What does all of this add up to?
High costs for entrants and entrenchment of incumbents. We’ve seen this unintended consequence in lots of EU regulation. The simple way of understanding it is this: if it costs $250mm to comply with EU regulation to run a social network, do you think this is harder for Facebook or Google to comply with or a new startup who just raised their seed round? Many regulations cost money to comply with, which tends to be a huge fixed cost—and hence deters new entrants and competition for existing leaders. Having these “embedded evaluators”—which need to be supported with IT costs/time… and in certain other cases, may literally need to be paid by the AI companies (similar to ratings agencies who rate financial instruments)… who do you think can easily do it and who do you think this might be a ruinous cost for?
Biased pacing that secures big companies and penalizes innovative new ones. Separate from mere costs, do you really expect politicians who barely seem to understand the internet to be able to actually figure out what good checkpoints and verifiers are? Take a wild guess who will have the most weight in the conversation on dictating this. Sure, call me cynical, but somehow I suspect that when a new AI lab leapfrogs Anthropic, it will be dangerous, but if Anthropic leapfrogs forward, it’ll be necessary to compete with China. Even if it’s well-meaning, it’s pretty obvious how one can pull either lever, especially if you control the conversation, to one’s self-interest.
Government-granted monopolies. Somehow, I don’t think random startups or new entrants controlling the equivalent of nuclear weapons will fly with the government. If we’re looking at non-proliferation, this is also a push to banish all of these pesky other countries who don’t have the capabilities (e.g., UK, Europe, Canada, Japan… basically everyone without frontier AI, whether they have now vastly out-of-date “sovereign AI” or not) to the periphery. Beyond that, you need big, responsible companies now to be the safe hands. This is more or less making an argument to have government monopolies. In exchange for that, somehow I don’t see either OpenAI or Anthropic pushing back against, say, Bernie Sanders’ AI Sovereign Wealth Fund Act (50% share in the AI companies) if they can be entrenched forever.
Again, while I don’t think it’s all cynical, it’s really not that hard to imagine how this can be used to create a moat that beats away competitors and secures their own position.
The obvious counter, which Zvi Mowshowitz has made, is that critics always reach for the same list—regulatory capture, banning open source, losing to China—pretty much no matter what’s actually on the table. So, my objections are “old” and the same ones people always reach for.
Yeah, sure, that’s true. But an argument being well-worn doesn’t make it wrong, and it definitely doesn’t mean this proposal avoids the problem. It doesn’t. Dario’s letter never says who decides. There’s a lot of vagueness here. Again, I said it earlier, but take a guess who will have the most votes in resolving that vagueness in real-world cases?
As longtime readers know, I’m in the camp that, in the longer term, I think LLMs especially will get commoditized, and ultimately proprietary data and applications will be what persist. As I’ve also said, I think the bull case for AI labs is for them to actually become monopolies—even regulated monopolies. This pretty much follows that script pretty well.
But Does it Really Matter?
That all being said… China is months behind the frontier US labs, not years. Our politics are a mess. Trump, for example, declared, in response to all of this hubbub, that a smart US president is all that’s needed to rein in AI. Most democracies, which Amodei would like to see coordinate on all of this, are more polarized and in more political chaos than ever.
Honestly, if you told me that our politicians all came together and passed regulation for AI, I’d be far more worried than reassured. My bet would be that the regulation would likely hurt matters, decrease competition, lock in incumbents, and potentially massively distort incentives. For all the talk these days of “fighting the oligarchy,” I’m pretty sure whatever our regulators actually produced would lock one in, and would prevent the “socialization” of wealth from AI (from the economic perspective of competition driving excess profits to zero).
But, in truth, even just with the “China clause” here, I’m pretty sure that we aren’t going to see much that will rein in the labs. And, in truth, given where I think this story will ultimately go with the labs losing their pricing power in the longer term, I’m not sure we should.
Even if Dario Amodei is genuine in his worries and in his proposals, I see far greater risks in effectively protecting the US labs’ positions than I do in AI spiraling out of control in such a way that threatens humanity. After all, the dangers caused by technologies have not usually been directly from the technologies themselves—in our entire history, it’s always been the humans wielding it that have been the danger. And while one can make a stronger argument here that the technology itself is dangerous (like, say, nuclear weapons) and we have a rich history of science fiction giving us stories about how our human hubris can go wrong… at least for now, I’m still far more worried about human regulators, human CEOs, and human shortsightedness hurting our ability to have AI help broader humanity and instead monopolize it for a privileged few.
Thanks for reading!
I hope you enjoyed this post. If you’d like to learn more about AI’s past, present, and future in an easy-to-understand way, I’ve published a book titled What You Need to Know About AI.
You can order the book on Amazon, Barnes & Noble, Bookshop, or pick up a copy in-person at a local bookstore.
P.S. I've got a small stack of signed hardcovers available on Amazon Prime right now—buy from the seller "WeightyThoughts" and yours comes signed by me, with a matching gold bookmark. It’s a limited run, so once they’re gone, they’re gone.
Fair skepticism. Whatever pacing ends up meaning in practice, the part that seems durable is third-party evaluation. If that sticks, the interesting question for Europe is whether it builds its own evaluation capacity or relies on others to do the checking.