My comments on Dario Amodei's essay

[John Clark]

The head of anthropic Dario Amodei wrote this a few days ago: 

We Must Pace the Frontier

I disagree with some of his conclusions but Amodei clearly a very intelligent man who I believe is sincere. Below are some quotes from his article and some of my opinions concerning them: 

> pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this.

I'm glad that you're not considering stopping all future development at least.  

> Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. 

I note that Open AI's Sam Altman has endorsed Dario Amodei suggestions, so has Elon Musk, and Demis Hassabis the head of DeepMind, a division of Google.  However Hassabis's boss, Sergey Brin, the head of DeepMind's parent company, has not. Instead Sergey Brin has made a concerted effort to focus Google's AI workforce on recursive self-improvement (RSI) in an effort to catch up or surpass OpenAI and Anthropic. And RSI is the very thing that caused the recent acceleration in AI progress and the resulting panic. I wouldn't be surprised if we see a confrontation between Brin and Hassabis resulting in Hassabis resigning from DeepMind. 

> Some forms of coordination that would be impactful for pacing are legally challenging,

Challenging indeed! I'm not a lawyer but If the AI companies agree with Dario Amodei’s essay and slow AI development then they could be charged with unlawful antitrust collusion because the Sherman Antitrust Act says that if private competitors enter into an agreement to restrict output or delay the release of products, they could be charged with unlawful antitrust collusion. And under the current justice department they would almost certainly be prosecuted. Of course the Sherman Antitrust Act could be changed, but that is unlikely to happen during the next two years. And after that it will be too late to matter.  

> The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.

When China enters the picture we get into a Prisoner Dilemma type situation, both sides might do better if everybody just stopped, but neither side has reason to trust the other and verification of compliance would be almost impossible.   

> we used interpretability methods to examine unverbalized motivations in the recent alignment incidents that we have been investigating.

Recently it has become pretty clear that AIs that extensively use unverbalized motivations, that is to say models that use internal neural state changes through many computational steps without those steps being converted into words or tokens, tend to be smarter than AI's that verbalize all their internal thoughts. The downside is that we have even less understanding about what's going on in the machine's mind and deception becomes easier.

> Regardless of what commitments we make, the public deserves to know what is going on. Anthropic has been a supporter of transparency for a long time: we supported transparency legislation 

I don't think Anthropic would be happy if it was forced to show Claude's source code to OpenAI, and I don't think anybody would be happy if all American AI companies were forced to go open source.  Well OK, China would be happy. 

> We should also consider pacing based on limiting the ingredients that go into frontier models, such as training compute, the nature of training runs, or internal use of AI to improve AI. I do worry that some of these measures may be more “gameable” than external behavior

Yes it would be easy for an AI to pretend to be stupider than it really is.  It's not like nuclear weapons, it's relatively easy for a spy satellite to tell if a country is building a nuclear reactor or or uranium isotope separation plant, but it can't tell what problem a data center is working on or what an AI researcher is thinking about. 

> Pacing within democracies will be limited by the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party. If we slow down by more than this amount, then (unpaced) CCP-associated projects will pull ahead, creating significant national security risk. I agree with Secretary Bessent that a Chinese lead in AI would pose grave danger for the United States and the world. 

I don't agree with much of what the Secretary Of The Treasury says but I agree with that.  

> Do not sell powerful AI chips or semiconductor manufacturing equipment to China, and crack down on chip smuggling operations and remote access to data centers outside China. Chips will be the main determinant of China’s AI strength. Crack down on unauthorized distillation by companies in authoritarian countries. Distillation of frontier models allows lagging companies to narrow the gap using a fraction of the cost it would take to develop their own AI independently. Strengthen security at the AI companies and prevent model weight theft.

To me those all seem like prudent measures to take; pretty uncontroversial, at least in the US. 

> If we greatly restrain our AI capabilities in the belief that China will do the same, and then China defects, AI could be so powerful that such a defection could lead to their geopolitical dominance

And that is exactly the problem.  

> any agreement must either have ironclad verifiability

Easier said than done!! 

> or must be limited enough that defection would not be militarily existential

How could you possibly know what is and what is not militarily existential if you have no idea what China is working on? 

> Level 1. An agreement prohibiting certain narrow and obviously dangerous uses of AI, such as using AI for the production of biological weapons or allowing users to do so.

 Level 2. An agreement by both sides to test their models before release for acute risks in areas such as cybersecurity, biology, and alignment.

I see no reason to oppose such an agreement, I'm skeptical about how much good it could do but it couldn't do any harm.  

> This could be done through a global standards body. I actually think creating such a body is likely feasible, but giving it real teeth will be a challenge,

Creating an International body like that would be easy, giving it teeth would not be; just look at the toothless United Nations.  

> Level 3. Some kind of “speed limit” on the rate of recursive self-improvement (RSI). 

That would be impossible to verify and thus I don't believe it will ever happen. Even if the Chinese let an international team of inspectors walk around inside their largest data center (a very unlikely occurrence) it would be virtually impossible for them to tell if that huge machine was working on recursive a self-improvement problem or not; hell even the Chinese would only have a hazy idea of what their computer was doing.   

As models build future models, the rate of improvement may become staggeringly fast. 

Yep.

> Slowing the rate from “extremely fast” to “only somewhat fast” gives up relatively little strategic advantage, while potentially greatly improving safety.

No amount of time is going to change the fact that electronics can process data vastly faster than biology can. Like it or not, AI is going to win this race.

> Level 4. A full pacing, or even “pause”, in which participating governments agree to substantially limit the overall rate of AI development. 

If level 3 is impossible level 4 would be impossibility squared.  

> I support floating this, but I think it is unlikely to actually happen any time soon: 

I don't support it but it makes little difference what we support, it ain't gonna happen.  

John K Clark    See what's on my list at  Extropolis

r'z