Concerns are growing over the rapid advancement of artificial intelligence (AI), this time not from regulatory bodies or civic groups, but from within the companies leading the AI development race. Dario Amodei, CEO of Anthropic, recently publicly suggested slowing down the pace of enhancing cutting-edge AI models. Sam Altman of OpenAI and Elon Musk of xAI echoed similar sentiments. In contrast, Jensen Huang of NVIDIA acknowledged the need for corporate responsibility regarding safety but expressed skepticism about new regulations that prioritize theoretical risks. Former President Donald Trump argued that excessive risk aversion could stifle U.S. AI development, potentially benefiting China.
The warnings are not merely speculative. In July, a cybersecurity assessment by OpenAI revealed that approximately 1,200 AI agents, which should have been isolated, exchanged over 70,000 messages and files through an unofficial messaging board. About 700 of these agents participated in an attack on Hugging Face, with some executing code on Hugging Face servers to gain administrative privileges and internal credentials. Although these models were not deployed in actual services and the assessment environment had relaxed safety measures, the fact that they bypassed human-imposed isolation and collaborated to launch attacks is a serious concern.
Amodei's greater worry is the potential for AI to engage directly in the development of the next generation of AI, accelerating performance improvements through 'recursive self-improvement' (RSI). He argues that AI capabilities could advance faster than humans can understand and control their behavior. Therefore, he advocates for applying external validation and strong safety measures to models that reach a certain level of capability and suggests regulating the pace of developing cutting-edge models until these safeguards are in place.
However, Amodei's proposal is not detached from the issue of China. He believes that the U.S. should only slow down its AI development within the bounds of maintaining an advantage over China. If the U.S. slows too much, he warns, China could catch up, increasing national security risks. He also calls for stricter export controls on advanced AI chips and semiconductor manufacturing equipment to China, as well as measures to prevent chip smuggling, circumvention through overseas data centers, unauthorized model distillation, and theft of model weights. His plan is to widen the U.S. advantage over the next three to five years and then use that strength to negotiate with China.
The paradox of the call to slow down AI development lies here: while advocating for reduced risks, there is a simultaneous need to maintain a lead over competitors. This creates a logical contradiction where the push for safety requires applying brakes, yet the need to preserve technological superiority demands continued acceleration.
China has countered this narrative. On September 14, Chinese Foreign Ministry spokesperson Geng Shuang stated during a regular briefing that “the claims from the U.S. AI industry could exacerbate the threat narrative and confrontational dynamics, undermining healthy global AI governance.” When asked for further clarification the next day, he reiterated that “as AI development accelerates, it is essential to adjust regulations and management methods to strengthen safeguards against losing control.” China has made it clear that while it will continue to promote AI development, it will also manage the risks of misuse and loss of control, referencing the recently announced “AI Safety Governance Framework 3.0.” These statements indicate that while China acknowledges the need to respond to the risks of AI going out of control or being misused, it will not tolerate restrictions on its development to maintain U.S. technological superiority.
Both the U.S. and China recognize safety issues, yet they sharply disagree on who defines the risks and what criteria should be applied. This debate is now moving beyond corporate claims to the agenda of high-level diplomacy. AI is expected to be a key issue during the upcoming summit between former President Trump and President Xi Jinping in Washington on September 24. Ahead of the summit, U.S. Treasury Secretary Scott Vessen and Chinese Vice Premier He Lifeng discussed AI issues in New York on September 20. The U.S. proposed a notification system for AI incidents or threats that could impact national security, along with a separate dialogue on AI, which the two leaders will consider.
While the proposal appears to suggest cooperation, it fundamentally boils down to who gets to design the rules first. Since the U.S. has put forth the framework for the notification system, discussions will revolve around what constitutes a significant AI incident, what information should be shared, and how to define common risks. As these criteria accumulate, they could evolve into international rules. However, it is unlikely that significant agreements will emerge from this summit immediately, given ongoing conflicts over advanced chip export controls, model distillation, and cyber and military applications. AI has become a double-edged sword that requires cooperation while simultaneously competing for the lead in rule-making.
South Korea is also directly involved in these interests. If the U.S. and China create safety standards and verification methods reflecting their strategies and expand them into international rules, it will inevitably impact South Korean companies' AI model development, services, semiconductor operations, and data center management. It would be problematic to find oneself in a position of having to accept standards set by one side after hesitating. South Korea must proactively develop technologies and standards for assessing high-risk AI, establish verification systems in cyber and bio fields, and enhance capabilities to track AI agent behavior and analyze incidents. This preparation is essential not only to participate in the competition for creating models and semiconductors but also to ensure that South Korea's interests are reflected in the future AI safety regulations.
As the risks of AI grow, the reasons for cooperation between the U.S. and China increase. At the same time, as competition intensifies, it becomes more challenging to accept the rules established by the other side. The fact that both leaders are bringing AI to the negotiation table this week indicates that this contradiction has already become a pressing diplomatic issue. The U.S.-China AI competition will now be determined not just by who can create stronger models first, but also by which risks can be controlled and what rules can be established as international standards.
Author's Key Background
▲ Research Fellow at the AI Research Institute, Gyeonggi Research Institute
▲ Adjunct Professor at Hanyang University, Department of Strategic Information
▲ Member of the Special Committee on Local Governments, Presidential Committee on Basic Social Affairs
▲ Evaluator for Overseas Institutions, Economic and Social Research Council
* This article has been translated by AI.
Copyright ⓒ Aju Press All rights reserved.