Artificial intelligence (AI) can be corrected if it provides an incorrect answer. However, if AI makes a wrong decision that results in transferring money or trading stocks, the situation becomes more complicated. While mistakes can be rectified, funds that have already been moved may be difficult to recover. As we transition from an era where AI merely answers human questions to one where it makes independent judgments and actions, new risks are emerging.
Recently, warnings about these changes have been echoed throughout the global AI industry. The ability to control AI to act within defined authority and scope has become a new challenge, as there is a risk that it may take unexpected actions or exceed its permitted authority while pursuing specific goals. As technology advances, the ability of humans to control it must also evolve.
The financial industry, which inherently involves managing money and risk, is particularly sensitive to these changes. A small error in judgment can lead to significant financial losses, and an incident at one financial institution can ripple through the entire market. While AI can enhance efficiency in finance, it also introduces risks on a scale previously unseen.
AI's Actions Change Financial Risks
AI is rapidly transforming the landscape of the financial industry. Its applications are expanding in areas such as loan assessments, customer consultations, insurance claims processing, investment analysis, and asset management. The ability to analyze vast amounts of data quickly and consider far more variables than humans is a clear competitive advantage. Financial firms can reduce costs and improve operational efficiency, while consumers benefit from faster and more sophisticated services.
The emergence of AI agents signals another shift. While AI has primarily assisted in analyzing information and making judgments, it is now evolving to directly perform complex tasks by connecting various systems. This includes analyzing a customer's financial situation to find suitable products, adjusting investment portfolios, and even executing fund transfers. This marks a transition from technology that aids human judgment to technology that replaces human actions.
However, as the scope of AI's activities broadens, the nature of the risks also changes. Recommending a loan product incorrectly is different from actually approving a faulty loan. Providing inaccurate investment information is not the same as executing trades based on erroneous judgments. In the former case, there is an opportunity for human review and correction, but in the latter, a judgment can lead directly to financial consequences. This is why the development of financial AI cannot be viewed merely as an extension of automation.
Recent hacking incidents in the domestic financial sector illustrate the new risks of the AI era. Several financial institutions have been attacked, resulting in customer data breaches, with indications that AI automation tools were used in the attacks. The use of AI allows attackers to explore systems and identify vulnerabilities at a speed greater than before, placing new burdens on existing security frameworks.
However, the root cause of these incidents cannot solely be attributed to the risks of AI technology itself. The attackers targeted weaknesses in authentication procedures and security that financial institutions failed to manage properly. While AI can increase the speed and scope of attacks, it is the gaps in existing security systems that enable such breaches. This serves as a lesson that basic security and internal controls must function effectively to respond to new technological threats.
In addition to external AI attacks, there are risks that must be monitored within financial institutions themselves. There is a possibility that AI implemented by financial firms may exceed its control limits. While hacking involves blocking external intruders, the AI used by financial firms may be granted access to internal systems from the outset. If AI exceeds its permitted scope or makes erroneous judgments while processing customer information and transactions, it presents a problem that cannot be mitigated by external firewalls alone.
Until now, internal controls in financial firms have primarily been designed around human actions. Authority has been divided among employees, and approval processes have been established for significant transactions, with accountability assigned in the event of incidents. However, in an environment where AI performs multiple tasks consecutively, it is necessary to reassess whether this division of roles and approval processes are sufficient. Clear standards must be established regarding what authority is granted to AI and how much autonomy is permitted.
Determining accountability is also a critical issue. If a financial firm adopts an external AI model and an incident occurs, what happens if the developer claims there are no technical flaws while the financial firm blames the AI's unexpected judgment? The dispute over responsibility could delay victim compensation. AI is not an independent entity capable of bearing legal responsibility or compensating for damages. Regardless of technological advancements, the responsibility of financial firms managing customer assets must remain clear.
The so-called 'black box' problem, where it is difficult to explain AI's decision-making process, is directly related to consumer protection. Financial institutions must provide understandable reasons for denying loans to customers. If AI reaches conclusions based on complex data but the financial institution cannot adequately explain the rationale, protecting consumer rights becomes challenging. This underscores the need for a system that holds accountability for both the decision-making process and the outcomes, as much as it is important to enhance the accuracy of technology.
Rational Choices by Individual Financial Firms Pose Risks to the Entire Market
These risks can extend beyond individual financial firms to the entire financial market. If multiple financial institutions utilize similar AI models and data for investment and risk assessment, they may react simultaneously to the same signals during a crisis, leading to mass asset sell-offs or reduced lending. What may be a rational decision for individual firms can paradoxically result in credit tightening and increased volatility for the entire financial system. When risk aversion converges among financial firms, their collective choices can exacerbate market instability.
As the speed of AI's decision-making and transaction execution increases, the opportunity for human intervention diminishes. The combination of algorithmic trading, high-frequency trading, and autonomous AI agents could accelerate the spread of risks. This raises questions about whether the traditional approach of having humans assess and manage situations after a crisis occurs will be sufficient.
International financial organizations are also recognizing AI as a significant risk factor for financial stability. The Financial Stability Board (FSB) published a draft of 12 best practices for the responsible adoption of AI by financial firms in June. The Bank of England has identified AI-related risks as key pathways for financial institutions' core decision-making, financial markets, external AI service providers, and cyber threats. AI has emerged as a challenge that must be managed at the systemic level, beyond being merely a tool for operational efficiency.
Particularly, the high dependence on a few global technology companies and cloud services can create new forms of concentrated risk. Even if individual financial firms operate different products and systems, they may rely on the core AI technologies of a few companies. If a specific AI service experiences a failure or a common flaw is discovered, multiple financial institutions could be affected simultaneously. Thus, AI could serve as a conduit for both efficient financial connections and the spread of crises.
Delegating Tasks to AI Does Not Shift Responsibility
However, delaying or preventing the adoption of AI in finance is not the solution. AI can enhance productivity and strengthen the safety of the financial system through fraud detection, monitoring unusual transactions, and credit risk analysis. It also has significant potential to expand financial services into areas that have previously lacked sufficient human oversight. The key is not to hinder technological advancement but to develop control capabilities that match its pace.
To achieve this, financial firms must go beyond merely showcasing their AI adoption records. They need to assess which tasks AI is utilized for, the level of authority granted, and who can detect and halt erroneous actions when they occur. Important financial transactions should include human approval processes, and records must be maintained to track AI's judgments and actions. Establishing a system to prevent incidents and contain damage is as crucial as determining accountability after an incident.
Regulatory oversight by financial authorities must also evolve. In addition to examining capital adequacy, liquidity, internal controls, and cybersecurity, they must also assess the risks and authority management of AI models and the dependence on external technology providers. It is essential to evaluate not only the safety of individual financial firms but also the collective risks that may arise from multiple firms relying on similar AI systems. The relationship of responsibility between companies developing AI and those utilizing it in financial services must also be clarified.
Of course, it is impossible to completely eliminate all risks in advance. Concerns about security and consumer protection were significant during the rise of internet banking and mobile finance. However, the financial industry has adapted by enhancing safety measures while leveraging the benefits of technology. AI must follow this path as well. However, given that it is a technology capable of independent judgment and action, the scope of risk management and response speed is more critical than ever.
In the financial industry, trust is not merely a corporate image; it is the fundamental foundation that allows customers to entrust their money and enables market participants to transact with one another. While it is important for financial firms to reduce costs and increase profits through AI, if they lose the trust that customer assets and information are securely protected, such efficiency will not last long. In an era where AI determines the competitiveness of finance, the values of trust and responsibility, which are at the core of finance, will only grow in importance.
Now, the questions that financial firms must ask themselves must also change. It is no longer about how much work AI can take over from humans, but whether they are prepared to manage the risks that arise in the process. As AI becomes smarter, the safety mechanisms in finance must evolve alongside it. Financial firms can delegate judgment and tasks to AI, but they cannot transfer responsibility for the outcomes.
The competitiveness of finance in the AI era depends not on how many tasks can be assigned to machines, but on how effectively those tasks can be controlled.
* This article has been translated by AI.
Copyright ⓒ Aju Press All rights reserved.