KAIST researchers have developed a technology that reduces additional training time for AI in specific fields such as healthcare and coding by up to 69.6%. This advancement allows training that previously took 10 hours to be completed in approximately 3 hours by eliminating redundant data and evenly distributing diverse data across multiple GPUs.
On November 2, KAIST announced that a research team led by Professor Lee Jae-gil from the Department of Computer Science has developed a fine-tuning technology for large language models called CluSTER. This technology groups similar data that have comparable impacts on learning and distributes them evenly across multiple GPUs.
Fine-tuning is the process of teaching an AI, which already possesses broad knowledge, to specialize in specific areas such as coding problems and medical inquiries. This process requires significant computational resources, necessitating the use of multiple GPUs, which are semiconductors that perform calculations quickly for AI training. Randomly dividing data can lead to multiple GPUs processing similar data repeatedly, resulting in diminished learning efficiency as they continue to solve problems that are alike.
CluSTER first analyzes the impact of each data point on AI learning and groups similar data together. It does not rely solely on shared topics or similar vocabulary. Even if problems appear different on the surface, they are classified together if they teach the AI similar problem-solving skills. For example, teaching a student addition through problems involving 'two apples and three apples' and 'two pencils and three pencils' follows the same principle, despite the different contexts.
After grouping, the technology selects data from each cluster and distributes it so that multiple GPUs can process diverse learning information evenly. This method reduces the number of similar problems while ensuring that essential problem types are retained for learning. The approach not only decreases the volume of data but also enhances the efficient use of selected data across multiple GPUs.
The research team fine-tuned large language models with approximately 7 billion and 13 billion parameters using three types of coding and medical training data, confirming effectiveness through eight performance evaluation tests. Despite selecting only about 31% of the total data for training, the model's performance remained nearly unchanged, while training time was reduced by up to 69.6%. The experiments were primarily conducted in an environment with four GPUs, and the technology also demonstrated superior performance in an eight-GPU setup compared to traditional methods.
This outcome shows that AI training efficiency can be improved without adding more GPUs, simply by selecting and distributing training materials effectively. The research team explained that this could reduce the time and GPU costs associated with additional training when developing AI for specific purposes such as customer support, coding assistance, and medical Q&A.
Professor Lee Jae-gil stated, “To enhance the efficiency of AI training, it is important not only to use more GPUs but also to intelligently select which data to train on. This research is significant as it demonstrates how to utilize existing GPU resources more efficiently, thereby reducing the training burden for large-scale AI models.”
* This article has been translated by AI.
Copyright ⓒ Aju Press All rights reserved.