In the current AI landscape, training large-scale models such as transformers and language models has become indispensable. However, these models, with their billions of parameters, place enormous demands on computational power, memory, and energy consumption. For instance, OpenAI's GPT-3 model, with its 175 billion parameters, requires several weeks of training and a significant amount of GPU resources. The high costs not only limit the accessibility of the technology but also raise concerns about energy efficiency and environmental impact. Therefore, finding efficient and sustainable methods for AI training is crucial.
The inefficiency in traditional large-scale model training primarily stems from the reliance on dense matrices, leading to substantial memory and computational requirements. Although techniques like matrix factorization and heuristic rank reduction have been proposed to address this issue, their practical effectiveness is limited. For example, GaLore, which uses a single-batch setting for training, incurs impractical runtime overheads, while LTE struggles with convergence in large-scale tasks. Clearly, there is an urgent need for a method that can reduce memory usage, computational costs, and training time without sacrificing performance.
Recently, researchers from the University at Albany, UC Santa Barbara, Amazon Alexa AI, and Meta have introduced a new framework called CoMERA (Computation and Memory Efficient Training via Rank-Adaptive Tensor Optimization). This framework combines memory efficiency and computational speed through rank-adaptive tensor compression, revolutionizing AI training.
Unlike traditional methods that focus solely on compression, CoMERA employs a multi-objective optimization approach to balance compression ratio and model accuracy. It leverages quantized embeddings and advanced tensor network contraction techniques to optimize GPU utilization, significantly reducing runtime overhead while maintaining excellent performance. Additionally, CoMERA incorporates CUDA Graph technology to minimize kernel launch latency in GPU operations, overcoming a critical bottleneck in conventional tensor compression methods.
The core of CoMERA lies in its adaptive tensor representation, which allows model layers to dynamically adjust their ranks based on resource constraints. By adjusting tensor ranks, the framework achieves compression without compromising the integrity of neural network operations. This dynamic optimization process is divided into two stages: an early stage focused on stable convergence and a later stage fine-tuning the ranks to meet specific compression goals.
Experimental results show that CoMERA performs exceptionally well across multiple models and tasks. In a six-encoder transformer model, CoMERA achieved a compression ratio of 43x in the early stage, optimized to 361x in the later stage. Compared to GaLore, it reduced memory consumption by 9x and increased training speed per epoch by 2-3x. In a transformer model trained on the MNLI dataset, CoMERA reduced the model size from 256MB to just 3.2MB while maintaining accuracy. In large-scale recommendation systems like DLRM, CoMERA achieved up to 99x model compression and a 7x reduction in peak memory usage. Furthermore, in the pre-training of large language models like CodeBERT, CoMERA demonstrated a 4.23x overall compression ratio and a 2x speedup in certain training stages.
The key conclusions of this research include: CoMERA achieved up to 361x compression in specific layers and 99x in full models, significantly reducing storage and memory requirements; the framework provides 2-3x faster training time per epoch for transformers and recommendation systems, saving computational resources and time; using quantized representations and CUDA Graph technology, CoMERA reduced peak memory consumption by 7x, making training on smaller GPUs feasible; CoMERA supports various architectures, including transformers and large language models, while maintaining or improving accuracy; by lowering the energy and resource demands of training, CoMERA promotes more sustainable AI practices, making cutting-edge models accessible to a broader audience.
The CoMERA framework successfully addresses the critical barriers of AI scalability and accessibility by enabling faster, memory-efficient training. Its adaptive optimization capabilities and compatibility with modern hardware make it an attractive choice, especially for organizations looking to train large models without incurring prohibitive costs. The results of this research pave the way for further exploration of tensor-based optimization in distributed computing and resource-constrained edge devices, injecting new vitality into the future development of AI technology.
