LoRA Without Regret
This research provides a detailed analysis of Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning (PEFT) method for large language models, comparing its performance against full fine-tuning (FullFT). The authors establish a "low-regret regime" where LoRA matches the performance and sample efficiency of FullFT, particularly for small-to-medium-sized datasets, provided key implementation details are correct. Operational benefits of LoRA, such as improved multi-tenant serving, reduced training memory footprint, and easier transferability, are highlighted as reasons for its growing popularity. The research emphasizes that for optimal performance, LoRA must be applied to all model layers, especially the MLP/MoE layers, and that its optimal learning rate is consistently about ten times higher than for FullFT. Finally, the analysis shows LoRA's significant advantage in reinforcement learning scenarios due to the inherently low information capacity required for such tasks, and discusses its computational efficiency advantage, requiring slightly more than two-thirds of the FLOPs of FullFT per training pass.