Question

In Parameter-Efficient Fine-Tuning (PEFT) using LoRA, what mathematical operation is applied to the frozen pre-trained weight matrix to adapt it to a new domain?

Accepted Answer

In LoRA, the adaptation of the frozen pre-trained weight matrix is performed through an additive operation. Specifically, the pre-trained weight matrix, denoted as W, remains frozen, meaning its values are never updated during fine-tuning. A new path is created alongside W consisting of two smaller, low-rank matrices called A and B. Matrix A has dimensions r by d, and matrix B has dimensions d by r, where d represents the original weight dimensions and r is a small rank hyperparameter significantly smaller than d. During training, the product of these two matrices, A multiplied by B, is calculated to form an update matrix of the same dimensions as W. This result is then added to the original frozen weight matrix. The final output for an input x is computed by taking the sum of the result of the frozen weight matrix applied to x and the result of the low-rank update matrices applied to x, expressed mathematically as Wx + BAx. By constraining the update to the product of two low-rank matrices, LoRA dramatically reduces the number of trainable parameters compared to full fine-tuning, while the addition operation ensures the model effectively incorporates the new domain-specific knowledge.

Home → All Courses → Engineering and Technology Courses → Natural Language Processing Engineering → Flashcard

In Parameter-Efficient Fine-Tuning (PEFT) using LoRA, what mathematical operation is applied to the frozen pre-trained weight matrix to adapt it to a new domain?