What the LoRA paper does not say

Low rank adaptation is one of the clearest papers in the fine-tuning literature. The method is four lines of code and the motivation is honest. What it leaves almost entirely to folklore is where to put the adapters.

What the paper establishes well

• Weight updates during fine-tuning have low intrinsic rank, so representing them with two thin matrices loses little.
• The adapters can be folded back into the base weights at inference, so there is no serving penalty.
• Memory during training drops sharply, because optimizer state is only kept for the adapters.

What it leaves open

The experiments adapt the attention projections and largely leave the feed forward blocks alone. The paper is upfront that this was a choice rather than a finding, but the choice has been inherited as though it were settled.

In practice placement has mattered more to me than rank. Doubling rank is a predictable, mild improvement. Moving adapters onto the feed forward layers can be the difference between a run that works and one that does not, and the paper gives no guidance for predicting which case you are in.

The question I would want answered

Is there anything about a task that predicts, before training, which modules need adapting? Right now the answer is a sweep, which is exactly the cost the method was supposed to avoid.