LARA: Lightweight Adapters in the Residual Stream for Composable Adaptation and Alignment
Pascal Ekin, Hyosun Choi, Wei Jie
- Digest date
- 2026-08-03
- Submitted
- 2026-07-25
- arXiv ID
- 2607.28669
Summary
LARA adapts models by reading hidden states at select layers and adding a low-rank correction back into the residual stream, rather than modifying weights like LoRA. This exposes an inference-time interpolation scale between base and adapted behavior, and allows many behaviors (7 tested, ~33MB overhead) to reside simultaneously on one frozen 1.5B model with per-token routing.
Abstract
We present LARA (Lightweight Additive Residual Adaptation), a method for efficient adaptation that operates in the residual stream of a frozen model rather than in its weights. Where LoRA adds an update of low rank to weight matrices, LARA reads the hidden state at a small set of layers and adds a correction of low rank back to the residual stream, leaving all base weights untouched. On a code fine-tuning task and on preference optimization (DPO), LARA matches LoRA at equal parameter counts. Because adaptation is a frozen base plus a residual, LARA exposes a scale {\gamma}, applied at inference, that interpolates smoothly between base and adapted behavior, a form of graded control that adaptation in weight space does not offer. Finally, because each behavior is a small residual module over a shared frozen base, many behaviors can be held resident at once and routed automatically per token. We place seven behaviors, six fine-tuned and one optimized for preference, on one frozen 1.5B model for roughly 33 MB of overhead, against one full model for each behavior. Because the base is untouched, behaviors are trained separately and selected per token rather than loaded on demand, which suits hosting many behaviors, and adding new ones, on a single model on a device.