Compress a full KV cache into "weights" in one forward pass, filling the intermediate memory layer that sits between lossless cache and fine-tuning.
The Memory Problem, Baseten | Compile 26
13 min total·Actually worth watching closely: ~4 min·3 must-watch clips
- 0:00 – 2:30Listen
The memory problem: the gap between two poles
Opens from long horizon agentic use cases: a full KV cache is relatively lossless but scales linearly in memory, while fine tuning, RAG and writing markdown files compress too hard. The layer in between is missing.
A reliable long horizon agent needs three things: compressed memory in the KV cache space, memory that can be used in the loop, and a mechanism to project memory back into the weights.
A purely spoken problem statement and motivation, no key visuals — fine to take in like a podcast.▶ Jump to 0:00 - 2:30 – 5:34Skim
The method landscape: the 2x2 and where related work sits
Classifies KV cache compression along selection vs synthesis and per context vs amortized, placing cartridges (per-context synthesis), pre-training-time methods, and this work (amortized + synthesis) in their quadrants.
This method sits in the amortized + synthesis quadrant: pay the training cost upfront, then compress in a single forward pass at inference time.
The core content is on the 2x2 slide (visual moments at 157s and 206s); one glance gets you the framework, so the commentary can be played fast.▶ Jump to 2:30 - 5:38 – 7:40Skim
The compactor architecture: amortization and cross-attention
Borrows the amortization idea from sparse autoencoders to train a black-box compactor. Since the KV cache holds a variable number of tokens, a small fixed set of learned queries cross-attends to every token — keys and values concatenated, then projected — to produce the compressed representation.
Rather than running an expensive inference-time optimization for each context, learn a function that compresses the KV cache in one forward pass.
There's an architecture slide at 384s; the diagram makes the cross-attention data flow faster to follow than audio alone. The rest can be skimmed.▶ Jump to 5:38 - 7:40 – 8:58Listen
Training details: identity initialization and KL distillation
Two key decisions: the output projection is initialized to the identity, so training starts from an information-preserving attention-weighted mixture; and the objective is the KL between the output distributions under the full and the compressed KV cache.
The KL distillation loss forces the model to behave the same before and after compression — the test of compression is whether what the model says changes.
Spoken training detail with no separate visual moment; both points come through by ear.▶ Jump to 7:40 - 9:01 – 10:55Watch
Why iterative compaction collapses, and how to fix it
The naive compress → generate a new block → compress loop starts to collapse after a few rounds when trained on single-pass compaction. The fix is to skip the immediately next block and compute KL divergence between full and compacted context on the subsequent block.
That one skip-block KL trick keeps 16 or even 32 iterations of compaction at quite high accuracy — the key to long-horizon usability.
Two visual moments land here, 555s and 602s: the collapse and the skip-block fix side by side are the most direct evidence for the trick — worth watching the board.▶ Jump to 9:01 - 10:55 – 12:15Watch
The insight: compaction is constructing weights
Points out that a compacted KV cache is structurally an MLP — query times keys is the up projection, softmax is the non-linearity, times values is the down projection — so compaction constructs weights directly from the context.
This is a new form of producing weights, distinct from gradient descent, with the potential to bring in-context learning's sample efficiency into durable knowledge.
The structural correspondence at 672s only lands if you follow it term by term on the board — the densest stretch in the talk.▶ Jump to 10:55 - 12:15 – 13:12Listen
Looking ahead: combining compaction with gradient descent
Closes on the future direction: use compaction to produce a block of weights in the first place, then also do gradient descent into that same block, for sample-efficient learning.
The endgame for memory is two weight-producing mechanisms working together — compaction for sample-efficient absorption, gradient descent into the same block for the rest.
Spoken outlook and wrap-up, no new visuals — good for tying the talk together as you finish.▶ Jump to 12:15