Entropy-reduction physics explains why token maxing keeps costing more for less: context is the other half of efficiency the whole industry skipped.
Intelligence Efficiency, Ben Geist | Compile 26 · Ben Geist
15 min total·Actually worth watching closely: ~5 min·3 must-watch clips
- 0:00 – 1:45Skim
You're buying tokens, not intelligence
Opens with the core contradiction: Ramp's data shows monthly token spend up 13x since January 2025, but returns on intelligence follow a logarithmic curve, and Uber and Meta have already started clamping down on token maxing.
Paying for intelligence is really paying for tokens; the two are not equal, and exponential consumption isn't economically sustainable
The spend-versus-intelligence curves (around 48s and 70s) are worth a glance; the rest is spoken setup▶ Jump to 0:00Speaker · Ben Geist - 1:45 – 4:10Skim
The LLM as an entropy reduction machine
Combines Shannon entropy with the thermodynamic view: how far a system's entropy can be reduced is bounded both by the work you put in and by the information you already have — like cleaning your room, you have to know where the dirty clothes are before the work pays off.
Entropy reduction has two levers, work and information; the chart at 163s names the goal — bend the intelligence curve from logarithmic toward linear
The entropy frame is the key to following the second half; the chart at 163s is worth watching, and the equations can be skimmed off the slides▶ Jump to 1:45Speaker · Ben Geist - 4:10 – 6:11Listen
The industry bet on only one half: work
Points out that today's advances — chain of thought, reasoning — sit almost entirely on the "more work" side, while the information side's mainstream tools, RAG and memory, don't scale well, are used for needle-in-a-haystack semantic search, and stay stuck in token space rather than latent space.
The efficiency bottleneck is the neglected information/context half, and existing methods never reach latent space
Pure argument with no key visuals — fine to listen to like a podcast▶ Jump to 4:10Speaker · Ben Geist - 6:14 – 9:20Watch
Experiment 1: a shared KV cache across agents
The classic supervisor-worker architecture rediscovers the same context over and over and burns an astronomical number of tokens; the fix is a globally shared KV cache, with each newly spawned worker initialized from task-relevant context compressed out of it.
At unchanged accuracy, worker tokens drop 42–57% and total tokens 21–31%
The architecture comparison drawn from 396s is the key to seeing where the waste is and how the cache is shared — visually dense▶ Jump to 6:14Speaker · Ben Geist - 9:20 – 11:20Skim
Experiment 2: sparse attention as a reranker
Models sparse attention as a reranker: document scores computed from the queries and keys of DeepSeek's lightning indexer (ReLU of the normed cosine scores summed over heads, averaged over tokens within a document), benchmarked on a multi-hop question-answer dataset.
Sparse attention matches or beats SOTA reranker models — its efficiency comes down to accurate context retrieval over its own token space
The scoring equations and comparison table at 595s are worth pausing on; the narration just follows the slides▶ Jump to 9:20Speaker · Ben Geist - 11:20 – 13:17Listen
Experiment 3: memory as its own modality
Work with Stanford's SNAP lab treating context injection as a modality, the way text, image and video are: a memory module takes docs plus a query and compresses them into 16 latent representations injected straight into a frozen Qwen 8B, with no change to the base model.
Context can bypass token space entirely and enter a frozen LLM as latent representations — the most radical rework of the information side
Mostly spoken method description; catching the pipeline is enough, and the results chart comes at the next stop▶ Jump to 11:20Speaker · Ben Geist - 13:17 – 14:34Watch
372x compression and three criteria for adoption
The memory module beats RAG-50's 55% exact match on TriviaQA while using 372 times fewer representations in the input; the close proposes three criteria for an ideal context-injection model: low latency, scaling over large corpuses, and essentially zero switching costs to the base model.
The three criteria are a selection checklist: usable at inference time, scalable, and reusable when the next base model ships without retraining
The results chart at 798s carries the talk's conclusion, and the three-criteria slide is worth a screenshot▶ Jump to 13:17Speaker · Ben Geist