Same model, only the search tool swapped, and it nearly hits the ceiling - the bottleneck is retrieval, not reasoning
How we taught agents to use good retrieval - Hanna Lichtenberg, Mixedbread AI · Amir
14 min total·Actually worth watching closely: ~6 min·3 must-watch clips
- 0:00 – 1:35Listen
The knowledge gap nobody is watching
The opening thesis: model reasoning capability has climbed exponentially in recent years while search and retrieval have barely moved in twenty, and a gap has opened between the two. Retrieval tools are the one main access pattern for the reasoning layer to reach the right knowledge.
What decides whether a model is useful in knowledge-heavy work like legal and finance isn't how well it thinks, but whether it can get to the right material.
Pure spoken setup, with the slide parked on the title page - fine to listen to while doing something else.▶ Jump to 0:00 - 1:35 – 3:35Skim
Two benchmarks pin the bottleneck down
A controlled comparison on BrowseComp Plus (a fixed corpus of 100,000 documents) and OfficeQA Pro (built on 100 years of US treasuries): first measure the ceiling when the right documents are put in front of the model, then switch to default retrieval tools searching the noisy corpus.
With default tools, answer quality drops 8-9 points. The model didn't get dumber, it just can't find things - which isolates the bottleneck cleanly on the retrieval side.
The conclusion takes a couple of sentences, but it rests on score comparison charts; glance at the size of the drop and keep listening rather than picking apart every number.▶ Jump to 1:35 - 3:35 – 5:00Watch
Swap the search tool and it comes back
Replacing the default tools with retrieval built on latent interaction lifts the scores right back - three points from the oracle on one benchmark, almost fully matched on the other. Then comes a look at what the queries the model actually writes look like.
Same model, same corpus, the only variable is the search tool, and the gap closes. Meanwhile the queries the model emits are keyword fragments that leave a semantic search system expecting natural language confused.
The verbatim query text on screen is the core evidence here; you have to read the actual words to feel what makes it gibberish - skip it and all that's left is an abstraction.▶ Jump to 3:35 - 5:00 – 7:01Listen
Why the model never learned to ask
Three causes behind bad queries: training for coding agents bred a regular-expression grabbing habit; models mimic the way humans type keywords into web search boxes; and mainstream retrieval benchmarks lean on short entity-based queries that structurally favor keyword matching. Then a prompting trick you can use on the spot.
Don't ask the model to write a search query - ask it to write one concise sentence describing what it wants to find. That single change sidesteps its old keyword habit.
All causal reasoning plus one line of prompt you can copy verbatim; nothing to watch, just catch the rewording.▶ Jump to 5:00 - 7:01 – 9:20Watch
A harness with four tools, each with its own job
The speakers switch and move into the solution: the search agent's tool lineup - one for overview summaries, one returning ten semantic search results, one filtering by metadata facets, one for exact matching - with deliberately short rounds and several queries fired in parallel each round.
Semantic search explores the different aspects of a question, exact matching does only what it's good at; splitting the two apart instead of blending them into one tool is what makes queries natural in the first place.
The tool split and the call flow are drawn out on the slide, and the speaker points at the example on the right several times; audio alone tends to blur the boundaries between the four tools.▶ Jump to 7:01 - 9:20 – 11:30Skim
How to induce good queries, and how to train them
First the five harness-level designs - articulate what evidence is needed, separate the tools' responsibilities, reframe the ask as a task, show a few good queries and how to divide a query into aspects, and run an initial search so the agent can see the corpus's language. Then training: deliberately a small model for speed, supervised fine-tuning with a larger teacher, then on-policy reinforcement learning with their own search reward.
The reward splits in two - a retrieval reward on ranking metrics and a judge's relevance scoring, and a trajectory reward specifically on whether the query reads as a natural sentence and whether the amount of exploration is right.
Dense, but presented as bulleted list slides; follow the headings for the structure and stop on the reward breakdown screen.▶ Jump to 9:20 - 11:30 – 13:10Watch
Post-training behavior and interim scores
A walkthrough of a real post-training trajectory to check that the behavior matches the design, then interim results from the version that hasn't been released yet.
The ranking metric reaches 0.4 on that benchmark, against 0.18 for the best-performing multi-hop agent in the original paper - more than double.
What each tool received is laid out line by line on screen, and it's the first-hand material for judging whether the method actually works; the score comparison in the second half is also a chart.▶ Jump to 11:30 - 13:10 – 14:26Skim
Production numbers and wrap-up
The production beta, used as the search tool for Gemini 3.5 Flash, ranks top one on the Snowflake MetQA benchmark with an accuracy of 93.4 while spending noticeably less than other combinations - verifiable yourself on the public leaderboard. It closes on how much room retrieval still has to improve.
Better retrieval isn't trading cost for accuracy: this combination is both more accurate and cheaper.
The key information is the rank and that accuracy number on the leaderboard screenshot; take one look to confirm, then the closing twenty seconds can just play.▶ Jump to 13:10