Don't let eval signal die in the dashboard — feed it back as retrieval weight, with benchmarks and a live demo
User Signal Dies at the Retrieval Boundary - Sonam Pankaj, StarlightSearch · Sonam Pankaj
16 min total·Actually worth watching closely: ~5 min·3 must-watch clips
- 0:00 – 1:35Skim
Why agents keep tripping over the same rock
Opens by pointing at what's missing from today's agent loop, ReAct included: it reasons, calls tools, retrieves, and loops until the task is done, with nothing feeding back whether it worked — so the same class of task fails over and over. Then the industry numbers land: 85% of AI failing, and 73% of pipeline failures coming from retrieval.
The problem isn't that the model isn't smart enough — the retriever is static and never learns.
This runs over a diagram of the agent loop and a slide of quoted figures. The diagram is simple: glance at the missing feedback edge and the two percentages and you have it — the narration itself can be sped up.▶ Jump to 0:00Speaker · Sonam Pankaj - 1:35 – 4:50Listen
Eval signal dies in the dashboard
The speaker describes something everyone recognizes: observability faithfully records every tool call, every completion, every exception, and evals judge the output pass or fail — but those conclusions stop at the dashboard and never reach the agent's context. So improvement falls to an engineer watching the data, rewriting the prompt by hand, redeploying, or simply upgrading to a more expensive model.
A layer is missing between evals and action, and its absence turns improvement that could be automated into manual labor.
Conceptual argument throughout, with the screen parked on text slides — nothing you need to watch, so listening while you move around loses nothing.▶ Jump to 1:35Speaker · Sonam Pankaj - 4:50 – 7:02Listen
Utility score: letting eval outcomes drive retrieval ranking
The core proposal arrives — retrieval isn't just semantic similarity to the current task, it's weighted by whether that memory has historically helped or hurt execution, and that score re-ranks the results. The accompanying runtime layer, Agent RGX, improves the agent while it is executing the task, unlike compile-time approaches such as DSPy that bake the lessons into the prompt.
Make the eval outcome a first-class signal in retrieval re-ranking, and improvement happens at runtime — no retraining, no hand-tuned prompts.
This is the stretch where you most need to follow the reasoning: the intuition behind the ranking and the line between it and compile-time approaches. The slides are just bullet backup — this one is for listening.▶ Jump to 4:50Speaker · Sonam Pankaj - 7:02 – 8:50Skim
Memory as reasoning, baked into a skill once it's proven
Memory shouldn't hold only preferences like the user likes a dark theme; it should hold task reasoning — check the settlement before issuing a refund so the customer isn't paid twice. Once enough memories accumulate, the reasoning is baked into skills, which also cures the old ailment of a system prompt still naming a column that no longer exists and nobody updating it. The scores come with it: 66% to 76% on the policy-following benchmark, 80% with skills.
Memory stores how to think, not what was remembered; once it's validated enough, it settles into a capability.
The substance sits in two places: the one-line refund rule, and two or three slides of bare numbers. The numbers stick after one look — no need to follow sentence by sentence.▶ Jump to 7:02Speaker · Sonam Pankaj - 8:50 – 10:25Listen
Three pitfalls the speaker owns up to
Unusually, time is spent on limitations: the cold start, where it's pure semantic search until enough reviews have accumulated; utility drift, where similar memories keep surfacing; and review quality, where noisy labels make the utility itself noisy.
The mechanism depends on high-quality feedback; cold start and noisy labels are prerequisites to think through before adopting it.
All three are spoken judgments with nothing on screen to match, but they weigh heavily on whether to adopt this — worth listening through.▶ Jump to 8:50Speaker · Sonam Pankaj - 10:25 – 12:40Watch
Demo, act one: it can't find that mouse
Cuts to a live SQL product-search agent. Input: find me a gaming mouse. Zero memories retrieved, and the agent answers that it couldn't find a gaming mouse in the product catalog — while a wireless mouse really is in the database, just named differently.
Static retrieval can't cross the gap between how a user phrases it and how the inventory names it.
The query in a real terminal, the zero-hit retrieval, and the couldn't-find-it answer are on screen at once — every link in the failure is visible, and away from the picture it's just a one-line summary.▶ Jump to 10:25Speaker · Sonam Pankaj - 12:40 – 14:26Watch
Demo, act two: feedback changes it on the spot
That failure is submitted as feedback and the same question runs again: this time the agent searches wireless mouse itself and returns the product. The speaker pulls up both tool-call trajectories side by side — the same search products tool, empty-handed once and returning results the next, with not a word of code or prompt changed.
The behavior change happens at runtime in production, with no redeploy.
The most persuasive frame in the talk is right here: two trajectories side by side, the before-and-after difference visible at a glance — the one piece of empirical grounding for the whole argument.▶ Jump to 12:40Speaker · Sonam Pankaj - 14:26 – 15:36Listen
Closing: scores keep moving, and the good ones become capabilities
Ends by returning to the mechanism: memory scores keep changing as past traces accumulate, and after about five reviews the findings can be baked into a skill — updating things the agent draws heavily on without changing the system prompt at all.
Eval signal flowing back closes the loop; updating an agent's knowledge no longer depends on someone rewriting prompts.
A purely spoken wrap-up with the screen already on the closing slide — hearing this last loop-closing point is enough to take the through-line with you.▶ Jump to 14:26Speaker · Sonam Pankaj