Accuracy falls to 13.6% at 741 tools: measured numbers on how tools should be selected
The 100-Tool Agent Is a Trap - Sohail Shaikh & Ankush Rastogi, Prosodica · Sohail Shaikh
28 min total·Actually worth watching closely: ~5 min·2 must-watch clips
- 0:01 – 4:12Skim
How the problem grows
Starts from the simplest approach: give the model every tool definition on every request — every function name, every description, every JSON schema. No problem at around ten tools; once the catalog grows, the agent starts calling the wrong function, confusing similar tools, even inventing tool names that do not exist.
The design does not fail because one tool is badly written — it fails because every request is forced to carry the entire catalog.
This stretch is text slides plus a set of token-overhead numbers; the slide around 03:35 that puts "741 tools, about 127,000 tokens just for the descriptions — and that is before the user's actual question" on one screen is worth a pause. The rest can simply play.▶ Jump to 0:01Speaker · Sohail Shaikh - 4:15 – 7:30Skim
Why accuracy collapses
Breaks the collapse down: irrelevant tools competing for attention inside the choice set, the lost-in-the-middle problem where hundreds of schemas packed into the middle of a long context are not reliably used, and tools with similar descriptions interfering with each other. The worsening cost and latency curves are laid out alongside.
Accuracy does not decay smoothly — it collapses past a certain scale, and cost and latency worsen at the same time from the same cause.
The whole segment is the failure reasons listed on a slide and then narrated, with the speaker pointing repeatedly at the items on screen. Reading the list is faster than sitting through it; the examples can play in the background.▶ Jump to 4:15Speaker · Sohail Shaikh - 7:33 – 11:07Watch
A different architecture: retrieval for tools
Puts the monolithic full-injection design and the layered retrieval-based design side by side, then explains the mechanism — offline, write a clear description for each tool and embed it into a store; at runtime, process the user's question the same way and pull back the closest few tools.
This is essentially the retrieval-augmented pattern applied to tools, so teams that already run vector retrieval need to build almost nothing new.
The side-by-side architecture diagrams are the core message here, and the layering is hard to reconstruct from audio alone; from about 09:35 the speaker walks the offline index build and the runtime retrieval path level by level.▶ Jump to 7:33Speaker · Ankush Rastogi - 11:10 – 14:15Listen
Just-in-time injection and the subtraction effect
Covers injection on demand: only the three to five retrieved schemas go to the model, cutting the token overhead from six figures to about a thousand. Plus an often-missed benefit — irrelevant tools disappear from the model's options entirely.
Half the value of routing is handing over the right tool; the other half is making sure the wrong ones never appear as options at all.
This stretch is essentially pure narration, with nothing new worth looking at on screen — you lose nothing listening to it while doing something else.▶ Jump to 11:10Speaker · Ankush Rastogi - 14:15 – 17:30Skim
How the benchmark was run and what it showed
After the evaluation setup, the head-to-head: full-injection accuracy falls from around 78% to just over 13% as the catalog grows, while retrieval-based injection holds above 83%; the wait for the first token climbs steeply on the first path and stays almost flat on the second.
The catalog can keep growing, but the working set the model actually faces each time has to stay very small.
The value is concentrated in the two comparison charts — read the direction of the curves and the axis scales and you have it; the spoken detail about the evaluation process is low density.▶ Jump to 14:15Speaker · Ankush Rastogi - 17:33 – 23:25Skim
Writing the code and rolling it out
Walks the implementation section by section: loop through the catalog and embed the descriptions into the vector store, embed the query at runtime and run the similarity search, then pass only the retrieved tools to the model. Then a worked example and a seven-step rollout checklist, including the advice to start with five candidates and how to evaluate.
The change lands in one place — the tool list passed to the model goes from the full catalog to the retrieval result. For most teams that is a sprint of work, not a platform rewrite.
The screen shows code snippets and a step list, so paging through the key lines beats listening straight through; if you plan to implement it, read the few code screens around 18:00 line by line.▶ Jump to 17:33Speaker · Sohail Shaikh - 23:30 – 28:20Skim
Risks, adoption threshold and resources
Two main risks — the router failing to retrieve the tool the model needs, and descriptions written so weakly the tool never gets selected — with countermeasures: widen K, run a second retrieval pass, or route to a broader tool group. Closes with the adoption threshold and reference material.
At ten or twenty tools do not over-design; bring in a router once you pass fifty, or once confusion and latency have already become real problems.
Risks and countermeasures are a bulleted list, so skim it; after 27:20 there is a page of reference links and speaker contacts — screenshot it if you want it, otherwise you are done.▶ Jump to 23:30Speaker · Sohail Shaikh