Five minutes on an edge case no RAG tutorial covers: what to do when all of your documents are relevant and the whole dataset gets replaced constantly.
When All Context Matters: Extended Cache Augmented Generation - Luis Romero-Sevilla, Orbis · Luis Romero-Sevilla
6 min total·Actually worth watching closely: ~1 min·1 must-watch clip
- 0:00 – 0:45Listen
Problem setup: everything is relevant, everything is replaced
The opening frames one very specific retrieval scenario: every document in the collection is relevant to answering the user's questions, and the documents go obsolete fast and are all replaced with new information — where the usual approaches fall over one by one.
This isn't a general RAG problem but a doubly constrained edge case — get the premise and every trade-off that follows makes sense.
No visual moments anywhere in the talk are worth pausing on; this stretch is a spoken problem statement, so just catch the premise.▶ Jump to 0:00Speaker · Luis Romero-Sevilla - 0:45 – 1:45Listen
Vector RAG: incomplete recall, but fast swaps
Analyzes the weakness of a simple vector RAG: it only retrieves the vectors that fall within the similarity threshold, which can't cover a need where everything is relevant; but inserts are fast, so a stale collection can simply be replaced with a new one.
Vector RAG's real advantage when updates are frequent isn't retrieval quality — it's that the collection can be swapped wholesale.
An expository comparison with no demo or key diagram; good for noting the pros and cons as you listen.▶ Jump to 0:45Speaker · Luis Romero-Sevilla - 1:45 – 2:35Listen
GraphRAG: answers well, builds slowly
GraphRAG uses an LLM to read through the whole collection and build a knowledge graph, and is good at synthesizing an answer drawn across all of it — but every wholesale data replacement means rebuilding the graph, which is expensive and slow.
GraphRAG only fits collections that barely change — rebuild cost is what kills it in this scenario.
No flagged visual moments; the weight is on the spoken argument about rebuild cost.▶ Jump to 1:45Speaker · Luis Romero-Sevilla - 2:35 – 3:15Listen
CAG and the context window ceiling
CAG loads all the documents into a large context window and caches the KV matrix, doing away with retrieval; but the window is limited, and filling it too full actually degrades answer quality.
CAG's bottleneck isn't only window size but 'fill it up and quality drops' — which leads straight into the sharding idea that follows.
A conceptual setup stretch with no visual dependency; grasp the KV caching mechanism and you can follow the next section.▶ Jump to 2:35Speaker · Luis Romero-Sevilla - 3:15 – 4:40Listen
eCAG: parallel vessels plus a supervisor model
Introduces eCAG: documents spread evenly across multiple parallel CAGs in different context vessels, each answering questions about its own content, with a smaller supervisor model interrogating each vessel, progressively building its internal understanding and synthesizing the final answer. Because all the caches load in parallel, knowledge building is far faster than GraphRAG, while the answers are more accurate than a simple RAG.
Don't organize the vessels by domain — with a very dense relationship between documents the supervisor ignores domains that look irrelevant at first glance, so an unordered, balanced distribution works better.
The core of the talk, but no live demo or must-see visual is flagged; the architecture is carried by speech alone, so focus on the interaction between the vessels and the supervisor.▶ Jump to 3:15Speaker · Luis Romero-Sevilla - 4:40 – 5:40Listen
An honest word on cost, and a 'no silver bullet' ending
Admits the KV cache can be quite expensive and can be eased by optimizing how much each vessel's cache holds on to; wraps up by noting that retrieval strategies all have downsides in compute, cost or speed, and that eCAG is only a custom answer to one very specific problem.
There is no solution that fits everything — pick your architecture along two axes: how often the data is replaced, and what share of the documents are relevant.
A concluding spoken argument with no diagram dependency; worth writing down as a rule of thumb for architecture choice.▶ Jump to 4:40Speaker · Luis Romero-Sevilla