RAG running live on an all-free local stack, proving clean chunking decides more than a big model
Bypassing the Multimodal Tax: Hybrid RAG, SQL RRF & UI Telemetry - Abed Matini, Ogilvy · Abed Matini
46 min total·Actually worth watching closely: ~26 min·3 must-watch clips
- 0:01 – 3:40Watch
Framing the problem, live demo on screen
He opens with the right half of the screen given over to an HR employee-handbook Q&A assistant running locally, demoing it as he speaks. Then the problem: hand a whole document to a cloud LLM and you pay tokens you did not need to spend, and you cannot see how it was chunked inside.
Handing documents to a cloud model costs more than money; it costs your control over chunking quality.
The whole stretch is delivered against that live Q&A interface and its admin backend on the right, so audio alone loses track of which step he is pointing at.▶ Jump to 0:01Speaker · Abed Matini - 3:43 – 6:55Skim
The all-local, CPU-only stack
He lists the components one by one: FastAPI and React for backend and frontend, Postgres for the vectors, Ollama for the local models, Langfuse for observability, all open source with no paid dependency, and a plain CPU is enough to run it.
A complete production RAG path can contain no paid service at all, and the hardware bar is just an ordinary machine.
This is essentially one stack slide read out point by point; visual density is low, so skim the component names and move on.▶ Jump to 3:43Speaker · Abed Matini - 6:59 – 11:35Skim
The structure-first ingestion pipeline
The end-to-end blueprint: PDFs, PowerPoints and images are converted to Markdown locally, cleaned, then chunked into the store, and user questions are retrieved only from your own store. He finishes by switching to the admin to show what the ingested documents look like.
Converting to structured text before chunking is the precondition that lets this architecture avoid outside dependencies.
The first four minutes or so are a static architecture diagram with narration, so listening is enough; around 10:50 he switches to the backend and starts clicking through real documents, which is worth slowing down for.▶ Jump to 6:59Speaker · Abed Matini - 11:39 – 18:37Watch
The chunking comparison
First, what chunking the whole 28-page handbook produces: acknowledgement pages, signature blocks and other meaningless fragments flooding in. Then the alternative, turning it into question-and-answer pairs and chunking on headings, with the same question asked of both stores and the answers and cited sources compared.
Once answers carry their source chunks, correctness becomes verifiable and debuggable; once chunks get too big you cannot even find where you went wrong.
It is hands-on work switching between two interfaces throughout, with the answer text and the source chunk it cites shown side by side; the gap between the two strategies is carried entirely by the screen.▶ Jump to 11:39Speaker · Abed Matini - 18:41 – 22:05Watch
The other three chunking strategies
He demos paragraph chunking, fixed 512-character chunks with 64 characters of overlap, and sentence-group chunking in turn, explaining that the overlap is what keeps neighbouring chunks from losing context.
Fixed-length chunking is not the optimum, but for disorganised data you have no time to clean it remains the pragmatic fallback.
He pulls the fixed-length chunks straight up on screen and you can see one break in the middle of a word; the flaw is abstract when described and obvious at a glance.▶ Jump to 18:41Speaker · Abed Matini - 22:08 – 29:10Listen
Images into the knowledge base, and function-only agents
How a screenshot gets into the knowledge base through an image-to-text model, which suits short-lived information such as a maintenance notice going up and coming down quickly. Then the most counterintuitive claim of the talk: the agents in this system are ordinary Python functions, with no LLM call at all.
If deterministic code can do the step, do not hand it to a model: it is faster, hallucination-free, and you can write tests for it.
This stretch is mostly argument, with the screen parked on a slide and no live operation, so following his reasoning is enough.▶ Jump to 22:08Speaker · Abed Matini - 29:11 – 33:52Listen
Hybrid vector plus keyword retrieval
Why semantic search alone falls short: it only ever gives you what is close in meaning. Product numbers, prices and medication names have to hit exactly, so keyword and vector results are fused and ranked together, and the number of retrievals is tuned to the business.
Any use case where users type an exact string will lose results to pure semantic search; the keyword path has to be there.
The whole section is spoken reasoning about retrieval with no matching demo, so you lose nothing by listening while doing something else.▶ Jump to 29:11Speaker · Abed Matini - 33:56 – 45:46Watch
Guardrails, observability, and a small-model finish
Guardrails go in the code and stop the request before it reaches the model: medical questions return a hand-off message, injection patterns are blocked before anything is sent, and the system prompt stays minimal. Then the locally deployed observability dashboard, and finally back to the admin to answer live with a 0.5B model.
Get both the data and the request under control before they reach the model, and a 0.5B model can handle the rest.
Around 35:30 he switches to the observability dashboard, showing per-session questions, how many chunks were hit and millisecond latency; after 42:25 he is back in the admin running real Q&A. In both places the screen itself is the evidence.▶ Jump to 33:56Speaker · Abed Matini