After 500,000 sensor names confused the LLM, a tree-structured rebuild won back 100% accuracy and a 300x cost cut
Semantic Blindness: 500,000 Sensors Confused an LLM - Raahul Singh & Vanč Levstik, Phaidra
16 min total·Actually worth watching closely: ~3 min·3 must-watch clips
- 0:00 – 2:10Listen
Semantic blindness: where the problem comes from
Phaidra's AI-factory operations agents break down at hundreds of thousands of devices: feeding about 500,000 messily named sensor names straight to the LLM works at demo scale, but a gigawatt-scale factory quickly exhausts the context window.
A product must work for every scenario without failing silently; a demo only needs one scenario to work — the essential gap between demo and production
Purely verbal problem setup with no key visuals — fine to listen to like a podcast▶ Jump to 0:00 - 2:10 – 5:41Listen
Autopsy of three failed paths
Walks through RAG/vector retrieval (which cannot tell chiller 6 from chiller 7), runs of similar tokens tripping the frequency penalty and cutting output short, and naive divide-and-conquer making the LLM invent phantom equipment while silently dropping real ones.
Vector similarity is helpless against equipment names that differ by one character, and silently dropping data directly erodes user trust in mission-critical systems
Mostly spoken post-mortem with no frames worth capturing; the point is hearing exactly why each path died▶ Jump to 2:10 - 5:41 – 7:01Skim
The key insight: a factory is a tree
AI factories are naturally tree-shaped hierarchies (data center → data hall → aisle → rack → GPU) whose depth grows very slowly while width explodes; the 'linearizer' describes to the LLM only the roughly four layers from root to leaf, not the full million nodes.
Cost should scale with hierarchy depth, not with equipment instances — this structural property is the fulcrum of the whole approach's sub-linear scaling
Both visual moments cluster on this segment's hierarchy and linearizer slides; a glance at the diagrams captures the structure, with the details carried by the narration▶ Jump to 5:41 - 7:01 – 10:31Skim
The LLM only plans — it never reads names
The LLM outputs a structured query plan of three elements — collect, scope, filter — and a deterministic backend resolves it with pre-indexed trees and set operations; for extremely vague queries, the LLM generates patterns to look for instead.
Recall and accuracy stay perfect under arbitrarily fuzzy queries, because traversal and matching are handed entirely back to deterministic code
The query-plan breakdown at 426s is the design core of the whole talk; the collect/scope/filter structure on the slide only truly clicks when you see it on screen▶ Jump to 7:01 - 10:31 – 11:59Skim
The two-or-three-step short loop
The end-to-end flow is just user query → planning LLM → deterministic solver → result set, set against a multi-step agentic loop that can keep running over and over; per-query cost holds constant at about 9,000 tokens.
Only with a short loop and a fixed number of steps does total cost stay flat, instead of blowing up with scale or query complexity
The architecture flow diagram at 631s tells the whole story in one slide; scan the diagram alongside the narration — there is no live demo▶ Jump to 10:31 - 11:59 – 13:38Skim
Production numbers: 100% and 300x
The old approach fell from about 80% correctness at 64 GPUs to about 30% at 460,000; the new one held 100% accuracy across every scale and across 66 cases on six production systems, while a single evaluation pass dropped from 116 million tokens to 390,000.
A 300x token reduction, and customer cost no longer grows with equipment count — the engineering rebuild directly rewrote the business cost curve
The comparison charts at 719s and 763s are data-dense slides; reading the numbers off the charts is faster and more accurate than hearing them retold▶ Jump to 11:59 - 13:41 – 16:23Listen
The method: running the Karpathy curve backwards
AI-native software should start at nearly pure software 3.0 to validate value fast, then, once it meets real scale, move the structural work you can data-model back into software 1.0; retrieval, set logic, counting and dedup — anything that must be 100% reproducible — goes to deterministic code.
Every 1.0 function you add is more reliable ground for the LLM to stand on — the LLM keeps only ambiguous parsing and answer synthesis
The close rises into a purely spoken methodology argument with no reliance on charts — good to just listen to and walk away with the division of labor▶ Jump to 13:41