Around the 200ms tyranny of latency: three engineering rules for real-time voice in, visuals out agents
Voice In, Visuals Out: The Agony and the Ecstasy - Allen Pike, Forestwalk Labs · Allen Pike
13 min total·Actually worth watching closely: ~4 min·2 must-watch clips
- 0:01 – 1:10Listen
Opening: Karpathy's new interaction paradigm
Opens with Karpathy's argument that voice is the human preferred input for AIs while we prefer visuals as the output, noting this is the opposite of how we've mostly been building and using AI, yet closer to what people actually prefer.
Voice in, visuals out isn't a gimmick—it's a reasoned account of how humans prefer to interact.
Pure setup with no key visuals; fine to just listen on the commute.▶ Jump to 0:01Speaker · Allen Pike - 1:10 – 2:40Watch
The leap in visual output
Shows how models have recently got to the point where they generate rich HTML, visualizations that explain things, interactive controls you can use to direct the model, and even beautiful illustrations—the ceiling on visuals out has really lifted.
This paradigm only became feasible in the last few months, because model visual generation just got good enough.
The 72s mark is the visual moment the LLM flagged; the output examples land far better on screen than in narration.▶ Jump to 1:10Speaker · Allen Pike - 2:40 – 4:40Listen
In defense of voice: the highest-bandwidth input
Speaking beats typing on words per minute and each word conveys more through tone; voice's bad name comes from slow and dumb experiences like Siri, not from voice itself.
Speaking is the ultimate way humans communicate—it's the bad implementations that got the blame wrong.
An argument carried by narration; no need to watch the screen.▶ Jump to 2:40Speaker · Allen Pike - 4:40 – 7:02Listen
The tyranny of latency
A seamless voice conversation needs the whole chain—networking, speech to text, inference—inside 200 milliseconds, which is close to impossible today; people are far more forgiving of visuals, up to about a second, so voice in, visuals out routes straight around the bottleneck.
No need to wait for novel inference architectures: switch to visual output and a one-second budget is enough to feel seamless.
The core argument; the numbers are all spoken, so the slides are only supporting.▶ Jump to 4:40Speaker · Allen Pike - 7:04 – 9:00Skim
Lesson one: use a really fast model
GPT-5 Mini showed P95 latencies of 5–10 seconds and was never fast enough; go with a Haiku-class model or a smaller open source one, and have it hand heavier work off to a larger model that can think, interleaving that result back into the responses.
A small fast model carries the real-time chain; the big model only does the heavier work off to the side.
There's a slide comparing latency numbers—glance at the figures; the conclusion is what matters.▶ Jump to 7:04Speaker · Allen Pike - 9:00 – 11:00Listen
Lesson two: fire inference eagerly as they speak
Don't wait for about a second of silence to start inference—that burns the budget; send inference every one or two seconds while the person is still talking, even if you're not entirely sure they've stopped.
Spend the latency budget while the user is still speaking, not after they finish.
A strategy argument with no demo footage; grasping the mechanism is enough.▶ Jump to 9:00Speaker · Allen Pike - 11:00 – 13:04Watch
Lesson three: a stable caching regimen, and the close
Keep about 90% of the context the same from request to request, use only that final 10%, and minimize output tokens for up to 90% cheaper, faster inference—an architecture most LLM applications are converging on.
A stable prefix plus minimal output is a general architecture for speed and cost, not a voice-only trick.
The caching structure is fastest to grasp from the diagram, and it's the one page worth screenshotting.▶ Jump to 11:00Speaker · Allen Pike