Swapping cloud Sonnet for on-device Llama 3.2 without losing performance: a right-sizing eval you can actually copy
Frontier results, on device - RL Nabors, Arize · Rachel Lee Nabors
31 min total·Actually worth watching closely: ~12 min·3 must-watch clips
- 0:01 – 6:49Skim
Why not reach for a cloud frontier model every time
Lays out the three costs of one-size-fits-all inference — privacy and trust, latency past the four-second limit of believability, and inference spend that keeps rising — then opens up the range of SLM options.
Tasks like summarization and content moderation don't need a trillion-parameter model; a one-billion-parameter model quantized runs about 2GB, small enough to ship on a phone.
Mostly spoken argument — the slides (SLM vs LLM parameter counts, quantized sizes, the NVIDIA claim) only need a glance, no frame-by-frame viewing▶ Jump to 0:01Speaker · Rachel Lee Nabors - 6:52 – 13:53Listen
The methodology: prototype big, deploy small
Walks through the four steps — prove it's possible on the most capable model, define success, evaluate small models from the bottom up, pick the SAGE model — and shows how the golden dataset and success measures were built from 14 real threads and 28 human-labeled examples.
Before swapping any model you need ground truth and quantified measures (JSON correctness, structural validity, factual consistency, length, P50/P95 latency); without them there is no way to judge "good enough".
Pure spoken methodology with low visual dependency — listen and take down the framework▶ Jump to 6:52Speaker · Rachel Lee Nabors - 13:53 – 17:20Skim
The baseline and the four candidate small models
Sets Claude Sonnet as the baseline (2.9 seconds average latency, about 22 cents for 14 tasks) and lines up the four on-device contenders — Qwen 2.5, Qwen 3, Llama 3.2, Gemma 4 — while flagging the trap of going on recommendations.
The total cost column for the local models is absolutely zilch, because that inference has been pushed to the consumer's device — but which candidate wins is a question for evals, not for word of mouth.
The baseline numbers and contender list are on slides — glance and note the figures; the argument itself comes through the audio▶ Jump to 13:53Speaker · Rachel Lee Nabors - 17:20 – 20:58Watch
The showdown: Llama 3.2 wins
Phoenix results put accuracy against latency for every candidate: the fastest model, Qwen 2.5, is clearly weakest on accuracy, while Llama 3.2 wins overall around 90%, with many responses pretty much indistinguishable from Claude's.
Speed alone is the wrong criterion, and any single metric picks the wrong model — you need the combined accuracy-and-latency view.
All the persuasive weight of this stretch is in the comparison chart; the gaps (eight seconds vs one second, the accuracy spread) only land if you look at it▶ Jump to 17:20Speaker · Rachel Lee Nabors - 20:58 – 28:00Watch
Prompt experiments and post-processing: closing the gap
Tests prompt techniques one isolated variable at a time — explicit rules made it worse, Chain of Thought wasn't worth the latency, Few Shot was the biggest improvement — then fixes the remaining structure and length problems for free in harness post-processing, bringing Llama 3.2 level with Claude Sonnet.
What works when prompting a small model runs against large-model intuition, and you have to open the evals and read them yourself — an LLM judge will favor the phrasing of its own family.
The experiment charts and the walkthrough of individual eval rows are the heart of this section; the counterintuitive conclusions are only convincing against the data▶ Jump to 20:58Speaker · Rachel Lee Nabors - 28:01 – 30:50Listen
Regression evals and the call to action
Recommends keeping evals running like CI/CD tests so you don't lose ground, then challenges the audience to go look at what they're sending to LLMs and ask whether a smaller model could handle it — and to take advantage of models already on the device, like Chrome's Prompt API reaching Gemini Nano.
The starting move is concrete: take one existing feature, run it through this process onto an on-device small model, and make evals a standing part of the pipeline.
The close is spoken call-to-action and outlook with little on screen — just catch the action list▶ Jump to 28:01Speaker · Rachel Lee Nabors