Can't reproduce that production agent failure? Stop fighting for determinism — record and replay turns it into a free regression test.
Your Agent Failed in Prod. Good Luck Reproducing It. - Tisha Chawla & Susheem Koul, Microsoft · Tisha Chawla
14 min total·Actually worth watching closely: ~4 min·2 must-watch clips
- 0:00 – 1:35Listen
A $190,000 bad trade, dashboards all green
The user said "sell a thousand dollars of stock" and the agent dumped 1000 into the quantity field, selling 1000 shares. The API returned 200 OK in thirty milliseconds, the dashboards sat perfectly green, zero alerts.
The most expensive agent failures trip no alert at all — at the system level it succeeded; it was only semantically completely wrong.
It's a story-driven retelling of the incident throughout, with the slides as backdrop; audio on your commute is enough.▶ Jump to 0:00Speaker · Tisha Chawla - 1:35 – 3:20Listen
Chase debuggability, not determinism
Pulls apart two words we keep mixing up: same input, same output (which a hosted API won't give you and which you don't actually want), and rebuilding a run that already happened well enough to debug it.
The question to ask isn't "how do I make the model deterministic?" but "how do I debug and retest a run I can't reproduce?"
This is the talk's pivot, carried by the distinction between two terms rather than by any diagram — just follow the wording.▶ Jump to 1:35Speaker · Tisha Chawla - 3:20 – 5:00Listen
Temperature zero is a misconception twice over
Dropping the temperature to zero doesn't fix a broken reasoning path — it just means the model makes the exact same logical error in the exact same place — and at the hardware level it doesn't buy real determinism either.
The randomness in generation shouldn't be eliminated — it's exactly what gives your agent its agency.
A counterintuitive stretch of argument with a turn in every sentence; follow the speaker's rhythm, the visuals won't help.▶ Jump to 3:20Speaker · Tisha Chawla - 5:00 – 7:04Skim
Where non-determinism comes from: batching and expert routing
From first principles: sampling determinism isn't system determinism; floating point addition isn't associative, so one shift in operation timing can flip the winning token; and the real culprit is that your request gets grouped with whatever else hits the server that millisecond, so once expert capacity overflows, the routing changes.
The same prompt run a thousand times can return dozens of different responses, and the reason is "who you got batched with" — not a variable you control.
This stretch is backed by a principles diagram; a glance at the batching and routing structure is enough, no need to study every frame.▶ Jump to 5:00Speaker · Tisha Chawla - 7:05 – 8:20Listen
Where to record: node boundaries, not the network layer
Half of what an agent does never goes over the network — local retrieval, in-process tools, memory state — and network-layer capture also shreds under streaming and async. The right move is to wrap every node in the workflow and record the meaning of what enters and leaves it, plus the full environment.
What you freeze isn't just the prompt but the whole envelope: model version, build ID, retrieved chunks — all of it counts as a variable.
This is design trade-offs and methodology; there's only a concept diagram on screen, so listening carries it.▶ Jump to 7:05Speaker · Susheem Koul - 8:21 – 9:40Watch
Demo: making the run that went wrong visible
All three nodes of the stock-selling agent are wrapped for recording: the planning step, the place order tool, and the finalize step. In the trace you can expand each layer to see the call the model generated after mistaking the dollar amount for a quantity, along with each node's model version and sampling parameters.
With input and output recorded per node, "who generated this wrong call" stops being guesswork — you can walk all the way back to the source.
This is the densest screen in the talk: the wrong call's arguments and the expandable detailed records are all on screen — audio alone misses it completely.▶ Jump to 8:21Speaker · Susheem Koul - 9:40 – 11:40Watch
Rewriting the incident into a free regression test
Load that recorded trace, enable replay mode, let the unchanged nodes emit their recorded values so the exact same execution path is reproduced, run only the newly guarded tool live, then write an assertion on the tool's output.
The whole test makes zero model calls and reruns the exact failure offline — the loop is annotate, record, visualize, understand, fix, replay, verify.
What matters is how the code is written, which nodes get stubbed, and what the assertion looks like; all of that is on screen, and it's the part you can copy straight back into your own project.▶ Jump to 9:40Speaker · Susheem Koul - 11:41 – 14:08Listen
Two kinds of testing and five takeaways
Guardrails belong on your tools, not on an attempt to control the model. Testing splits in two: for the deterministic nodes (guardrails, tool calls), stub the model's output with recorded values and kick probability out of the window; subjective dimensions like tone and trajectory go to LLM-as-a-judge instead.
Both matter equally, but don't run them together — pin down what can be made deterministic first, and hand only the rest to subjective judgment.
A closing summary and list of takeaways; the screen is a plain text list, so listen through and note the five points.▶ Jump to 11:41Speaker · Susheem Koul