How an $85K failed banking chatbot shipped in 8 weeks — with the model picked in week 7
The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks · Sandipan Bhaumik
37 min total·Actually worth watching closely: ~7 min·3 must-watch clips
- 0:15 – 7:16Skim
Why the demo looks great and production falls over
Dissects the classic enterprise AI failure mode — every conversation starting with which model to choose, no evaluation, no tracing, no data strategy — then puts the five-pillar framework forward as the answer.
Evaluation, tracing, data, orchestration and governance all have to be planned before you touch any code
Mostly argument; the one frame worth pausing on is the five-pillar diagram at around 313 seconds — a glance at that slide is enough.▶ Jump to 0:15Speaker · Sandipan Bhaumik - 7:16 – 12:25Listen
Evaluation: the specification for your AI system
How to define success in numbers (60% of queries deflected to the agent, 85% accuracy), how to build the evaluation data set with domain experts from around 200 real cases, and the three-layer architecture: deterministic checks, LLM as a judge, behavioural evaluation.
Behavioural evaluation — did the agent call the right tools, did it get stuck in a loop — matters most and is most often left out
Pure spoken methodology with no key visuals; fine to listen to while doing something else.▶ Jump to 7:16Speaker · Sandipan Bhaumik - 12:25 – 16:27Watch
Tracing and data: agents don't forgive bad data
Uses the full trace of an overdraft fee waiver to show exactly what production observability has to capture, then argues the data foundation is the most important pillar — 60% of the speaker's time on a typical project goes there.
Humans are forgiving about wrong data; an agent will find it and answer wrong, confidently, without you knowing
The decision-chain trace example at around 758 seconds is the densest frame in the talk — worth following on screen to see what gets captured at each step.▶ Jump to 12:25Speaker · Sandipan Bhaumik - 16:27 – 23:29Skim
The data stack, a central tracing layer and orchestration patterns
Covers how Delta Lake brings database-like properties to unstructured data, a centralized tracing layer across frameworks and clouds, and the trade-offs between orchestrator-worker and choreography multi-agent orchestration.
Enterprise AI inevitably spans frameworks and clouds, so tracing data has to be collected into one shared central layer if it is to serve dashboards and monitoring
Diagram-heavy (Delta Lake at 1010s, the centralized tracing layer at 1182s, the two orchestration patterns at 1266/1308s) — skimming the diagrams for structure is enough, and the Databricks product section can be sped up.▶ Jump to 16:27Speaker · Sandipan Bhaumik - 23:29 – 30:32Listen
Case study: the banking chatbot that picked its model in week 7
A retail banking support chatbot that had already failed once at a cost of $85K, rebuilt week by week on the five pillars: six weeks on the evaluation set, tracing and data foundation, then in week 7 running the candidate models against the evaluation data set and deciding on the numbers. In production in 8 weeks.
Choose the model last, not first — once you have your own evaluation data set, the decision turns out to be very fast
Mostly narrative; the week-by-week timeline is easy to follow by ear. The operating metrics slide six weeks after go-live, at 1778 seconds, is worth a glance.▶ Jump to 23:29Speaker · Sandipan Bhaumik - 30:32 – 36:47Listen
The production incident playbook, and wrapping up
The playbook AI projects most often lack: detect with the eval dashboard, diagnose with tracing, contain by pulling the prompt version, then fix and feed the case back into the evaluation set. Closes with downloadable templates and checklists.
Prompts go in Git, and the commit message must record which failure the change was made for; the evaluation data set is a living system that needs an owner and governance
Mostly process narration; remember to screenshot or note how to get the resources on the download page at 2186 seconds.▶ Jump to 30:32Speaker · Sandipan Bhaumik