Let anyone ask business questions in natural language, then freeze the answer into a widget with zero ongoing LLM cost
Why Can't Anyone Answer Questions About the Business? — Garrett Galow, WorkOS · Garrett Galow
19 min total·Actually worth watching closely: ~4 min·3 must-watch clips
- 0:14 – 3:41Listen
Why nobody can answer business questions
Unpacks the old path for internal questions: prebuilt dashboards are rigid and miss the long tail, so non-technical colleagues have to find someone with SQL access, go back and forth on context and detail, and get a one-off answer shared in Slack. It happens every day and doesn't scale.
The bottleneck isn't a missing dashboard — it's that answering long-tail questions depends on the few people who know SQL.
Pure problem statement and motivation, no key visuals — fine to listen to like a podcast.▶ Jump to 0:14Speaker · Garrett Galow - 3:41 – 4:50Watch
Enter Studio: the agent queries for itself
Shows Studio's agent recognizing what it has access to — Linear, Notion, Snowflake — then figuring out the schemas, picking the tables it needs, and running a series of queries to find the data on its own.
The agent doesn't run a query you wrote; it explores the schema itself and queries repeatedly until it has the answer.
An LLM-flagged key visual moment (221.81s) — watching the agent run queries is far more direct than hearing it described.▶ Jump to 3:41Speaker · Garrett Galow - 4:50 – 6:57Skim
Architecture breakdown
The entry point is the internal Studio dashboard or a Slack bot; a small API parses the request and runs it through LangGraph, an agent tied to an LLM (Opus) plus the tools and guidance layer that drive the queries, with session state stored in Convex.
The combination: LangGraph for orchestration, Opus for generation, the guidance layer as the safety net, Convex for session state.
An architecture slide (visual moment at 289.56s) — glance at the diagram to fix the component relationships; no need to follow line by line.▶ Jump to 4:50Speaker · Garrett Galow - 6:57 – 8:50Watch
From one-off answer to lasting asset
After the answer lands, one more sentence freezes it into a reusable widget sliceable by time. Support shifts from engineers sharing SQL queries to pulling real data themselves in Slack from a widget with the query already wired in.
The self-serve flywheel: one-off questions get answered on the spot, recurring ones get frozen into widgets and shared with everyone.
Three consecutive visual moments (416.73/469.61/516.88s) form the complete demo chain — the stretch most worth watching on screen.▶ Jump to 6:57Speaker · Garrett Galow - 8:50 – 12:10Listen
Three reliability pillars
Sequencing: run a pre-flight checklist first and only inject a tool's usage context when the agent actually decides to call it, to avoid blowing out the context window. Layering: a base prompt plus org-level rules, with the LLM explicitly told to distrust stale product knowledge from its training data. Validation: valid SQL doesn't mean data came back, so the query must actually run and be validated before it's hardcoded into a widget.
"Valid SQL, zero rows" is the sneakiest trap — the agent has to run the query and confirm data comes back.
Mostly methodology, no demo footage; the reasoning is continuous and rewards focused listening.▶ Jump to 8:50Speaker · Garrett Galow - 12:10 – 14:19Listen
The guidance layer: writing domain conventions into context
Schemas alone can't query a sprawling internal database correctly. The guidance layer has to encode domain conventions — how a customer is represented in Snowflake, how to join tables effectively, and filters like pulling only active, non-deleted entities.
With the conventions written in, the hit rate is very high — but it's still trust but verify.
There's a context excerpt on screen around 729.88s worth a glance, but the core is the spoken governance reasoning — mostly listening.▶ Jump to 12:10Speaker · Garrett Galow - 14:19 – 18:48Listen
Q&A: governance, determinism, and cost
Answers the worry that a wrong query becomes accepted truth; explains that a widget is JS the LLM generates once, so it's deterministic code at runtime with no ongoing LLM cost; and covers data access moving to org connectors with default permissions and Studio roles.
The LLM is involved only at generation; runtime is purely deterministic — the Opus spend goes to that one step, and it's worth it.
Q&A needs no visuals and hides the key answers on architecture and cost trade-offs — good to get through at higher speed.▶ Jump to 14:19Speaker · Garrett Galow