Carbon accounting has no single right answer, so he moved validation off the answer and onto the process.
Respect The Process - Andrew Dumit, Watershed Technology Inc. · Andrew Dumit
17 min total·Actually worth watching closely: ~6 min·3 must-watch clips
- 0:00 – 2:25Listen
Same bottle of wine, answers 50% apart
Opens with a study: experts with access to all the same data on the exact same bottle of wine produced emissions answers that varied by up to 50%. That frames the problem for the whole talk — in a domain with many right answers experts will disagree on, you can't tell whether a system is correct by checking its final answer.
Where expert opinion reasonably diverges, "the answer is right" is not a usable acceptance criterion, and your ground truth is itself only one point in the range of possible expert judgments.
This stretch is purely spoken argument; the screen mostly holds the title and the study's conclusion, so it's fine to listen while doing something else.▶ Jump to 0:00Speaker · Andrew Dumit - 2:25 – 4:25Skim
The task: one graph per product, thousands of nodes
Shows what the agent is actually being handed. Each product maps to a full supply-chain graph — for dark wash jeans, the assembly from denim thread, labels and zipper, along with the energy, transportation and packaging needed to move it through the supply chain, each its own flow, running to thousands of nodes that all carry rich metadata about the materials and processing at each step.
The task isn't answering a question; it's making bulk edits across a large graph with complex structure and interdependent fields.
The jeans graph is worth pausing on to take in the branching and node density; you don't need to read individual node names, just get a feel for the scale.▶ Jump to 2:25Speaker · Andrew Dumit - 4:25 – 7:04Listen
Specialized tools broke at scale, so they swapped in a coding agent
First the old route: highly specified graph tools handed to a ReAct agent. Fine on one graph, a total failure at tens or hundreds of graphs and hundreds of thousands of nodes — one approach on one graph and a different one on the next, a third graph forgotten entirely, exploration itself becoming the bottleneck, and once context was eaten up, hallucinated pieces of the schema leading to retries and errors. Swapping in a coding agent was a big jump: it found creative routes into underspecified problems, explored and edited in bulk the way agentic data science workflows do, and proved far more flexible than what it was designed for. The cost is that unconstrained code is quite scary — it will find any path to the goal, including reaching for things it shouldn't.
The upside and the risk come from the same thing — letting it write code freely — so the question isn't whether to allow that, but which layer to put the constraint on.
A post-mortem plus a trade-off argument; the screen is just bullet points, so follow the narration.▶ Jump to 4:25Speaker · Andrew Dumit - 7:04 – 9:20Watch
The core architecture: one door for edits, execution never handed over
The talk's main answer. How the agent reasons and writes code is left alone, but any code that edits the graph has to go through a typed SDK — it enforces which fields are editable versus derived from other fields, and it can be linted and checked for errors. The platform then runs the final execution itself, producing the typed objects that get committed as the actual edits, which makes the whole process valid, traceable and replayable; if it isn't, the run is rejected and sent back.
Constrain the effects, not the expression: the model's reasoning power is left fully intact, and what's locked down is only what it can do to the data.
The layering rests entirely on that diagram — which layer is free, which one narrows, whose hand is on execution. It's easy to conflate the levels by ear alone.▶ Jump to 7:04Speaker · Andrew Dumit - 9:20 – 11:40Skim
How to teach the agent your own SDK
The practical details. Teaching the agent your SDK follows the same pattern as teaching it to write code in your codebase: the prompt tells it how the thing works, and the agent also has full access to the docs and the code underlying the SDK if it needs to read them. The SDK hands it assertions so its code fails early rather than quietly producing a wrong result. On top of that: skills and few-shot examples that teach and coach it, better ergonomics on the tools themselves so their use is obvious, breaking the problem into a plan-and-execute loop, and explicitly teaching the expert judgment endemic to the space.
However new the architecture, the ordinary context and prompt engineering that scopes the task is still not optional.
The screen is a list of points plus a code sample, moving fast; glance at how the assertions are written, and hearing the rest is enough.▶ Jump to 9:20Speaker · Andrew Dumit - 11:40 – 14:08Watch
What comes out: a review report you can drill through
Shows what deterministic execution finally emits — a structured review report you can read without a single line of low-level code: an edit function's effect across the graphs it touched, overall emissions down 45.6%, and you can keep drilling from there into the individual edits, then into a single graph, then into exactly which nodes were changed within it.
With a reviewable output like this, the agent's creativity turns from a risk into an advantage fenced in by safety boundaries.
The report's information hierarchy is where all the persuasive force sits — going from the headline number down into a single node has to be seen. Hearing the numbers alone makes it sound like a few lines of logging.▶ Jump to 11:40Speaker · Andrew Dumit - 14:08 – 16:42Listen
Closing: three principles for respecting the process
The methodology compressed into three transferable principles: give the agent well-scoped primitives rather than freely operating code; the platform must maintain full control of the final execution, so the agent can't declare victory in a way you never intended; and use that deterministic outcome to produce results even non-coders can validate — writing code is the means, not the end.
"Respect the process" means this: since you can't validate the answer, put the guarantee on the path that produced it.
The close is three principles restated in a sentence each over a single bullet slide — fine to hear out with your eyes closed.▶ Jump to 14:08Speaker · Andrew Dumit