Same model, only the harness changed — the score jumped more than 20 points. The evidence and the how-to are both in this talk.
What if the harness mattered more than the model? - Aditya Bhargava, Etsy · Aditya Bhargava
32 min total·Actually worth watching closely: ~13 min·3 must-watch clips
- 0:01 – 5:45Skim
A contrarian opening: the harness is badly underrated
Challenges the industry wisdom that the models are so good you can just keep the harness simple, frames an agent as a model plus a harness, and brings in HarnessBench: the same model scores more than 20 points apart with only the harness changed.
The harness isn't glue code — it's a performance variable on the same order as the model, and the weaker the model, the more the harness compensates.
Mostly spoken argument, but the HarnessBench chart at 2:22 (52.4% vs 76.2%) is the foundation of everything that follows and is worth stopping for; the rest can be played at speed.▶ Jump to 0:01Speaker · Aditya Bhargava - 5:48 – 9:16Skim
The example task and a first failure that had to happen
Sets up the task that runs through the whole talk: fix a bug in a median function (for even-length input it should average the two middle numbers). The barest version, written in the speaker's own language Agency, passes the prompt straight to the LLM — which replies that it needs to see the existing code, since it can't read or write files.
Just the model can't do anything; this starting failure is the control case for every rung of the harness ladder that follows.
The task code and Agency's syntax (TypeScript-flavored with a Python feel) read at a glance from the slides — glance at the buggy code at 8:13 and the first example at 8:42; the narration you can just listen through.▶ Jump to 5:48Speaker · Aditya Bhargava - 9:25 – 14:47Watch
Adding tools and a safety mechanism: handler approval
Shows that in Agency every function automatically becomes a tool — Agency builds a JSON schema from it and uses the docstring as the tool description — then the standard library's interrupt-and-handler mechanism: destructive operations pause for human approval before running, with the built-in log viewer for auditing every call.
Capability and safety are designed together and safety is the default — the standard library raises an interrupt before anything dangerous, so arbitrary reads and writes to your file system aren't allowed out of the box.
This stretch is dense with live demo: functions becoming tools at 10:11, stepping through calls in the log viewer at 11:16, the handler approval flow at 13:25 — the screen carries information the audio alone can't.▶ Jump to 9:25Speaker · Aditya Bhargava - 14:53 – 19:21Skim
PFA: safety that doesn't need a human
Locks an argument with .partial — the read tool's directory is pinned to demo — so the LLM can neither change it nor see that it exists, which draws the capability boundary structurally. The agent now reads the files without approval and produces the right fix, but only proposes it; the code isn't actually changed yet.
Partial Function Application is safety with zero human input: the constraint lives in the function signature, not in the prompt.
The whole idea is that one line of code — glance at the snippet at 16:01 and the run at 17:28; the reasoning in between can be played at speed.▶ Jump to 14:53Speaker · Aditya Bhargava - 19:24 – 26:30Watch
The feedback loop closes, then subagents
With a ReAct feedback loop the agent finally does the job end to end: reads the files, runs the tests and sees them fail, writes the fix, runs the tests again and confirms they pass. Then subagents, which are just ordinary functions — tools grouped by purpose so the top-level agent only picks which one to call, growing capability without bloating context.
The ladder is complete: model, tools, handlers, PFA, feedback loop; and subagents add capability without bloating context.
The two demos most worth watching are both here: the full feedback loop at 20:12 and the two subagents running in parallel at 23:36 (one fixing the test, one explaining Jensen's inequality in a limerick) — the screen is the argument.▶ Jump to 19:24Speaker · Aditya Bhargava - 26:31 – 32:02Skim
The built-in optimizer, and pause-and-resume to close
Demos the optimize modifier: the optimizer checks which prompts need optimizing, runs the agent to establish a baseline, then rewrites the prompt and hits the objective within the first iterations — systematically measuring and improving instead of guessing and checking. It closes on Agency's killer feature: at an interrupt, execution state can be paused and resumed later at that exact point.
Building a truly good harness requires language-level support: an execution model that can be optimized and paused and resumed, which most frameworks' human-in-the-loop can't do.
The optimizer going from baseline to objective at 27:04 is worth a glance, but this stretch is mostly conclusions — the language features and the vision of escaping vendor dependence with local open source models are fine to just listen to.▶ Jump to 26:31Speaker · Aditya Bhargava