Real eval scores on screen: the full loop for letting a coding agent tune an AI agent
Agents Building Agents - Alfonso Graziano, Nearform · Alfonso Graziano
30 min total·Actually worth watching closely: ~13 min·3 must-watch clips
- 0:00 – 5:00Skim
Why an agent can be tested at all
Breaks down what an AI agent is made of and which classes of problems it can solve, then introduces the foundation for everything that follows: a golden dataset plus scorers built together with subject matter experts. The expected output doesn't have to be text — it can be 'call this tool, with these parameters, in this chain'.
A non-deterministic system still needs a test suite — the dataset and scorers end up giving you a single accuracy number, used to set a baseline, catch regressions and drive improvement. Without that number, none of the automated optimization that follows is possible.
Mostly concept slides and bullet lists; listening along is enough, with a glance at the screen to see what the dataset looks like.▶ Jump to 0:00Speaker · Alfonso Graziano - 5:00 – 7:06Watch
A nearly bare agent, only 18%
Shows the minimal agent with no tools at all, run against the whole suite with the most naive scoring — an 18% pass rate. Along the way, the three common reasons an agent fails: missing the right tools, a system prompt that isn't good enough, and not retrieving the context it should have.
The knowledge in the model's weights only covers the simplest questions; everything else needs tools and external information. Much of what gets called optimization later is just repeatedly refining the system prompt.
The real agent code and the eval results are on screen — you need to see how short the code is and how low the score is; the numbers alone don't land.▶ Jump to 5:00Speaker · Alfonso Graziano - 7:06 – 9:56Skim
The inspiration: let a coding agent run its own experiments
Cites Karpathy's autoresearch experiment: a coding agent repeatedly changing the code and hyperparameters of a deep learning algorithm — the more experiments run, the higher the accuracy and the lower the loss. Every point on the chart is one independent experiment.
As long as the objective can be scored automatically, a coding agent can climb steadily through sheer volume of trial and error — port that to agent development and all you're missing is a score.
It's all about that scatter/trend chart — see that the x axis is experiment count and which way the points are heading; the narration you can just follow.▶ Jump to 7:06Speaker · Alfonso Graziano - 9:56 – 14:10Watch
The autonomous optimization loop: 18% to 83%
Introduces the main body of the method: a loop that keeps running the evals, updating the system's code, trying new system prompts and creating new tools, then re-running to verify. The improvements cover edge cases, the system prompt, tool descriptions and even the tools' own logic — around 10 iterations take that 18% bare agent to 83%.
Improvement doesn't only happen at the prompt layer — how a tool is described and how its internal logic is written are variables the loop can discover too.
The loop running and where each round's change lands are both on screen; seeing which line it actually touched is far more useful than the abstract 'it improves itself'.▶ Jump to 9:56Speaker · Alfonso Graziano - 14:10 – 18:24Listen
One hypothesis per change, always revertible
The engineering constraints inside the loop: each hypothesis gets its own branch, the evals re-run after the change, an improvement means carry on and a regression means roll back, with a memory file and past reports carried across iterations to generate the next hypothesis.
Failed hypotheses are valuable too — a human can read back what they were trying to prove and steer the direction in the next round, which is why every step leaves a trace.
Mostly process narration; following the logic is enough. The one thing worth looking up for is the change log around 16:50, where every improvement, every rollback and its hypothesis is recorded in full.▶ Jump to 14:10Speaker · Alfonso Graziano - 18:24 – 25:24Watch
A real production agent: 67% to 86%
Switches to an agent already running in production and already humanly optimized: 67% at the start, 86% after the loop, and confirmed without cheating by tampering with the dataset or the scorers. Then how live feedback gets collected: thumbs up/down with a note, plus subject matter experts annotating real traces.
You have to explicitly forbid the optimization process from touching the dataset and the scorers — otherwise a great-looking number is just self-deception. And a system humans had already optimized still gained nearly 20 points, which says human tuning was nowhere near the ceiling.
Both the score comparison in the first half and the trace annotation UI in the second half are carried by the screen — especially how an expert marks up what went wrong on a real trace.▶ Jump to 18:24Speaker · Alfonso Graziano - 25:24 – 30:12Listen
Closing the fix loop, and building the bench the agent works on
How the whole loop closes: after the failure clusters, humans and subject matter experts triage by priority (a cluster can be a false positive, or an intended behavior), the coding agent produces the fix, existing real traces get reused directly as regression tests, and every failure mode goes back into the golden dataset. It all lands on one line — what you're really building is the environment the coding agent works in.
What decides the outcome isn't one clever prompt, it's that environment: spec-driven development, quality gates (linting, unit tests, evals), code review, context engineering and observability. With those in place, the agent can change the code, validate its own changes, and keep proposing the next ones.
All methodology wrap-up, nothing to watch — listen and check off which piece your own team is missing.▶ Jump to 25:24Speaker · Alfonso Graziano