Hand the whole agent iteration loop over to agents; the eval gate is the only termination condition.
The Agentic AI Engineer - Benedikt Sanftl, Mutagent · Benedikt Sanftl
35 min total·Actually worth watching closely: ~8 min·3 must-watch clips
- 0:01 – 7:02Listen
Why iterating on agents by hand runs out of road
Argues from experience building agents that the manual loop — implement the change, look through traces, A/B test — bottlenecks on human review and breaks down once you're rolling out hundreds of agents. Makes the core case for making the loop itself agentic, with the developer becoming the loop designer.
The human's role shifts from running the iteration to designing the loop and setting the eval termination gate.
Pure talk, setting up the problem and the idea; nothing critical on screen, fine to listen to like a podcast.▶ Jump to 0:01Speaker · Benedikt Sanftl - 7:02 – 14:04Listen
Spec as blueprint, eval as termination condition
How a spec should capture requirements, success criteria, context and tool boundaries, and stay decoupled from implementation to hedge against fast-moving frameworks; eval-driven development is the agent world's TDD, answering when something is good enough.
With the spec kept separate from implementation, a coding agent can build it against any framework — the framework choice stays yours.
Conceptual framing; the density is in the audio, not the slides.▶ Jump to 7:02Speaker · Benedikt Sanftl - 14:04 – 18:53Listen
What makes an eval useful
Evaluation has to cover the whole trajectory — was the context complete, was every tool output right — not just the final output; argues that binary criteria beat score-based LLM-as-judge, and that the judge has to be calibrated for scoring noise.
A binary criterion tells you what to fix when it fails; an uncalibrated judge can't support the claim that the new version is better.
Substantive methodology, but argued verbally — listen and take notes on the points.▶ Jump to 14:04Speaker · Benedikt Sanftl - 18:57 – 25:05Listen
Production diagnostics and the autonomous optimization loop
After go-live, failure modes are grouped by root cause, new evals are generated to detect the problems before remedies are generated, and the criteria flow back into the spec; with an eval suite in place the loop runs itself — vary, experiment, ship automatically when everything is green.
A complete eval suite is a product of discovery: the real edge cases only grow out of production failures and user feedback.
The loop is held together by narration with nothing demoed on screen — put your attention on the order of the stages.▶ Jump to 18:57Speaker · Benedikt Sanftl - 25:11 – 28:17Skim
Mutagent platform architecture
The two research-preview agents — evaluator and diagnostics — connected through an orchestrator and all running in your own environment; connectors pull traces and incidents from sources including ticketing systems and Slack, and output goes back out as GitHub PRs or edits to markdown agent definitions.
The whole system runs inside your environment, and both ends plug straight into your existing engineering pipeline.
Mostly an architecture slide (at 1564s there's an 'as you can see' pointing at it) — glance at the diagram for the component relationships; no need to follow every sentence.▶ Jump to 25:11Speaker · Benedikt Sanftl - 28:17 – 32:18Watch
Diagnostics agent live demo
Burak drives it hands-on: diagnostics can be scoped to a single agent or skill, pulling traces from Langfuse, local Claude transcripts or JSONL; guided search targets a specific user-reported problem, and the output is an HTML report with frequency statistics over a time window and recursive why chains.
The report doesn't just list problems — it follows the why chain down to the root cause and counts how often each occurs, which reading traces by hand can't give you.
The most visually dependent stretch of the talk; the key frames the LLM flagged (1697s, 1850s) are all here — you have to see the interface and the report structure yourself.▶ Jump to 28:17Speaker · Burak - 32:18 – 34:45Watch
Picking remedies, correcting assumptions, handing off the task
The report points at where the issue originates (a particular tool, for instance) and offers multiple-choice remedies; the assumptions block surfaces what was assumed without code access so you can correct it; once you're happy with all the decisions you get a markdown task definition to take back to your coding agent.
Making assumptions explicit is the design that lets an LLM's diagnosis be trusted — not a patch bolted on afterwards.
The closing stretch of the demo; three flagged visual moments (1938s, 1962s, 2009s) cluster here — the assumptions block and the decisions page only make sense on screen.▶ Jump to 32:18Speaker · Burak