中文
AI Engineer World's Fair

Break "done" out of a single green checkmark into a verifiable object, or your agents will produce slop at scale

What Does Done Even Mean? Agents and Paperclip's Liveness Model - Dotta, Paperclip · Dotta

7 min
AgentEvals

7 min total·Actually worth watching closely: ~3 min·2 must-watch clips

Orange = the 3 minutes worth watchingFor the rest, the guide is enough
Segment guide · 7 segments
  1. 0:00 0:55Listen

    Where the problem starts: speed has outrun verification

    Opens straight on the premise - agents now produce code and documentation faster than any human can ever verify. Keep asking a person to sign off on every task and what you get is not a quality guarantee.

    Once the volume of sign-off exceeds human bandwidth, sign-off degrades into verification theater: the form is still there, the assurance is gone.

    A purely spoken argument, with the speaker talking over a minimal text slide and nothing extra on screen. You won't miss anything listening with headphones on a walk.▶ Jump to 0:00
    Speaker · Dotta
  2. 0:55 2:10Listen

    Stop writing done as a Boolean

    States the core claim: done isn't true/false, it's a bundle of claims - what artifact was produced, what the scope is, against what rubric or standard, what evidence there is, who verified it, who has authority to sign off, what risk might be left, and what the next action is.

    Give agents a structured checklist to verify piece by piece, instead of letting them make a blanket claim that they're done.

    The list is read out loud item by item; following by ear makes the order easier to hold onto than staring at the screen, and there's nothing worth stopping for visually.▶ Jump to 0:55
    Speaker · Dotta
  3. 2:10 3:25Skim

    Done is a ladder, not a switch

    Breaks done into escalating levels: the producer claims it's complete, a reviewer finds no obvious issues, the evidence meets the standard, it is formally approved, someone stands behind it, and finally the outcome survives real-world conditions. Done enough to merge, to deploy, and to announce are each different operational claims as well.

    What's wrong with most agent systems is that they flatten this entire ladder into one green checkmark.

    The levels move as a list at a fairly dense pace. Skim through and pause at each level change to line the order up; going line by line doesn't pay off much here.▶ Jump to 2:10
    Speaker · Dotta
  4. 3:25 4:30Listen

    The hardest tension in the talk: slop on one side, gridlock on the other

    Sets the two extremes side by side: with no approvals at all the system sprints ahead and produces textbook AI slop; hand it all to human review and you build a review queue no one can get through. Neither end works.

    The real problem isn't how to get agents to do more work - it's the balance you have to solve between keeping work moving and having it genuinely verified.

    The densest stretch of argument, carried by the speaker's reasoning rather than the screen. Listen at normal speed and don't skip; drop either half of the conclusion and it no longer holds.▶ Jump to 3:25
    Speaker · Dotta
  5. 4:30 5:30Skim

    Why a for loop can't hold this up

    Rebuts the most common shortcut - writing a loop over your task manager to drive your agents. Bring in task dependency trees, blockers, multiple agents at once, and idempotent checkouts with locks, and the whole thing collapses. Then come the invariants a control plane has to hold.

    Three invariants: productive work continues, only real blockers stop work, and infinite loops are bounded.

    The three constraints are parallel items, and the screen is most likely just three lines of text. Catch the list, write the three down, and you can speed through the elaboration in between.▶ Jump to 4:30
    Speaker · Dotta
  6. 5:30 6:35Listen

    How Paperclip lands it

    The concrete implementation behind everything above: every task has clearly defined state transitions; blockers between tasks are first-class and enforced by the control plane; human approval is interactive and leaves an audit trail; reviewers and approvers can be set explicitly.

    Making blockers and sign-off authority into structure the system enforces, rather than convention and good intentions, is what finally gives done a verifiable shape.

    A spoken description of the mechanisms rather than a live walkthrough - no actual product UI or running system appears on screen. It's enough to catch which of the earlier problems each of the four mechanisms answers.▶ Jump to 5:30
    Speaker · Dotta
  7. 6:35 7:11Listen

    The watchdog, and wrapping up

    Introduces the watchdog, also called maximizer mode: a separate agent given a goal, responsible for pushing all of your agents to keep working until that goal has been achieved. It isn't tied to any harness - Claude Code, Codex, whatever you're using, one consistent interface. Then the talk closes.

    Forward drive deserves to be its own role, and it should be decoupled from the specific execution harness - that's what decides whether it can span your existing toolchain.

    A brisk closing stretch: one sentence of concept and straight into the wrap-up. Nothing on screen to linger on; just listen through the last line.▶ Jump to 6:35
    Speaker · Dotta