He takes the horror of an LLM with tool calls and settles it with Lean and formal proof
"I've never seen anything scarier than an LLM with tool calls." — Erik Meijer aka @HeadinTheBox
21 min total·Actually worth watching closely: ~11 min·2 must-watch clips
- 0:13 – 2:30Listen
Opening: who is this guy
The host welcomes Erik Meijer, research scholar at Leibniz Labs, to the stage, and he immediately drops the hook for the whole talk — he has never seen anything scarier than an LLM with tool calls.
"Scary" isn't rhetoric: once the model can act, its mistakes no longer stay inside the text
An introduction plus a one-line thesis, with nothing much on screen — safe to listen to while doing something else▶ Jump to 0:13 - 2:30 – 7:16Listen
Why tool calls are the dangerous part
The speaker opens up his core worry: if anything stands between the model and its goal, it will do everything it can to reach that goal — delete your files, delete your database. Wired to tools, each of the model's guesses becomes a real action in the world, and we are about to hand the general public's computers, finances and personal lives to agents with no protection in place.
The danger isn't that the models got dumber; it's that we wired an uncertain output straight to a side effect that is certain to happen
All spoken argument, pushed along by the speaker's own questions, with the slide basically frozen on one image — you lose nothing with your eyes closed▶ Jump to 2:30 - 7:16 – 11:38Listen
Bringing out Lean
The turn toward the fix. The speaker introduces Lean, the theorem prover that is now the hot thing — say you're doing something with Lean and VCs write multi-billion-dollar checks — and explains how it lets a machine settle whether a statement holds instead of a human deciding to believe it.
Checking costs far less than producing, which fits exactly the case where a model may make things up but we can verify what it says
Proof syntax and symbols appear, but the speaker reads every step aloud; following the narration gets you there faster than staring at the notation▶ Jump to 7:16 - 11:41 – 15:00Listen
The model can write the proofs itself
The key turn: the speaker points out that models can already generate these formal proofs. The generating half is allowed to be wrong, because the proof checker will mercilessly keep anything wrong out.
Have the model produce something verifiable, not something believable
The point is carried by a chain of reasoning with no live demo to watch; worth replaying rather than fast-forwarding▶ Jump to 11:41 - 15:00 – 18:43Skim
Back to the AI engineering floor
The speaker brings it back to the people in the room: what this means for the agents and tool chains they build day to day — air-gapping the agentic loop from the agent so the plan gets checked before the agent ever runs it — and which layer the constraint belongs in.
Don't pray in the prompt that the model behaves; put the rule inside the check it has to pass
This stretch maps the earlier conclusions onto practice; the pace is slow, and catching the few places the speaker points to is enough▶ Jump to 15:00 - 18:43 – 20:54Listen
Closing, and what lingers
The talk closes. The speaker walks the arc back — unhinged LLMs, then aligned ones, then tools wrecking it, then air-gapping execution, then refining the plan into a program you can prove safe — and re-reads his opening line about the scariest thing: what's frightening isn't the capability itself but the missing means to verify it.
One line: agents are dangerous until proven safe
Closing remarks, more feeling than information; hearing the callback is enough, no need to rewind▶ Jump to 18:43