How PostHog refines trillions of noisy signals into green PRs an engineer can merge on waking up
Self Driving Products: Product Signals to Pull Requests — Joshua Snyder, PostHog · Joshua Snyder
15 min total·Actually worth watching closely: ~4 min·2 must-watch clips
- 0:15 – 2:00Listen
The vision: from dashboards to automatic PRs
Proposes flipping observability from 'read a dashboard, then investigate for days' into 'a product signal triggers a background agent that submits a PR' — engineers only review, and low-risk changes can ship straight behind a feature flag.
Outcomes such as a rejected PR or a bad deployment feed back into the next generation, making this a learning loop rather than one-way automation
The opening is a purely spoken vision statement with no key visuals — fine to listen to while doing something else▶ Jump to 0:15Speaker · Joshua Snyder - 2:00 – 4:10Skim
The five-step pipeline and prompt-injection defense
Lays out the whole pipeline — ingest, group, research the root cause, assess actionability, execute in a sandbox — and explains that because signal sources are public, an attacker can create an error on your site carrying a malicious instruction, so an LLM safety classifier has to sit right at the top.
Any agent pipeline taking signals from public sources needs a prompt-injection filter at the entrance; that's a safety prerequisite, not an optimization
The architecture overview is mostly a slide diagram — one glance to fix the five steps in mind is enough, no need to watch second by second▶ Jump to 2:00Speaker · Joshua Snyder - 4:10 – 7:16Watch
Signal normalization and the promotion threshold
Heterogeneous signals are normalized into a single structure (source product, type, content, importance weight) before being embedded and grouped, and a report is promoted once its accumulated weight crosses a threshold; shows how a random null pointer exception gets connected to a Slack message saying checkout is broken.
Cutting noise isn't about filtering individual signals but about letting signals from different sources corroborate each other after normalization and converge, by weight, into one problem
At 6:31 the speaker points at the screen — "all of your errors over here" — while walking through how the grouping flows; this stretch genuinely depends on the visuals, so watch it▶ Jump to 4:10Speaker · Joshua Snyder - 7:16 – 9:30Listen
The research agent: SDK, sandbox, three kinds of tools
The research agent runs the Claude agent SDK in a Modal sandbox with their own MCP server (pulling in log data alongside a session replay makes results way more accurate), the codebase context, and external MCPs like Linear and Notion; its output covers what the problem is, how urgent it is, and who should review the PR.
Letting the agent pull in whatever related data it wants on demand — rather than stuffing everything into context up front — is the main source of root-cause accuracy
This section is spoken discussion of tool setup and experience; the information is in the words, not on screen▶ Jump to 7:16Speaker · Joshua Snyder - 9:30 – 11:40Listen
The actionability gate: three-way triage
Throw an agent at any problem and it will just try to fix something, so vague reports spawn noisy PRs; reports are therefore split three ways — not enough evidence goes back into the signal pool, product decisions go to a human inbox, and only immediately actionable ones go to an agent.
Error tracking signals are very specific and a coding agent works on them really well; Slack feedback and session replays are much more generic and have to be triaged before they ever reach the execution agent
The triage logic is purely conceptual — just catch the criteria for the three paths▶ Jump to 9:30Speaker · Joshua Snyder - 11:40 – 13:00Listen
Sandbox execution: iterating until the PR is green
Clones the user's repo into a sandbox, writes the fix and pushes a PR; a failing CI run or a comment on the PR rehydrates the snapshotted sandbox and triggers a rerun, iterating until the PR is green.
The bar for delivery isn't 'a PR was opened' but 'a green PR' — engineers wake up to something they can review directly rather than a pile of CI failures
The mechanism is carried entirely by spoken description with no live demo; the point is the rehydrate trigger logic▶ Jump to 11:40Speaker · Joshua Snyder - 13:00 – 14:18Listen
Four lessons from building it
A local vibe check doesn't work for varied customer data — without evals on representative data you're fumbling in the dark; off-the-shelf embeddings match structural rather than semantic similarity, so normalize first; and while experimenting, pretend tokens are free, then after a hundred runs distill the agent step into a one-shot LLM call or a smaller model.
'Pretend tokens are free while experimenting, then distill the cost down later' is the most counterintuitive and most transferable takeaway in the talk
The lessons are delivered as a spoken list — worth listening closely and writing the four down▶ Jump to 13:00Speaker · Joshua Snyder - 14:18 – 15:18Listen
What's next: automatic experiments and automatic shipping
Closes by looking ahead at where the pipeline goes: running experiments automatically, approving and shipping easy changes behind a feature flag, and learning continuously from every outcome.
The end state is a fully closed loop — signal → fix → ship → outcome fed back — and the current pipeline is only the first leg
A short spoken outlook at the end; no need to watch the screen▶ Jump to 14:18Speaker · Joshua Snyder