A hands-on audio AI class carried almost entirely by live demos, from transcription to singing
From Transcription to Live Music: Gemini's Audio Stack — Thor Schaeff, Google DeepMind · Thor Schaeff
19 min total·Actually worth watching closely: ~12 min·3 must-watch clips
- 0:14 – 2:19Listen
Laying out the map of the whole audio stack
Introduces the foundation of DeepMind's audio work: Gemini's audio understanding, which catches not only the words but the context of the conversation, emotion, several people talking over each other, and a mix of accents and languages. On the open-model side, this can also run on-device.
The starting point isn't transcription, it's comprehension — everything that follows grows on top of it.
Largely concepts and capability boundaries delivered standing up; the screen carries nothing beyond the talk itself, so listening while doing something else costs you nothing.▶ Jump to 0:14Speaker · Thor Schaeff - 2:19 – 7:16Watch
EchoScript: transcription squeezed into a single request
A small app called EchoScript does the demo: one request to the model, plus prompt instructions and an agreed output structure, and the response comes back carrying a summary, timestamps, speaker identification, language identification, English translation of non-English content and emotion labels. Where there's context, speakers get labeled with their actual names.
Structured output collapses multi-task transcription into a single call, and the dedicated transcription pipeline can go.
The point is the response on screen itself — the density of all those kinds of information crammed into one view is the most convincing part of this stretch, and skipping the visuals leaves just a verbal promise.▶ Jump to 2:19Speaker · Thor Schaeff - 7:16 – 9:35Skim
Speech generation plays by different rules: from picking a voice to directing one
Explains why only about thirty base voices remain: rather than filtering a vast voice library by gender and accent, you customize by describing the scene, the accent and the performance in natural language. The speaker shows how such a request should be organized — audio profile, scene, directing notes, examples, target text.
A voice is malleable because the model already knows what various accents and pacing sound like, which makes describing more effective than filtering.
Mostly the components of the prompt listed on screen and talked through; the structure is clear at a glance, so there's no need to follow every sentence.▶ Jump to 7:16Speaker · Thor Schaeff - 9:35 – 14:17Watch
Accent transformation and the real-time speech-to-speech model
First the same base voice is turned into an Irish accent and a Singaporean one on the spot; then comes the newly launched Gemini 3.1 Flash Live — full-duplex, speech-to-speech, ingesting text, audio and video in real time over a WebSocket and returning speech plus transcription, demoed live with camera vision as an input.
It isn't a cascade of speech-to-text, then a big model, then text-to-speech bolted together — the thinking lives right inside the audio model.
Both demos land in your ears and eyes: the accent difference has to be heard, and the camera segment has to be watched to judge the latency and how natural it feels.▶ Jump to 9:35Speaker · Thor Schaeff - 14:17 – 16:00Listen
A few pitfalls before you actually ship
Admits real-time audio development has a real learning curve, and gives two concrete limits: an accent set in the system instruction gets applied across languages, producing things like German with an Irish accent, so multilingual scenarios need their own instructions; and screen sharing and video frames currently come in at one frame per second. Also, all the Gemini APIs ship with Skills for coding agents.
Accent settings bleed across languages, and video input is capped at one frame per second — both directly shape your design.
All spoken, experience-report style; the information is in the words, so you won't miss anything without the visuals.▶ Jump to 14:17Speaker · Thor Schaeff - 16:00 – 18:10Watch
Lyria 3 and Live Jukebox: ordering a song off the menu
Introduces Lyria 3, which generates music with lyrics, split into a Clip model for 30-second pieces and a Pro model for full-length songs; then Live Jukebox hooks music generation up as a tool for the real-time conversational model — the audience describes the song they want by voice, and the model asks for details before calling the tool.
'Real-time voice agent + generative media tool call' is an architecture template you can copy straight across.
The value of this stretch is the live feel of it all running end to end — how the conversation goes back and forth, what the model asks about, and whether the song is any good. Only your own eyes and ears can confirm that.▶ Jump to 16:00 - 18:10 – 19:14Skim
Wrap-up and where to start
Closes out the talk and points to where to try things and find the related resources.
If you want to build, the entry points are in this stretch — note them down before you go.
Mostly links and a resource list on screen; a glance or a screenshot is enough, no need to listen it through.▶ Jump to 18:10Speaker · Thor Schaeff