World Model · podcast knowledge graph

How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen

2026-10-01 · 70 min · 19 entities

episode page audio

Asserted relationships

  • → references Mel Scale person
    0.77
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": https://en.wikipedia.org/wiki/Mel_scale
  • → references Victor Zue person
    0.77
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": https://en.wikipedia.org/wiki/Victor_Zue
  • → references Cocktail Party Effect person
    0.77
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": https://en.wikipedia.org/wiki/Cocktail_party_effect
  • → references Speaker Diarisation person
    0.77
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": https://en.wikipedia.org/wiki/Speaker_diarisation
  • → works at PolyAI company
    0.50
    evidence rules-v5
    CTO of PolyAI
  • → works at PolyAI company
    0.50
    evidence rules-v5
    CTO of PolyAI
  • → discusses Technology company
    0.40
    evidence rules-v5
    Feed category: Technology
  • → references BitterLesson website
    0.38
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": http://incompleteideas.net/IncIdeas/BitterLesson.html
  • → references poly.ai website
    0.38
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": https://poly.ai/

Entities found in this episode

persons 8

  • mentioned Mel Scale person
    0.90
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": https://en.wikipedia.org/wiki/Mel_scale
  • mentioned Victor Zue person
    0.90
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": https://en.wikipedia.org/wiki/Victor_Zue
  • mentioned Cocktail Party Effect person
    0.90
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": https://en.wikipedia.org/wiki/Cocktail_party_effect
  • mentioned Speaker Diarisation person
    0.90
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": https://en.wikipedia.org/wiki/Speaker_diarisation
  • references Mel Scale person
    0.77
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": https://en.wikipedia.org/wiki/Mel_scale
  • references Victor Zue person
    0.77
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": https://en.wikipedia.org/wiki/Victor_Zue
  • references Cocktail Party Effect person
    0.77
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": https://en.wikipedia.org/wiki/Cocktail_party_effect
  • references Speaker Diarisation person
    0.77
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": https://en.wikipedia.org/wiki/Speaker_diarisation

companys 5

  • mentioned PolyAI company
    0.62
    evidence rules-v5
    CTO of PolyAI
  • mentioned Technology company
    0.50
    evidence rules-v5
    Feed category: Technology
  • works at PolyAI company
    0.50
    evidence rules-v5
    CTO of PolyAI
  • works at PolyAI company
    0.50
    evidence rules-v5
    CTO of PolyAI
  • discusses Technology company
    0.40
    evidence rules-v5
    Feed category: Technology

websites 4

  • mentioned BitterLesson website
    0.45
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": http://incompleteideas.net/IncIdeas/BitterLesson.html
  • mentioned poly.ai website
    0.45
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": https://poly.ai/
  • references BitterLesson website
    0.38
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": http://incompleteideas.net/IncIdeas/BitterLesson.html
  • references poly.ai website
    0.38
    evidence rules-v5
    Link in episode "How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen": https://poly.ai/

podcasts 1

concepts 1

  • mentioned Shawn Wen concept
    0.42
    evidence rules-v5
    How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen
Episode description as stored
Tsung-Hsien (Shawn) Wen, CTO of PolyAI, tells Tim Scarfe why voice agents are harder than text agents. Voice adds time, and a good conversation depends on adapting to the person on the line, not just on reasoning to the best answer. Shawn describes an audio-native model (Dialog-RSN-1) that first predicts a turn-taking signal, then replies in text with citations, and writes the transcript last so enterprises can audit it.Along the way: training on real, noisy calls with synthetic noise added, and why over-cleaned audio made the new model worse. Latency, and what a voice agent should do while it thinks. Why a voice with a hint of regional accent beats a generic one. Why public benchmarks fall short for voice, why enterprises want to own their agent harness, and whether behaviour belongs in the harness or in the weights.The last stretch is about working with agents: cognitive debt, the shift from producing content to checking it, Wispr Flow, building tools that agents can use, and whether slop is in the eye of the reader.This episode was produced in partnership with PolyAI.https://poly.ai CHAPTERS0:00 Why voice agents are harder than text4:27 What enterprises want, and why PolyAI built its own model9:04 How an audio-native voice model works15:21 Training data, spectrograms and synthetic noise20:36 The cocktail party problem and the future of turn-taking25:21 Latency, adaptive reasoning and keeping callers' trust31:33 Voices, personality and the uncanny valley36:32 How do you benchmark a voice agent?42:00 Harness engineering and owning the intelligence45:09 Well-specified problems and auditable agents49:46 Weight adaptation and cognitive debt56:31 Agents at work: Wispr Flow, voice and tool building1:02:50 The next decade of voice, and what counts as slopREFERENCESThe Bitter Lesson: http://www.incompleteideas.net/IncIdeas/BitterLesson.html [9:05]Retrieval-augmented generation: https://arxiv.org/abs/2005.11401 [13:23]Mel scale: https://en.wikipedia.org/wiki/Mel_scale [17:16]Victor Zue: https://en.wikipedia.org/wiki/Victor_Zue [17:45]Cocktail party effect: https://en.wikipedia.org/wiki/Cocktail_party_effect [20:39]Speaker diarisation: https://en.wikipedia.org/wiki/Speaker_diarisation [21:16]