World Model · podcast knowledge graph

How to Find the Agent Failures Your Evals Miss with Scott Clark - #767

2026-05-07 · 53 min · episode 767 · 16 entities

episode page audio

Asserted relationships

  • 0.68
    evidence rules-v4
    How to Find the Agent Failures Your Evals Miss with Scott Clark - #767
  • → hosted by Sam Charrington person
    0.55
    evidence rules-v4
    Feed author/publisher: Sam Charrington
  • → works at Distributional company
    0.50
    evidence rules-v4
    CEO of Distributional
  • → works at Distributional company
    0.50
    evidence rules-v4
    CEO of Distributional
  • → discusses Science concept
    0.40
    evidence rules-v4
    Feed category: Science
  • → discusses Technology company
    0.40
    evidence rules-v4
    Feed category: Technology
  • → discusses News concept
    0.40
    evidence rules-v4
    Feed category: News
  • → discusses Tech News concept
    0.40
    evidence rules-v4
    Feed category: Tech News
  • → hosted by TWIML company
    0.40
    evidence rules-v4
    Feed author/publisher: TWIML

Entities found in this episode

companys 7

  • mentioned TWIML company
    0.70
    evidence rules-v4
    Feed author/publisher: TWIML
  • mentioned Distributional company
    0.62
    evidence rules-v4
    CEO of Distributional
  • mentioned Technology company
    0.50
    evidence rules-v4
    Feed category: Technology
  • works at Distributional company
    0.50
    evidence rules-v4
    CEO of Distributional
  • works at Distributional company
    0.50
    evidence rules-v4
    CEO of Distributional
  • discusses Technology company
    0.40
    evidence rules-v4
    Feed category: Technology
  • hosted by TWIML company
    0.40
    evidence rules-v4
    Feed author/publisher: TWIML

concepts 5

  • mentioned Science concept
    0.50
    evidence rules-v4
    Feed category: Science
  • mentioned Tech News concept
    0.50
    evidence rules-v4
    Feed category: Tech News
  • discusses Science concept
    0.40
    evidence rules-v4
    Feed category: Science
  • discusses News concept
    0.40
    evidence rules-v4
    Feed category: News
  • discusses Tech News concept
    0.40
    evidence rules-v4
    Feed category: Tech News

persons 3

  • mentioned Scott Clark person
    0.72
    evidence rules-v4
    How to Find the Agent Failures Your Evals Miss with Scott Clark - #767
  • mentioned Sam Charrington person
    0.70
    evidence rules-v4
    Feed author/publisher: Sam Charrington
  • hosted by Sam Charrington person
    0.55
    evidence rules-v4
    Feed author/publisher: Sam Charrington

podcasts 1

Episode description as stored
In this episode, Scott Clark, co-founder and CEO of Distributional, joins us to explore how teams can reliably operate and improve complex LLM systems and agents in production. Scott introduces a Maslow’s hierarchy of observability: telemetry for logging, monitoring for known signals, and post-production or online analytics to surface unknown unknowns. We dig into examples of real-world failures Scott’s team has seen in production systems, such as “lazy” tool-use hallucinations that standard evals miss, and how mapping traces into vector fingerprints enables clustering and topic discovery to uncover emergent behaviors. Scott explains how analytics can feed the data flywheel by generating evals, guardrails, and training data, and why online, adaptive approaches are essential for non-stationary models. We also touch on practical how-to’s such as instrumentation with OpenTelemetry, the GenAI semantic conventions, and the role of dedicated analytics tools. The complete show notes for this episode can be found at https://twimlai.com/go/767.