World Model · podcast knowledge graph

Grokking, Generalization Collapse, and the Dynamics of Training Deep Neural Networks with Charles Martin - #734

2025-06-05 · 85 min · episode 734 · 18 entities

episode page audio

Asserted relationships

  • 0.68
    evidence rules-v4
    Grokking, Generalization Collapse, and the Dynamics of Training Deep Neural Networks with Charles Martin - #734
  • → hosted by Sam Charrington person
    0.55
    evidence rules-v4
    Feed author/publisher: Sam Charrington
  • → founded Calculation Consulting company
    0.54
    evidence rules-v4
    founder of Calculation Consulting
  • → founded Calculation Consulting company
    0.54
    evidence rules-v4
    founder of Calculation Consulting
  • → works at Calculation Consulting company
    0.50
    evidence rules-v4
    founder of Calculation Consulting
  • → works at Calculation Consulting company
    0.50
    evidence rules-v4
    founder of Calculation Consulting
  • → discusses Science concept
    0.40
    evidence rules-v4
    Feed category: Science
  • → discusses Technology company
    0.40
    evidence rules-v4
    Feed category: Technology
  • → discusses News concept
    0.40
    evidence rules-v4
    Feed category: News
  • → discusses Tech News concept
    0.40
    evidence rules-v4
    Feed category: Tech News
  • → hosted by TWIML company
    0.40
    evidence rules-v4
    Feed author/publisher: TWIML

Entities found in this episode

companys 9

  • mentioned TWIML company
    0.70
    evidence rules-v4
    Feed author/publisher: TWIML
  • mentioned Calculation Consulting company
    0.68
    evidence rules-v4
    founder of Calculation Consulting
  • founded Calculation Consulting company
    0.54
    evidence rules-v4
    founder of Calculation Consulting
  • founded Calculation Consulting company
    0.54
    evidence rules-v4
    founder of Calculation Consulting
  • mentioned Technology company
    0.50
    evidence rules-v4
    Feed category: Technology
  • works at Calculation Consulting company
    0.50
    evidence rules-v4
    founder of Calculation Consulting
  • works at Calculation Consulting company
    0.50
    evidence rules-v4
    founder of Calculation Consulting
  • discusses Technology company
    0.40
    evidence rules-v4
    Feed category: Technology
  • hosted by TWIML company
    0.40
    evidence rules-v4
    Feed author/publisher: TWIML

concepts 5

  • mentioned Science concept
    0.50
    evidence rules-v4
    Feed category: Science
  • mentioned Tech News concept
    0.50
    evidence rules-v4
    Feed category: Tech News
  • discusses Science concept
    0.40
    evidence rules-v4
    Feed category: Science
  • discusses News concept
    0.40
    evidence rules-v4
    Feed category: News
  • discusses Tech News concept
    0.40
    evidence rules-v4
    Feed category: Tech News

persons 3

  • mentioned Charles Martin person
    0.72
    evidence rules-v4
    Grokking, Generalization Collapse, and the Dynamics of Training Deep Neural Networks with Charles Martin - #734
  • mentioned Sam Charrington person
    0.70
    evidence rules-v4
    Feed author/publisher: Sam Charrington
  • hosted by Sam Charrington person
    0.55
    evidence rules-v4
    Feed author/publisher: Sam Charrington

podcasts 1

Episode description as stored
Today, we're joined by Charles Martin, founder of Calculation Consulting, to discuss Weight Watcher, an open-source tool for analyzing and improving Deep Neural Networks (DNNs) based on principles from theoretical physics. We explore the foundations of the Heavy-Tailed Self-Regularization (HTSR) theory that underpins it, which combines random matrix theory and renormalization group ideas to uncover deep insights about model training dynamics. Charles walks us through WeightWatcher’s ability to detect three distinct learning phases—underfitting, grokking, and generalization collapse—and how its signature “layer quality” metric reveals whether individual layers are underfit, overfit, or optimally tuned. Additionally, we dig into the complexities involved in fine-tuning models, the surprising correlation between model optimality and hallucination, the often-underestimated challenges of search relevance, and their implications for RAG. Finally, Charles shares his insights into real-world applications of generative AI and his lessons learned from working in the field. The complete show notes for this episode can be found at https://twimlai.com/go/734.