World Model · podcast knowledge graph

Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony

2024-10-30 · 142 min · 28 entities

episode page audio

Asserted relationships

  • → references Zyphra product
    0.77
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://github.com/Zyphra/Zamba2
  • → appeared on "The Cognitive Revolution" podcast
    0.68
    evidence rules-v4
    Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony
  • → hosted by Erik Torenberg, Nathan Labenz person
    0.55
    evidence rules-v4
    Feed author/publisher: Erik Torenberg, Nathan Labenz
  • → discusses Society & Culture concept
    0.40
    evidence rules-v4
    Feed category: Society & Culture
  • → discusses Technology concept
    0.40
    evidence rules-v4
    Feed category: Technology
  • → discusses Business concept
    0.40
    evidence rules-v4
    Feed category: Business
  • → discusses Entrepreneurship concept
    0.40
    evidence rules-v4
    Feed category: Entrepreneurship
  • → hosted by Turpentine company
    0.40
    evidence rules-v4
    Feed author/publisher: Turpentine
  • → references CognitiveRevolution website
    0.38
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://notion.com/cognitiverevolution
  • → references wandb.me website
    0.38
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://wandb.me/cr
  • → references zyphra.com website
    0.38
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://zyphra.com/
  • → references Zamba2 7b website
    0.38
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://zyphra.com/post/zamba2-7b
  • → references statespace.info website
    0.38
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://statespace.info/
  • → references Exclusiveaccess website
    0.38
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://turpentine.co/exclusiveaccess

Entities found in this episode

websites 12

  • mentioned CognitiveRevolution website
    0.45
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://notion.com/cognitiverevolution
  • mentioned wandb.me website
    0.45
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://wandb.me/cr
  • mentioned zyphra.com website
    0.45
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://zyphra.com/
  • mentioned Zamba2 7b website
    0.45
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://zyphra.com/post/zamba2-7b
  • mentioned statespace.info website
    0.45
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://statespace.info/
  • mentioned Exclusiveaccess website
    0.45
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://turpentine.co/exclusiveaccess
  • references CognitiveRevolution website
    0.38
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://notion.com/cognitiverevolution
  • references wandb.me website
    0.38
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://wandb.me/cr
  • references zyphra.com website
    0.38
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://zyphra.com/
  • references Zamba2 7b website
    0.38
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://zyphra.com/post/zamba2-7b
  • references statespace.info website
    0.38
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://statespace.info/
  • references Exclusiveaccess website
    0.38
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://turpentine.co/exclusiveaccess

concepts 8

  • mentioned Society & Culture concept
    0.50
    evidence rules-v4
    Feed category: Society & Culture
  • mentioned Technology concept
    0.50
    evidence rules-v4
    Feed category: Technology
  • mentioned Business concept
    0.50
    evidence rules-v4
    Feed category: Business
  • mentioned Entrepreneurship concept
    0.50
    evidence rules-v4
    Feed category: Entrepreneurship
  • discusses Society & Culture concept
    0.40
    evidence rules-v4
    Feed category: Society & Culture
  • discusses Technology concept
    0.40
    evidence rules-v4
    Feed category: Technology
  • discusses Business concept
    0.40
    evidence rules-v4
    Feed category: Business
  • discusses Entrepreneurship concept
    0.40
    evidence rules-v4
    Feed category: Entrepreneurship

persons 3

products 2

  • mentioned Zyphra product
    0.90
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://github.com/Zyphra/Zamba2
  • references Zyphra product
    0.77
    evidence rules-v4
    Link in episode "Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony": https://github.com/Zyphra/Zamba2

companys 2

  • mentioned Turpentine company
    0.70
    evidence rules-v4
    Feed author/publisher: Turpentine
  • hosted by Turpentine company
    0.40
    evidence rules-v4
    Feed author/publisher: Turpentine

podcasts 1

  • appeared on "The Cognitive Revolution" podcast
    0.68
    evidence rules-v4
    Training Zamba: A Hybrid Model Master Class with Zyphra's Quentin Anthony
Episode description as stored
In this episode of The Cognitive Revolution, Nathan dives deep into the world of state space models with returning co-host Jason Meaux and special guest Quentin Anthony, Head of Model Training at Zyphra. Explore the cutting-edge Zamba 2-7b model, which combines selective state space and attention mechanisms. Uncover practical insights on model training, architectural choices, and the challenges of scaling AI. From learning schedules to hybrid architectures, loss metrics to context length extension, this technical discussion covers it all. Don't miss this in-depth conversation on the future of personalized, on-device AI. Check out more about Zyphra and Jason Meaux here: Zyphra's website: https://www.zyphra.com Zamba2-7B Blog: https://www.zyphra.com/post/zamba2-7b Zamba2 GitHub: https://github.com/Zyphra/Zamba2 Tree attention: https://www.zyphra.com/post/tree-attention-topology-aware-decoding-for-long-context-attention-on-gpu-clusters Jason's Meaux Twitter: https://x.com/KamaraiCode Jason's Meaux website: https://www.statespace.info Be notified early when Turpentine's drops new publication: https://www.turpentine.co/exclusiveaccess SPONSORS: Weights & Biases RAG++: Advanced training for building production-ready RAG applications. Learn from experts to overcome LLM challenges, evaluate systematically, and integrate advanced features. Includes free Cohere credits. Visit https://wandb.me/cr to start the RAG++ course today. Shopify: Shopify is the world's leading e-commerce platform, offering a market-leading checkout system and exclusive AI apps like Quikly. Nobody does selling better than Shopify. Get a $1 per month trial at https://shopify.com/cognitive Notion: Notion offers powerful workflow and automation templates, perfect for streamlining processes and laying the groundwork for AI-driven automation. With Notion AI, you can search across thousands of documents from various platforms, generating highly relevant analysis and content tailored just for you - try it for free at https://notion.com/cognitiverevolution LMNT: LMNT is a zero-sugar electrolyte drink mix that's redefining hydration and performance. Ideal for those who fast or anyone looking to optimize their electrolyte intake. Support the show and get a free sample pack with any purchase at https://drinklmnt.com/tcr CHAPTERS: (00:00:00) Teaser (00:00:42) About the Show (00:01:05) About the Episode (00:03:09) Introducing Zyphra (00:07:28) Personalization in AI (00:12:48) State Space Models & Efficiency (Part 1) (00:19:22) Sponsors: Weights & Biases RAG++ | Shopify (00:21:26) State Space Models & Efficiency (Part 2) (00:22:23) Dense Attention to Shared Attention (00:29:41) Zyphra's Early Bet on Mamba (Part 1) (00:33:18) Sponsors: Notion | LMNT (00:36:00) Zyphra's Early Bet on Mamba (Part 2) (00:37:22) Loss vs. Model Quality (00:44:53) Emergence & Grokking (00:50:06) Loss Landscapes & Convergence (00:56:55) Sophia, Distillation & Secrets (01:09:00) Competing with Big Tech (01:23:50) The Future of Model Training (01:30:02) Deep Dive into Zamba 1 (01:34:24) Zamba 2 and Mamba 2 (01:38:56) Context Extension & Memory (01:44:04) Sequence Parallelism (01:45:44) Zamba 2 Architecture (01:53:57) Mamba Attention Hybrids (02:00:00) Lock-in Effects (02:05:32) Mamba Hybrids in Robotics (02:07:07) Ease of Use & Compatibility (02:12:10) Tree Attention vs. Ring Attention (02:22:02) Zyphra's Vision & Goals (02:23:57) Outro SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://www.linkedin.com/in/nathanlabenz/