Sonnet 5 review: I ran 64 generations to find out if it's worth it

2026-06-30 · 26 min · 22 entities

episode page audio

Asserted relationships

→ hosted by Claire Vo person

0.55

evidence rules-v5
Feed author/publisher: Claire Vo
→ discusses Technology company

0.40

evidence rules-v5
Feed category: Technology
→ references penname.co website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://penname.co/
→ references cursor.com website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://cursor.com/
→ references chatprd.ai website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://chatprd.ai/
→ references clairevo.com website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://clairevo.com/
→ references Models website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://deepmind.google/models/gemini/pro
→ references swebench.com website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://swebench.com/
→ references Claude Sonnet 5 website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://anthropic.com/news/claude-sonnet-5
→ references Introducing Gpt 5 5 website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://openai.com/index/introducing-gpt-5-5
→ references Claude Opus 4 8 website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://anthropic.com/news/claude-opus-4-8

Entities found in this episode

websites 18

mentioned penname.co website

0.45

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://penname.co/
mentioned cursor.com website

0.45

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://cursor.com/
mentioned chatprd.ai website

0.45

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://chatprd.ai/
mentioned clairevo.com website

0.45

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://clairevo.com/
mentioned Models website

0.45

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://deepmind.google/models/gemini/pro
mentioned swebench.com website

0.45

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://swebench.com/
mentioned Claude Sonnet 5 website

0.45

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://anthropic.com/news/claude-sonnet-5
mentioned Introducing Gpt 5 5 website

0.45

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://openai.com/index/introducing-gpt-5-5
mentioned Claude Opus 4 8 website

0.45

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://anthropic.com/news/claude-opus-4-8
references penname.co website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://penname.co/
references cursor.com website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://cursor.com/
references chatprd.ai website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://chatprd.ai/
references clairevo.com website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://clairevo.com/
references Models website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://deepmind.google/models/gemini/pro
references swebench.com website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://swebench.com/
references Claude Sonnet 5 website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://anthropic.com/news/claude-sonnet-5
references Introducing Gpt 5 5 website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://openai.com/index/introducing-gpt-5-5
references Claude Opus 4 8 website

0.38

evidence rules-v5
Link in episode "Sonnet 5 review: I ran 64 generations to find out if it's worth it": https://anthropic.com/news/claude-opus-4-8

persons 2

mentioned Claire Vo person

0.70

evidence rules-v5
Feed author/publisher: Claire Vo
hosted by Claire Vo person

0.55

evidence rules-v5
Feed author/publisher: Claire Vo

companys 2

mentioned Technology company

0.50

evidence rules-v5
Feed category: Technology
discusses Technology company

0.40

evidence rules-v5
Feed category: Technology

Episode description as stored

I’ve been testing every major frontier model release since the start of the year, and when Anthropic dropped Sonnet 5, I wanted more than a vibe check. I got tired of one-off tests I couldn’t repeat or compare over time, so I built something better: the How I AI Bench, a repeatable eval harness I constructed live using Claude Code while recording this episode. I ran Sonnet 5 blind against four other frontier models (Sonnet 4.6, Opus 4.8, GPT-5.5, and Gemini 3 Pro) across PRD quality, prototype generation, agentic task completion, and agent personality. The results were not what I expected. What you’ll learn: What Anthropic claims Sonnet 5 improves over Sonnet 4.6, and where the benchmark data actually backs that up How I built the How I AI Bench in under 45 minutes using Claude Code, starting from my own stored session history Why I combined human vibe scoring (70%) with LLM as judge scoring (30%) instead of trusting either alone How to set up a local HTML scoring page so you can rate AI outputs on gut feel and export those scores as JSON Which model I recommend for PRDs, which for complex prototypes, and which for chatting with an agent daily — Brought to you by: Runway —The creative AI platform for images, video and more Hyperagent —Deploy fleets of agents that handle real work — In this episode, we cover: (00:00) Sonnet 5 is out (01:55) What Anthropic claims (04:02) Why I’m done with one-off vibe checks (05:05) Building the How I AI Bench live with Claude Code (07:42) The scoring system (10:43) Agent voice eval (11:57) Quick recap (13:58) Results: The How I AI index leaderboard (21:21) What I’m improving for the next run (22:16) Generating a Claire-weighted index (23:53) Model-by-task recommendations — Tools referenced: • Claude Sonnet 5: https://www.anthropic.com/news/claude-sonnet-5 • Claude Opus 4.8: https://www.anthropic.com/news/claude-opus-4-8 • GPT-5.5 (OpenAI): https://openai.com/index/introducing-gpt-5-5/ • Gemini 3 Pro (Google DeepMind): https://deepmind.google/models/gemini/pro/ • Cursor: https://www.cursor.com/ — Other references: • SWE-bench Pro (agentic coding benchmark referenced): https://www.swebench.com/ — Where to find Claire Vo: ChatPRD: https://www.chatprd.ai/ Website: https://clairevo.com/ LinkedIn: https://www.linkedin.com/in/clairevo/ X: https://x.com/clairevo — Production and marketing by https://penname.co/ . For inquiries about sponsoring the podcast, email jordan@penname.co.