Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)

2025-09-25 · 107 min · 52 entities

episode page audio

Asserted relationships

→ references The Wire website

0.77

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://en.wikipedia.org/wiki/The_Wire
→ references Pachinko National Book Award Finalist book

0.77

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Pachinko-National-Book-Award-Finalist/dp/1455563935
→ references Occam'S Razor person

0.77

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://en.wikipedia.org/wiki/Occam%27s_razor
→ references Apple China Capture Greatest Company book

0.77

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Apple-China-Capture-Greatest-Company/dp/1668053373
→ references Artificial Intelligence Modern Approach Global book

0.77

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Artificial-Intelligence-Modern-Approach-Global/dp/1292401133
→ references Sh Reya person

0.77

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://x.com/sh_reya
→ references Machine Learning Tom M Mitchell book

0.77

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Machine-Learning-Tom-M-Mitchell/dp/1259096955
→ hosted by Lenny Rachitsky person

0.55

evidence rules-v4
Feed author/publisher: Lenny Rachitsky
→ recommended Pachinko National Book Award Finalist book

0.50

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Pachinko-National-Book-Award-Finalist/dp/1455563935
→ recommended Pachinko National Book Award Finalist book

0.50

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Pachinko-National-Book-Award-Finalist/dp/1455563935
→ recommended Apple China Capture Greatest Company book

0.50

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Apple-China-Capture-Greatest-Company/dp/1668053373
→ recommended Apple China Capture Greatest Company book

0.50

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Apple-China-Capture-Greatest-Company/dp/1668053373
→ recommended Artificial Intelligence Modern Approach Global book

0.50

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Artificial-Intelligence-Modern-Approach-Global/dp/1292401133
→ recommended Artificial Intelligence Modern Approach Global book

0.50

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Artificial-Intelligence-Modern-Approach-Global/dp/1292401133
→ recommended Machine Learning Tom M Mitchell book

0.50

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Machine-Learning-Tom-M-Mitchell/dp/1259096955
→ recommended Machine Learning Tom M Mitchell book

0.50

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Machine-Learning-Tom-M-Mitchell/dp/1259096955
→ discusses Technology company

0.40

evidence rules-v4
Feed category: Technology
→ discusses Business concept

0.40

evidence rules-v4
Feed category: Business
→ discusses Entrepreneurship concept

0.40

evidence rules-v4
Feed category: Entrepreneurship
→ references penname.co website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://penname.co/
→ references cursor.com website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://cursor.com/
→ references Claude Code person

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://anthropic.com/claude-code
→ references statsig.com website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://statsig.com/
→ references sh-reya.com website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://sh-reya.com/
→ references hamel.dev website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://hamel.dev/
→ references Building Eval Systems That Improve company

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://lennysnewsletter.com/p/building-eval-systems-that-improve
→ references mercor.com website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://mercor.com/
→ references nurtureboss.io website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://nurtureboss.io/
→ references braintrust.dev website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://braintrust.dev/
→ references julius.ai website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://julius.ai/

Entities found in this episode

websites 20

mentioned The Wire website

0.90

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://en.wikipedia.org/wiki/The_Wire
references The Wire website

0.77

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://en.wikipedia.org/wiki/The_Wire
mentioned penname.co website

0.45

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://penname.co/
mentioned cursor.com website

0.45

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://cursor.com/
mentioned statsig.com website

0.45

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://statsig.com/
mentioned sh-reya.com website

0.45

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://sh-reya.com/
mentioned hamel.dev website

0.45

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://hamel.dev/
mentioned mercor.com website

0.45

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://mercor.com/
mentioned nurtureboss.io website

0.45

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://nurtureboss.io/
mentioned braintrust.dev website

0.45

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://braintrust.dev/
mentioned julius.ai website

0.45

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://julius.ai/
references penname.co website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://penname.co/
references cursor.com website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://cursor.com/
references statsig.com website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://statsig.com/
references sh-reya.com website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://sh-reya.com/
references hamel.dev website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://hamel.dev/
references mercor.com website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://mercor.com/
references nurtureboss.io website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://nurtureboss.io/
references braintrust.dev website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://braintrust.dev/
references julius.ai website

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://julius.ai/

books 16

mentioned Pachinko National Book Award Finalist book

0.90

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Pachinko-National-Book-Award-Finalist/dp/1455563935
mentioned Apple China Capture Greatest Company book

0.90

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Apple-China-Capture-Greatest-Company/dp/1668053373
mentioned Artificial Intelligence Modern Approach Global book

0.90

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Artificial-Intelligence-Modern-Approach-Global/dp/1292401133
mentioned Machine Learning Tom M Mitchell book

0.90

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Machine-Learning-Tom-M-Mitchell/dp/1259096955
references Pachinko National Book Award Finalist book

0.77

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Pachinko-National-Book-Award-Finalist/dp/1455563935
references Apple China Capture Greatest Company book

0.77

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Apple-China-Capture-Greatest-Company/dp/1668053373
references Artificial Intelligence Modern Approach Global book

0.77

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Artificial-Intelligence-Modern-Approach-Global/dp/1292401133
references Machine Learning Tom M Mitchell book

0.77

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Machine-Learning-Tom-M-Mitchell/dp/1259096955
recommended Pachinko National Book Award Finalist book

0.50

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Pachinko-National-Book-Award-Finalist/dp/1455563935
recommended Pachinko National Book Award Finalist book

0.50

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Pachinko-National-Book-Award-Finalist/dp/1455563935
recommended Apple China Capture Greatest Company book

0.50

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Apple-China-Capture-Greatest-Company/dp/1668053373
recommended Apple China Capture Greatest Company book

0.50

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Apple-China-Capture-Greatest-Company/dp/1668053373
recommended Artificial Intelligence Modern Approach Global book

0.50

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Artificial-Intelligence-Modern-Approach-Global/dp/1292401133
recommended Artificial Intelligence Modern Approach Global book

0.50

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Artificial-Intelligence-Modern-Approach-Global/dp/1292401133
recommended Machine Learning Tom M Mitchell book

0.50

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Machine-Learning-Tom-M-Mitchell/dp/1259096955
recommended Machine Learning Tom M Mitchell book

0.50

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://amazon.com/Machine-Learning-Tom-M-Mitchell/dp/1259096955

persons 8

mentioned Occam'S Razor person

0.90

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://en.wikipedia.org/wiki/Occam%27s_razor
mentioned Sh Reya person

0.90

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://x.com/sh_reya
references Occam'S Razor person

0.77

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://en.wikipedia.org/wiki/Occam%27s_razor
references Sh Reya person

0.77

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://x.com/sh_reya
mentioned Lenny Rachitsky person

0.70

evidence rules-v4
Feed author/publisher: Lenny Rachitsky
hosted by Lenny Rachitsky person

0.55

evidence rules-v4
Feed author/publisher: Lenny Rachitsky
mentioned Claude Code person

0.45

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://anthropic.com/claude-code
references Claude Code person

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://anthropic.com/claude-code

companys 4

mentioned Technology company

0.50

evidence rules-v4
Feed category: Technology
mentioned Building Eval Systems That Improve company

0.45

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://lennysnewsletter.com/p/building-eval-systems-that-improve
discusses Technology company

0.40

evidence rules-v4
Feed category: Technology
references Building Eval Systems That Improve company

0.38

evidence rules-v4
Link in episode "Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)": https://lennysnewsletter.com/p/building-eval-systems-that-improve

concepts 4

mentioned Business concept

0.50

evidence rules-v4
Feed category: Business
mentioned Entrepreneurship concept

0.50

evidence rules-v4
Feed category: Entrepreneurship
discusses Business concept

0.40

evidence rules-v4
Feed category: Business
discusses Entrepreneurship concept

0.40

evidence rules-v4
Feed category: Entrepreneurship

Episode description as stored

Hamel Husain and Shreya Shankar teach the world’s most popular course on AI evals and have trained over 2,000 PMs and engineers (including many teams at OpenAI and Anthropic). In this conversation, they demystify the process of developing effective evals, walk through real examples, and share practical techniques that’ll help you improve your AI product. What you’ll learn: 1. WTF evals are 2. Why they’ve become the most important new skill for AI product builders 3. A step-by-step walkthrough of how to create an effective eval 4. A deep dive into error analysis, open coding, and axial coding 5. Code-based evals vs. LLM-as-judge 6. The most common pitfalls and how to avoid them 7. Practical tips for implementing evals with minimal time investment (30 minutes per week after initial setup) 8. Insight into the debate between “vibes” and systematic evals — Brought to you by: Fin —The #1 AI agent for customer service Dscout —The UX platform to capture insights at every stage: from ideation to production Mercury —The art of simplified finances — Where to find Shreya Shankar • X: https://x.com/sh_reya • LinkedIn: https://www.linkedin.com/in/shrshnk/ • Website: https://www.sh-reya.com/ • Maven course: https://bit.ly/4myp27m — Where to find Hamel Husain • X: https://x.com/HamelHusain • LinkedIn: https://www.linkedin.com/in/hamelhusain/ • Website: https://hamel.dev/ • Maven course: https://bit.ly/4myp27m — In this episode, we cover: (00:00) Introduction to Hamel and Shreya (04:57) What are evals? (09:56) Demo: Examining real traces from a property management AI assistant (16:51) Writing notes on errors (23:54) Why LLMs can’t replace humans in the initial error analysis (25:16) The concept of a “benevolent dictator” in the eval process (28:07) Theoretical saturation: when to stop (31:39) Using axial codes to help categorize and synthesize error notes (44:39) The results (46:06) Building an LLM-as-judge to evaluate specific failure modes (48:31) The difference between code-based evals and LLM-as-judge (52:10) Example: LLM-as-judge (54:45) Testing your LLM judge against human judgment (01:00:51) Why evals are the new PRDs for AI products (01:05:09) How many evals you actually need (01:07:41) What comes after evals (01:09:57) The great evals debate (1:15:15) Why dogfooding isn’t enough for most AI products (01:18:23) OpenAI’s Statsig acquisition (1:23:02) The Claude Code controversy and the importance of context (01:24:13) Common misconceptions around evals (1:22:28) Tips and tricks for implementing evals effectively (1:30:37) The time investment (1:33:38) Overview of their comprehensive evals course (1:37:57) Lightning round and final thoughts — LLM Log Open Codes Analysis Prompt: Please analyze the following CSV file. There is a metadata field which has an nested field called z_note that contains open codes for analysis of LLM logs that we are conducting. Please extract all of the different open codes. From the _note field, propose 5-6 categories that we can create axial codes from. — Referenced: • Building eval systems that improve your AI product: https://www.lennysnewsletter.com/p/building-eval-systems-that-improve • Mercor: https://mercor.com/ • Brendan Foody on LinkedIn: https://www.linkedin.com/in/brendan-foody-2995ab10b • Nurture Boss: https://nurtureboss.io/ • Braintrust: https://www.braintrust.dev/ • Andrew Ng on X: https://x.com/andrewyng • Carrying Out Error Analysis: https://www.youtube.com/watch?v=JoAxZsdw_3w • Julius AI: https://julius.ai/ • Brendan Foody on X—“evals are the new PRDs”: https://x.com/BrendanFoody/status/1939764763485171948 • Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences: https://dl.acm.org/doi/abs/10.1145/3654777.3676450 • Lenny’s post on X about evals: https://x.com/lennysan/status/1909636749103599729 • Statsig: https://statsig.com/ • Claude Code: https://www.anthropic.com/claude-code • Cursor: https://cursor.com/ • Occam’s razor: https://en.wikipedia.org