On 19 October 2026 we searched 17 channels in the archive for how teams know an agent works, and read 530 comments. Twelve of those channels had answers worth citing. The most liked answer was another question. Under Nick Saraev's video where Claude Code reads his genome, 131 people liked this reply: “Plot twist: AI hallucinated half of it, and now you are worse off. ... how could you know?” (131).
Builders put it more bluntly. “If you get 100% accuracy with an LLM based system, it means you haven't tested enough” (74), under YC's episode on vertical AI agents.
The teams that answer the question on camera describe one order of work. They read the failures, write a pass/fail check for each one, and test the checker. Then they build a private benchmark and keep watching after launch. They say it on Lenny's Podcast, AI Engineer, YC, a16z and Latent Space, and on the channels of Peter Yang, Nick Saraev and Nate Herk.
Read the failures first
Jake Heller sold Casetext to Thomson Reuters for $650M, and he says where AI products go wrong. “The hard part frankly isn't building it. The hard part is getting it right… And this is the thing that I see most people not doing because they build like demo level stuff that frankly is like 60 to 70% accurate” (15:15).
Hamel Husain and Shreya Shankar run the most popular AI eval course online, with more than 4,500 students by Peter Yang's count (0:00). On Lenny's Podcast they open the traces of an apartment-leasing assistant and take notes. The rule for a note: “just write down the first thing that you see that's wrong, the most upstream error. Don't worry about all the errors” (21:26). The mistake to avoid at this stage: “people are like, let me automate this with an LLM” (24:29). You stop when new traces stop showing new kinds of failure. “A lot of people realize like, oh, okay, like I only need to do 40. I only need to do 60. Actually, I only need to do like 15” (30:35).