Follow-up: Can you extract and summarize a blog? (it's worse than you thought)

[Dan Russell]

Sunday, April 13, 2025

Follow-up: Can you extract and summarize a blog?

 In a moment of curiosity, 

I tried the same task as in our previous post (retrieve the last 10 blog posts from SearchResearch) with Grok and Claude... just for comparison purposes.  

I did the same query, but the results weren't much better.  

Here's the view of the sheet for Claude: 

Claude's results: Red shading means the answer is totally wrong (or a 404 error); yellow shading means it's around 50% right; green means 100% correct.  This actually isn't bad.

For contrast, here are Grok's answers: 

Grok's results.  Truly terrible.

As we've discussed, it's a good practice to iterate when you search, and the same is true when using LLMs.  

I gave both systems a second try, after giving them both the additional prompt of [be sure to give accurate links to the blog posts.  give only high quality summaries of the pages you find.]  

Both systems said that they would do better.  Bemusingly, Grok said:  "I apologize for the oversight in providing links that may not lead to valid pages. I’ve rechecked each URL by attempting to access them and verifying whether they resolve to actual, relevant blog posts on searchresearch1.blogspot.com."  

But here are the results of the second attempt, Claude first: 

Claude second attempt: About the same (9/10 correct), just a different error.

Despite protestations of "rechecking each URL,"  Grok actually performed worse, getting a solid 100% of the links wrong.  

Grok fails in a spectacular way. Nothing is correct.

I don't know about you, but I'm worried about the future of Agents when the major LLM providers can't get a simple request correct.  

The irony, of course, is that checking for valid URLs is really simple.  But the AI systems don't do it.  

SearchResearch Lessons 

1. Be very, very, very cautious about trusting LLM output.  Don't trust, but validate.  While LLMs CAN do a lot of great things, they can also make monumental errors.  

But have faith, and keep on searching the way you've learned.  

--

Daniel M. Russell 

Check out my book:  The Joy of Search (2019) 

Available in fine bookstores (and online) everywhere.