SearchResearch (5/1/25): How good are those AI summaries anyway? Let us know!
Dan Russell
Thursday, May 1, 2025
SearchResearch (5/1/25): How good are those AI summaries anyway? Let us know!
I don't know about you...
 |
| P/C [slide showing someone using AI for summarization] Gemini (May 1, 2025) |
... but while the entire world seems to be having ecstatic paroxysms about the incredible capabilities of generative AI, my experience has been a little less wonderful.
On more than one occasion I've spent hours trying to track down a factual claim, a citation, or a reference that some AI system made, only to find out that it's bogus or purely hallucinatory.
That's my experience. Don't get me wrong--there's a lot of good stuff in the AI that's deployed, but I find myself constantly having to check the output of an AI to make sure that the bad stuff doesn't overwhelm the good stuff.
For this week, I'd love to hear your stories about using AI and then having to fact-check it, only to discover that things went way off.
Let's ask a very specific Challenge for this week:
1. How has your experience of using AI for summarization worked out?
I really want to hear your stories. Let me give you one example.
I asked Gemini 2.0 (Flash) to [summarize this article and give citations to the claims that you make in the summary].
It seemed to do a good job, but one of the citations was to an article by a famous author in a well-known publication, on a date that was plausible, with a title that is very consistent with everything he's written over the past few years.
But try as I might, I could NOT find that damn article. I eventually went to the journal's searchable archive website and found that no article by that name was ever published. The whole thing wasted a full hour of my time.
I definitely do NOT want to be in the situation of the lawyers for Mike Lindell who submitted legal briefs with LLM hallucinated citations.
So I want to hear your stories about using AI to summarize other texts. How's that working out for you?
Share your AI summarization stories in the comments below so we can all learn from your work.
What's your critical and careful analysis of the quality of AI as a text summarizing tool?
True stories only, please. And definitely nothing written by AIs.
I'll summarize your stories--lovingly, by my human eyes, hands, and brains--and let you know what the SRS crew has to say about this.
Keep searching.
