PSA: Do not believe citations created by Gemini (Bard)
Dan Russell
Saturday, February 10, 2024
PSA: Do not believe citations created by Gemini (Bard)
As you probably know,
... I'm writing a book. That means every day I'm looking things up, checking into the backstory so I can be sure that what I'm writing is accurate, up-to-the-minute, and full of the zesty and intriguing insider information that you've come to expect from SearchResearch.
Today I was looking up the difference in behavior between the Eurasian Tree sparrow (Passer montanus) and the ordinary House sparrow (P. domesticus). Oddly, it turns out that even though they're very similar, the Eurasian Tree sparrow has a much smaller range that the House sparrow. (Buy the book when it comes out to learn why anyone might be interested in this strange-but-true fact.) For two very similar birds, the question is obvious: Why?
In the process of searching for the answer, I thought that I should give Gemini (Bard) a try. So I posed this question to it:
[ why have house sparrows expanded their range dramatically since being introduced into the US, while Eurasian tree sparrows have not? They're so similar, you'd think they would expand at a similar rate. ]
And it gave a reasonable answer. At least it looks reasonable.
So I asked it to give me me a few citations:
[ can you suggest further reading in the scientific literature about the differences in range expansion?]
No surprise, it gave me a set of reasonable looking titles of papers I should read. In fact, these titles look great! I need to read these articles, and what's even better, there are actual links to the articles.
What could go wrong? Here's what I saw (I added the red Xs):
To start with, NONE of those papers actually exist. That's a serious deal-breaker.
Yes, I checked everywhere, but no, I could not find any of these papers in the scholarly literature or even the trash heap. They just don't exist.
And those links? They go to actual papers, but not the papers cited, and in three cases, the links went to papers that weren't about sparrows at all but about random other topics!
Which brings up the basic question--if Gemini can't get the citations and links correct, should I really trust the answer it gave to my first question?
My answer: I'm not going to do so. It seems drunk.
Just for giggles, I gave the same questions to ChatGPT-4. Its answer to the first question was very similar to Gemini's, but to its credit, when I asked for citations, it replied with a hedge, listing places I could go look on my own, but NOT giving any direct citations. At least the journals are real journals. (Yes, there is an journal called The Auk, it's been published since 1884).
I thought that ChatGPT would take the day here, but then, at the end of its answer, it included this very odd suggestion:
Did I read that right? "... here are some hypothetical examples of articles or chapters... though these exact titles may not exist."
Maybe that's helpful? (It certainly suggests some key terms that might be useful in doing regular searches.)
But "hypothetical examples"? I'm not sure this is really much of an addition to my search task.
My bottom line? Don't trust Gemini to tell you any actual citations... it just makes stuff up. (And that, my friends, makes me very concerned about the quality of the summaries it makes to complex or subtle questions...)
Keep searching! Your SRS skills are still relevant!
