Verifying Textbook Facts
Anand S
Using LLMs to find errors is fairly hallucination-proof.
If they mess up, it's just wasted effort. If they don't, they've uncovered a major problem!
Varun fact-checked Themes in Indian History, the official NCERT Class 12 textbook.
Page-by-page, he asked Gemini to:
- Extract each claim. E.g. "Clay was locally available to the Harappans" on page 12.
- Search online for the claim. E.g. ASI site description and by Encyclopedia Britannica.
- Fact-check each claim. E.g. "Clay was locally available to the Harappans" is confirmed by both sources.
Here is his analysis and verifier code.
Before hitting rate limits, he scanned 56 pages and fact checked 45 claims from a dozen pages (detailed report). Even among this small sample, there was an interesting error on Page 22:
Only broken or useless objects would have been thrown away.
In reality, migrants abandon useful objects and ritual deposits give us intact, useful objects.
So, is this an error in the textbook worth correcting? Maybe. Worth exploring.
BTW, the original analysis hallucinated sources: "Discard and Reuse in the Ancient Near East" and "Recycling in the Bronze Age" from Cambridge. But for fact-checking, this doesn't matter. The point is to flag questionable claims for further review.
There are others:
- Did R.E.M. Wheeler take over as Director-General of the Archeological Survey of India in 1944 (according to the book) or in 1940 (according to other sources)? Worth checking the ASI records?
- [Ashoka's army was] 600,000 foot-soldiers, 30,000 cavalry and 9,000 elephants? The book says "some historians consider these accounts to be exaggerated" but the consensus is that these are almost certainly exaggerated. Worth rewording?
If LLMs find half a dozen questionable claims from a dozen pages, fact-checking textbooks seems an effective use for LLMs.
Especially when 25 million students read them every year.
I have personally suffered from errors in history textbooks.
I wrote my class 9 history exam in 1989-90. The Hindu published a series titled "This Day That Age" with news from 50 years ago. That morning's was that Germany invaded Poland with Russian aid. I proudly wrote this in my exam and lost a mark, since the history textbook clearly mentioned that Russia opposed Germany.
(That I feel indignant 36 years later tells you how sorry my life is.)