incremental reindexing error

[verr...@uic.edu]

Hi all --

I've been using XTF for almost 2 years now for a mixed collection of EAD and images with xml metadata files.  We recently had an odd error and I'm not entirely sure where it came from or where to look to prevent it from happening again. If anyone has any thoughts or leads please let me know.

We have not made any changes to the code recently. However, users are perpetually adding more content and performing incremental reindexes.  One day users started getting this error:

*** Error: class java.lang.IllegalStateException

java.lang.IllegalStateException: doc counts differ for segment _3uy: fieldsReader shows 89728 but segmentInfo shows 445030

at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:164)

at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:140)

at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:126)

at org.apache.lucene.index.IndexReader$1.doBody(IndexReader.java:155)

at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:579)

at org.apache.lucene.index.IndexReader.open(IndexReader.java:147)

at org.apache.lucene.index.IndexReader.open(IndexReader.java:142)

at org.cdlib.xtf.textIndexer.XMLTextProcessor.openIdxForReading(XMLTextProcessor.java:3843)

at org.cdlib.xtf.textIndexer.XMLTextProcessor.open(XMLTextProcessor.java:550)

at org.cdlib.xtf.textIndexer.SrcTreeProcessor.open(SrcTreeProcessor.java:142)

at org.cdlib.xtf.textIndexer.TextIndexer.doIndexing(TextIndexer.java:474)

at org.cdlib.xtf.textIndexer.TextIndexer.main(TextIndexer.java:339)

Once they started getting it incremental reindexes were broken.  We performed a clean reindex and everything seemed fine again, incremental reindexes were back online.  Then a few days later it happened again, we did another clean index and it's been working ever since.  That was about 2 weeks ago and I know users have been performing incremental reindexes since then and no more weird error.

Any thoughts?  Could it be due to resource limitations?  Any deeper insight into what's going on other than, the index got messed up? I saw a post for 2008 or something that talked about this problem with a clean index but that's not what we're experiencing.

Thanks!

Esther Verreau

UIC Library

[Martin Haye]

Hi Esther,

I haven't seen that error before. If I were to hazard a guess I'd say somehow there's a file being corrupted. Is it possible you have two separate processes trying to write to the same index at the same time?

--Martin

--
You received this message because you are subscribed to the Google Groups "XTF Users List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xtf-user+unsubscribe@googlegroups.com.
To post to this group, send email to xtf-...@googlegroups.com.
Visit this group at https://groups.google.com/group/xtf-user.
For more options, visit https://groups.google.com/d/optout.