Reindexing issue on 3.6.1

[Ian Maxon]

Hi everyone,

While staging an upgrade of a 3.1.x Gerrit instance to 3.6, I came across an issue doing an offline reindex while migrating from 3.5.2 to 3.6.1, here is the stacktrace:

------------------

logging.LoggingContextAwareRunnable@31fae5a3]]]] WARN  com.google.gerrit.server.index.change.AllChangesIndexer : Failed to index change 3008

com.google.gerrit.exceptions.StorageException: Failed to reindex change 3008 in index version 77 (current patch set = 1)

at com.google.gerrit.server.index.change.ChangeIndexer.indexImpl(ChangeIndexer.java:253)

at com.google.gerrit.server.index.change.ChangeIndexer.doIndex(ChangeIndexer.java:212)

at com.google.gerrit.server.index.change.ChangeIndexer.index(ChangeIndexer.java:208)

at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectIndexer.index(AllChangesIndexer.java:247)

at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectIndexer.lambda$call$1(AllChangesIndexer.java:236)

at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)

at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)

at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)

at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)

at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)

at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)

at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)

at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)

at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)

at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)

at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)

at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)

at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectIndexer.call(AllChangesIndexer.java:236)

at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectIndexer.call(AllChangesIndexer.java:200)

at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)

at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)

at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)

at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:95)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)

at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

Caused by: com.google.gerrit.exceptions.StorageException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Document contains at least one immense term in field="messageexact" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[84, 101, 115, 116, 32, 116, 104, 101, 32, 99, 111, 109, 112, 97, 114, 97, 98, 108, 105, 116, 121, 32, 111, 102, 32, 102, 117, 122, 122, 121]...', original message: bytes can be at most 32766 in length; got 44968

at com.google.gerrit.lucene.LuceneChangeIndex.insert(LuceneChangeIndex.java:267)

at com.google.gerrit.lucene.LuceneChangeIndex.insert(LuceneChangeIndex.java:98)

at com.google.gerrit.server.index.change.ChangeIndexer.indexImpl(ChangeIndexer.java:247)

... 29 more

Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Document contains at least one immense term in field="messageexact" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[84, 101, 115, 116, 32, 116, 104, 101, 32, 99, 111, 109, 112, 97, 114, 97, 98, 108, 105, 116, 121, 32, 111, 102, 32, 102, 117, 122, 122, 121]...', original message: bytes can be at most 32766 in length; got 44968

at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566)

at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547)

at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:88)

at com.google.gerrit.lucene.LuceneChangeIndex.insert(LuceneChangeIndex.java:262)

... 31 more

Caused by: java.lang.IllegalArgumentException: Document contains at least one immense term in field="messageexact" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[84, 101, 115, 116, 32, 116, 104, 101, 32, 99, 111, 109, 112, 97, 114, 97, 98, 108, 105, 116, 121, 32, 111, 102, 32, 102, 117, 122, 122, 121]...', original message: bytes can be at most 32766 in length; got 44968

at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:806)

at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:447)

at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:403)

at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)

at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478)

at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1571)

at com.google.gerrit.lucene.AutoCommitWriter.updateDocument(AutoCommitWriter.java:98)

at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1316)

at com.google.gerrit.lucene.AutoCommitWriter.addDocument(AutoCommitWriter.java:51)

at com.google.gerrit.lucene.AbstractLuceneIndex.lambda$insert$1(AbstractLuceneIndex.java:278)

at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)

at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)

at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)

at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)

... 3 more

Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 44968

at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:263)

at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:149)

at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:796)

... 16 more

-----------------

What could be a potential cause of this? The upgrade chain was fine up until 3.6.x, and I looked through the release notes and couldn't find anything that seemed like it would be related to something like changed message constraints or something along those lines.

Thanks,

• Ian

[David Åkerman]

Hi Ian,

We encountered the same thing: https://bugs.chromium.org/p/gerrit/issues/detail?id=16043

Best regards,

David