Upgrade to Gerrit 3.10.1 - error in full reindex - need help ASAP

[Guy Levkowitz]

Hey

I'm doing upgrade from Gerrit 3.9.5  to 3.10.1 

I have run :
java -jar release.war init -d . --batch --delete-caches --install-all-plugins --no-auto-start
java -jar /opt/gerrit/bin/gerrit.war reindex

it is more than 3 hours that it runs and i can see in the output on the reindex  - it is normal output  ? :  

[2024-09-02 11:16:03,094] [Index-Batch-11[Index changes slice 1/9 of project sp_config_vpn1][sp_config_vpn1-1]] WARN  com.google.gerrit.server.index.change.AllChangesIndexer : Failed to index change 1145827
com.google.gerrit.exceptions.StorageException: Failed to reindex change 1145827 in index version 86 (current patch set = 2)
        at com.google.gerrit.server.index.change.ChangeIndexer.indexImpl(ChangeIndexer.java:343)
        at com.google.gerrit.server.index.change.ChangeIndexer.doIndex(ChangeIndexer.java:297)
        at com.google.gerrit.server.index.change.ChangeIndexer.index(ChangeIndexer.java:293)
        at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.lambda$new$1(AllChangesIndexer.java:284)
        at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.index(AllChangesIndexer.java:325)
        at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.lambda$call$3(AllChangesIndexer.java:311)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
        at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
        at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.call(AllChangesIndexer.java:311)
        at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.call(AllChangesIndexer.java:264)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
        at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:703)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: com.google.gerrit.exceptions.StorageException: java.util.concurrent.ExecutionException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
        at com.google.gerrit.lucene.LuceneChangeIndex.insert(LuceneChangeIndex.java:242)
        at com.google.gerrit.lucene.LuceneChangeIndex.insert(LuceneChangeIndex.java:104)
        at com.google.gerrit.server.index.change.ChangeIndexer.indexImpl(ChangeIndexer.java:337)
        ... 29 more
Caused by: java.util.concurrent.ExecutionException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:594)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:573)
        at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91)
        at com.google.gerrit.lucene.LuceneChangeIndex.insert(LuceneChangeIndex.java:239)
        ... 31 more
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:908)
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:921)
        at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1542)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1830)
        at com.google.gerrit.lucene.AutoCommitWriter.updateDocument(AutoCommitWriter.java:98)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1470)
        at com.google.gerrit.lucene.AutoCommitWriter.addDocument(AutoCommitWriter.java:51)
        at com.google.gerrit.lucene.AbstractLuceneIndex.lambda$insert$1(AbstractLuceneIndex.java:291)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
        at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
        ... 3 more
Caused by: java.lang.OutOfMemoryError: Java heap space

[Sven Selberg]

The index operation needs more heap:

[Matthias Sohn]

Looks like reindexing ran out of heap space. You can try to increase the heap size and/or reduce the number of reindexing threads.

I'd start with settings similar to the ones you use to run your gerrit server.

-Matthias 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/d0ba94b7-d70c-4dfa-860e-fdf78c560340n%40googlegroups.com.

[Guy Levkowitz]

Do you think to cancel the reindex in the middle (CTL+C)? 

it runs for more than 3.5 hours 
you mean Increase the heapLimit ? 

/opt/gerrit/etc/gerrit.config:    heapLimit = 64g

ב-יום שני, 2 בספטמבר 2024 בשעה 11:25:44 UTC+3, Matthias Sohn כתב/ה:

[Sven Selberg]

On Monday, September 2, 2024 at 10:44:11 AM UTC+2 Guy Levkowitz wrote:

Do you think to cancel the reindex in the middle (CTL+C)? 

it runs for more than 3.5 hours 
you mean Increase the heapLimit ? 

/opt/gerrit/etc/gerrit.config:    heapLimit = 64g

This setting is only read if/when you start gerrit using the bin/gerrit.sh script:
"These settings are applied only if Gerrit is started as the container process through Gerrit’s 'gerrit.sh' rc.d compatible wrapper script."

 

Do you monitor the java heap so that you can investigate heap usage at the time of the index operation?

[Guy Levkowitz]

Hey

well it look some time but it ended and looks fine

the question is should I increase the 
/opt/gerrit/etc/gerrit.config:    heapLimit = 64g
to more ? 

the server has 98GB

ב-יום שני, 2 בספטמבר 2024 בשעה 11:56:15 UTC+3, Sven Selberg כתב/ה:

[Sven Selberg]

On Monday, September 2, 2024 at 12:21:18 PM UTC+2 Guy Levkowitz wrote:

Hey

well it look some time but it ended and looks fine

the question is should I increase the 
/opt/gerrit/etc/gerrit.config:    heapLimit = 64g
to more ? 

As I wrote I don't think this setting has any effect on the offline reindex.

[Matthias Sohn]

On Mon, Sep 2, 2024 at 2:07 PM Sven Selberg <sven.s...@axis.com> wrote:

On Monday, September 2, 2024 at 12:21:18 PM UTC+2 Guy Levkowitz wrote:

Hey

well it look some time but it ended and looks fine

the question is should I increase the 
/opt/gerrit/etc/gerrit.config:    heapLimit = 64g
to more ? 

As I wrote I don't think this setting has any effect on the offline reindex.

Instead when starting the reindex command pass -Xmx64g to the JVM

 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/2152ffaa-cbe2-44fc-9685-fecb34b1b990n%40googlegroups.com.

[Guy Levkowitz]

so i can run like this:

java -jar  -Xmx64g  /opt/gerrit/bin/gerrit.war reindex

ב-יום שני, 2 בספטמבר 2024 בשעה 15:12:43 UTC+3, Matthias Sohn כתב/ה:

[Matthias Sohn]

On Mon, Sep 2, 2024 at 2:36 PM Guy Levkowitz <sil...@gmail.com> wrote:

so i can run like this:

java -jar  -Xmx64g  /opt/gerrit/bin/gerrit.war reindex

yes, note that you can run java -h and java -X to get help from java about command line options ;-) 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/b90dec7f-a05f-4533-809d-6d9683ed92e3n%40googlegroups.com.

[Guy Levkowitz]

thanks

ב-יום שני, 2 בספטמבר 2024 בשעה 15:41:45 UTC+3, Matthias Sohn כתב/ה: