Gerrit upgrade 3.5.2 to 3.8.3 - Reindex process gets stuck forever

391 views
Skip to first unread message

Hakan AVCI

unread,
Apr 19, 2024, 9:54:48 AM4/19/24
to Repo and Gerrit Discussion
Hello Everyone,

I have been trying to upgrade our Gerrit instance from 3.5.2 to 3.8.3 on our staging instance with whole data replicated from Prod instance to mimic production upgrade. Everything is fine except only one issue which is reindex is not getting completed due to one of the repositories that I found after many reindex tries :) 

Workaround was for me to move this bare repo out of the git folder that is inside of the Gerrit home_folder (original location) to somewhere else, then offline reindex that followed by init process, was successful. I had to do this because otherwise Gerrit would never start due to incomplete index. But for sure, I have to make all changes of this repo are visible in UI anyway. However, online reindex also act like offline so it's getting stuck somewhere while it's trying to index changes for that problematic repo I have mentioned. Result was the same when I delete whole index folder and run reindex to let Gerrit creates indexes from scratch. I also ran git gc in aggressive mode (takes around 2 hours to complete) but this did not help either. This repo, unfortunately, contains lots of binaries and its current size is around 14G.

My question is how can I get more info about reindex process ( --verbose in offline reindex did not really help to figure this out ), what is it doing and how can I see which exact change is causing this issue?


Thanks in advance!
Hakan.

Daniele Sassoli

unread,
Apr 19, 2024, 10:09:16 AM4/19/24
to Repo and Gerrit Discussion
Running git fsck on the repo might be a good first step.
Also keep in mind that when updating from 3.5 to 3.8 you will first need to upgrade to 3.6 to run the copy-approvals command. 


Thanks in advance!
Hakan.

Luca Milanesio

unread,
Apr 19, 2024, 10:27:05 AM4/19/24
to Repo and Gerrit Discussion, Luca Milanesio, Daniele Sassoli
Good point, how did you perform the upgrade?

If you check my presentation of Gerrit v3.8 at the last Gerrit User Summit (10:09) on the GerritForge’s TV channel, you’ll see that it says:
- Upgrade to v3.5
- Run copy-approval
- Upgrade to v3.8

Did you run the copy-approval tool?

Luca.




Thanks in advance!
Hakan.

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/2f7d8cb0-8994-4cde-8dbb-71ac42bf0499n%40googlegroups.com.

Matthias Sohn

unread,
Apr 19, 2024, 10:30:45 AM4/19/24
to Hakan AVCI, Repo and Gerrit Discussion
Did you provide enough heap and CPUs to the reindexing ?

Did you observe Java gc activity during reindexing ?
On a huge repo with limited heap and CPUs you could overload the JVM so that it's mostly busy running Java gc.
On large repos you may need to reduce the number of reindexing threads to reduce the load on the JVM.

You can exclude proejcts when reindexing all changes, see gerrit.config option index.excludeProjectFromChangeReindex

You can then separately index the excluded project using the gerrit index changes-in-project command,
see https://gerrit-review.googlesource.com/Documentation/cmd-index-changes-in-project.html.
In between you can also change the indexing configuration to e.g. use less threads on the huge repo.

-Matthias
 

Thanks in advance!
Hakan.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Hakan AVCI

unread,
Apr 19, 2024, 10:44:02 AM4/19/24
to Repo and Gerrit Discussion
On Friday 19 April 2024 at 16:27:05 UTC+2 Luca Milanesio wrote:

On 19 Apr 2024, at 15:09, Daniele Sassoli <daniele...@gmail.com> wrote:



On Friday 19 April 2024 at 14:54:48 UTC+1 Hakan AVCI wrote:
Hello Everyone,

I have been trying to upgrade our Gerrit instance from 3.5.2 to 3.8.3 on our staging instance with whole data replicated from Prod instance to mimic production upgrade. Everything is fine except only one issue which is reindex is not getting completed due to one of the repositories that I found after many reindex tries :) 

Workaround was for me to move this bare repo out of the git folder that is inside of the Gerrit home_folder (original location) to somewhere else, then offline reindex that followed by init process, was successful. I had to do this because otherwise Gerrit would never start due to incomplete index. But for sure, I have to make all changes of this repo are visible in UI anyway. However, online reindex also act like offline so it's getting stuck somewhere while it's trying to index changes for that problematic repo I have mentioned. Result was the same when I delete whole index folder and run reindex to let Gerrit creates indexes from scratch. I also ran git gc in aggressive mode (takes around 2 hours to complete) but this did not help either. This repo, unfortunately, contains lots of binaries and its current size is around 14G.

My question is how can I get more info about reindex process ( --verbose in offline reindex did not really help to figure this out ), what is it doing and how can I see which exact change is causing this issue?
 
Running git fsck on the repo might be a good first step.
Also keep in mind that when updating from 3.5 to 3.8 you will first need to upgrade to 3.6 to run the copy-approvals command. 

Good point, how did you perform the upgrade?

If you check my presentation of Gerrit v3.8 at the last Gerrit User Summit (10:09) on the GerritForge’s TV channel, you’ll see that it says:
- Upgrade to v3.5
- Run copy-approval
- Upgrade to v3.8

Did you run the copy-approval tool?

Yes I did, that was first thing I have done on prod instance, then I have copied everything to staging. I was following https://groups.google.com/g/repo-discuss/c/HlW4WhKcUp4 thread.

I actually saw a comment about copy-approval from you here in thread: https://groups.google.com/g/repo-discuss/c/S8shRzWnFrQ, which as I understand copy-approval should not impact any upgrade process. Or did I interpret this wrong?

Hakan AVCI

unread,
Apr 19, 2024, 10:52:12 AM4/19/24
to Repo and Gerrit Discussion
On Friday 19 April 2024 at 16:30:45 UTC+2 Matthias Sohn wrote:
On Fri, Apr 19, 2024 at 3:54 PM Hakan AVCI <hakana...@gmail.com> wrote:
Hello Everyone,

I have been trying to upgrade our Gerrit instance from 3.5.2 to 3.8.3 on our staging instance with whole data replicated from Prod instance to mimic production upgrade. Everything is fine except only one issue which is reindex is not getting completed due to one of the repositories that I found after many reindex tries :) 

Workaround was for me to move this bare repo out of the git folder that is inside of the Gerrit home_folder (original location) to somewhere else, then offline reindex that followed by init process, was successful. I had to do this because otherwise Gerrit would never start due to incomplete index. But for sure, I have to make all changes of this repo are visible in UI anyway. However, online reindex also act like offline so it's getting stuck somewhere while it's trying to index changes for that problematic repo I have mentioned. Result was the same when I delete whole index folder and run reindex to let Gerrit creates indexes from scratch. I also ran git gc in aggressive mode (takes around 2 hours to complete) but this did not help either. This repo, unfortunately, contains lots of binaries and its current size is around 14G.

My question is how can I get more info about reindex process ( --verbose in offline reindex did not really help to figure this out ), what is it doing and how can I see which exact change is causing this issue?

Did you provide enough heap and CPUs to the reindexing ?
 
I think so, 16 CPU and 24G memory and since it's a dedicated staging instance there was no other load at all. 


Did you observe Java gc activity during reindexing ?

If you meant taking a heap dump during the reindex and look inside, then my answer is no. 
 
On a huge repo with limited heap and CPUs you could overload the JVM so that it's mostly busy running Java gc.
On large repos you may need to reduce the number of reindexing threads to reduce the load on the JVM.

You can exclude proejcts when reindexing all changes, see gerrit.config option index.excludeProjectFromChangeReindex

But then this will lead that no change is visible in UI for that repository, right?
 

You can then separately index the excluded project using the gerrit index changes-in-project command,
 
Yes, I also tried that, same result. I even let it run for 4-5 days but there was no new change has indexed (I was checking UI to see if some new changes are visible).

Luca Milanesio

unread,
Apr 19, 2024, 10:56:05 AM4/19/24
to Repo and Gerrit Discussion, Luca Milanesio, Hakan AVCI

On 19 Apr 2024, at 15:44, Hakan AVCI <hakana...@gmail.com> wrote:



On Friday 19 April 2024 at 16:27:05 UTC+2 Luca Milanesio wrote:

On 19 Apr 2024, at 15:09, Daniele Sassoli <daniele...@gmail.com> wrote:



On Friday 19 April 2024 at 14:54:48 UTC+1 Hakan AVCI wrote:
Hello Everyone,

I have been trying to upgrade our Gerrit instance from 3.5.2 to 3.8.3 on our staging instance with whole data replicated from Prod instance to mimic production upgrade. Everything is fine except only one issue which is reindex is not getting completed due to one of the repositories that I found after many reindex tries :) 

Workaround was for me to move this bare repo out of the git folder that is inside of the Gerrit home_folder (original location) to somewhere else, then offline reindex that followed by init process, was successful. I had to do this because otherwise Gerrit would never start due to incomplete index. But for sure, I have to make all changes of this repo are visible in UI anyway. However, online reindex also act like offline so it's getting stuck somewhere while it's trying to index changes for that problematic repo I have mentioned. Result was the same when I delete whole index folder and run reindex to let Gerrit creates indexes from scratch. I also ran git gc in aggressive mode (takes around 2 hours to complete) but this did not help either. This repo, unfortunately, contains lots of binaries and its current size is around 14G.

My question is how can I get more info about reindex process ( --verbose in offline reindex did not really help to figure this out ), what is it doing and how can I see which exact change is causing this issue?
 
Running git fsck on the repo might be a good first step.
Also keep in mind that when updating from 3.5 to 3.8 you will first need to upgrade to 3.6 to run the copy-approvals command. 

Good point, how did you perform the upgrade?

If you check my presentation of Gerrit v3.8 at the last Gerrit User Summit (10:09) on the GerritForge’s TV channel, you’ll see that it says:
- Upgrade to v3.5
- Run copy-approval
- Upgrade to v3.8

Did you run the copy-approval tool?

Yes I did, that was first thing I have done on prod instance, then I have copied everything to staging. I was following https://groups.google.com/g/repo-discuss/c/HlW4WhKcUp4 thread.

I actually saw a comment about copy-approval from you here in thread: https://groups.google.com/g/repo-discuss/c/S8shRzWnFrQ, which as I understand copy-approval should not impact any upgrade process. Or did I interpret this wrong?

Well, I did not find that remark in the thread :-)
Please use the video and the release notes as a reference: individual sentences in a long discussion thread cannot be taken out of context. The copy-approval is mandatory and must be part of your upgrade process.

Luca.


Luca.




Thanks in advance!
Hakan.

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/2f7d8cb0-8994-4cde-8dbb-71ac42bf0499n%40googlegroups.com.


--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Hakan AVCI

unread,
Apr 19, 2024, 11:07:15 AM4/19/24
to Repo and Gerrit Discussion
On Friday 19 April 2024 at 16:56:05 UTC+2 Luca Milanesio wrote:

On 19 Apr 2024, at 15:44, Hakan AVCI <hakana...@gmail.com> wrote:



On Friday 19 April 2024 at 16:27:05 UTC+2 Luca Milanesio wrote:

On 19 Apr 2024, at 15:09, Daniele Sassoli <daniele...@gmail.com> wrote:



On Friday 19 April 2024 at 14:54:48 UTC+1 Hakan AVCI wrote:
Hello Everyone,

I have been trying to upgrade our Gerrit instance from 3.5.2 to 3.8.3 on our staging instance with whole data replicated from Prod instance to mimic production upgrade. Everything is fine except only one issue which is reindex is not getting completed due to one of the repositories that I found after many reindex tries :) 

Workaround was for me to move this bare repo out of the git folder that is inside of the Gerrit home_folder (original location) to somewhere else, then offline reindex that followed by init process, was successful. I had to do this because otherwise Gerrit would never start due to incomplete index. But for sure, I have to make all changes of this repo are visible in UI anyway. However, online reindex also act like offline so it's getting stuck somewhere while it's trying to index changes for that problematic repo I have mentioned. Result was the same when I delete whole index folder and run reindex to let Gerrit creates indexes from scratch. I also ran git gc in aggressive mode (takes around 2 hours to complete) but this did not help either. This repo, unfortunately, contains lots of binaries and its current size is around 14G.

My question is how can I get more info about reindex process ( --verbose in offline reindex did not really help to figure this out ), what is it doing and how can I see which exact change is causing this issue?
 
Running git fsck on the repo might be a good first step.
Also keep in mind that when updating from 3.5 to 3.8 you will first need to upgrade to 3.6 to run the copy-approvals command. 

Good point, how did you perform the upgrade?

If you check my presentation of Gerrit v3.8 at the last Gerrit User Summit (10:09) on the GerritForge’s TV channel, you’ll see that it says:
- Upgrade to v3.5
- Run copy-approval
- Upgrade to v3.8

Did you run the copy-approval tool?

Yes I did, that was first thing I have done on prod instance, then I have copied everything to staging. I was following https://groups.google.com/g/repo-discuss/c/HlW4WhKcUp4 thread.

I actually saw a comment about copy-approval from you here in thread: https://groups.google.com/g/repo-discuss/c/S8shRzWnFrQ, which as I understand copy-approval should not impact any upgrade process. Or did I interpret this wrong?

Well, I did not find that remark in the thread :-)

Yea it's not that important anyway :)
 
Please use the video and the release notes as a reference: individual sentences in a long discussion thread cannot be taken out of context. The copy-approval is mandatory and must be part of your upgrade process.

I have done copy-approval run already, also I'm trying read as much as I can including release notes and discussion and everything :) Thanks for all of those guidelines. I will also watch the video in case (I hope) I'm missing something.

Luca Milanesio

unread,
Apr 19, 2024, 11:12:53 AM4/19/24
to Repo and Gerrit Discussion, Luca Milanesio, Hakan AVCI

On 19 Apr 2024, at 15:52, Hakan AVCI <hakana...@gmail.com> wrote:



On Friday 19 April 2024 at 16:30:45 UTC+2 Matthias Sohn wrote:
On Fri, Apr 19, 2024 at 3:54 PM Hakan AVCI <hakana...@gmail.com> wrote:
Hello Everyone,

I have been trying to upgrade our Gerrit instance from 3.5.2 to 3.8.3 on our staging instance with whole data replicated from Prod instance to mimic production upgrade. Everything is fine except only one issue which is reindex is not getting completed due to one of the repositories that I found after many reindex tries :) 

Workaround was for me to move this bare repo out of the git folder that is inside of the Gerrit home_folder (original location) to somewhere else, then offline reindex that followed by init process, was successful. I had to do this because otherwise Gerrit would never start due to incomplete index. But for sure, I have to make all changes of this repo are visible in UI anyway. However, online reindex also act like offline so it's getting stuck somewhere while it's trying to index changes for that problematic repo I have mentioned. Result was the same when I delete whole index folder and run reindex to let Gerrit creates indexes from scratch. I also ran git gc in aggressive mode (takes around 2 hours to complete) but this did not help either. This repo, unfortunately, contains lots of binaries and its current size is around 14G.

My question is how can I get more info about reindex process ( --verbose in offline reindex did not really help to figure this out ), what is it doing and how can I see which exact change is causing this issue?

Did you provide enough heap and CPUs to the reindexing ?
 
I think so, 16 CPU and 24G memory and since it's a dedicated staging instance there was no other load at all. 

Well, the reindex is really putting extra pressure that you typically don’t have. Depending on the size of the repo and the complexity of the submit rules, the reindex may take lots of memory and CPU.



Did you observe Java gc activity during reindexing ?

If you meant taking a heap dump during the reindex and look inside, then my answer is no. 

No, you should run it with the verbose GC logging enabled:

 
On a huge repo with limited heap and CPUs you could overload the JVM so that it's mostly busy running Java gc.
On large repos you may need to reduce the number of reindexing threads to reduce the load on the JVM.

You can exclude proejcts when reindexing all changes, see gerrit.config option index.excludeProjectFromChangeReindex

But then this will lead that no change is visible in UI for that repository, right?

Sure, but you can reindex that later on, as Matthias suggested, with the command  'gerrit index changes-in-project’

 

You can then separately index the excluded project using the gerrit index changes-in-project command,
 
Yes, I also tried that, same result. I even let it run for 4-5 days but there was no new change has indexed (I was checking UI to see if some new changes are visible).

You should get a thread dump and see if there is any activity stuck, also check the error_log for errors.

HTH

Luca.


see https://gerrit-review.googlesource.com/Documentation/cmd-index-changes-in-project.html.
In between you can also change the indexing configuration to e.g. use less threads on the huge repo.

-Matthias
 

Thanks in advance!
Hakan.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Hakan AVCI

unread,
Apr 22, 2024, 10:36:26 AM4/22/24
to Repo and Gerrit Discussion
On Friday 19 April 2024 at 17:12:53 UTC+2 Luca Milanesio wrote:

On 19 Apr 2024, at 15:52, Hakan AVCI <hakana...@gmail.com> wrote:



On Friday 19 April 2024 at 16:30:45 UTC+2 Matthias Sohn wrote:
On Fri, Apr 19, 2024 at 3:54 PM Hakan AVCI <hakana...@gmail.com> wrote:
Hello Everyone,

I have been trying to upgrade our Gerrit instance from 3.5.2 to 3.8.3 on our staging instance with whole data replicated from Prod instance to mimic production upgrade. Everything is fine except only one issue which is reindex is not getting completed due to one of the repositories that I found after many reindex tries :) 

Workaround was for me to move this bare repo out of the git folder that is inside of the Gerrit home_folder (original location) to somewhere else, then offline reindex that followed by init process, was successful. I had to do this because otherwise Gerrit would never start due to incomplete index. But for sure, I have to make all changes of this repo are visible in UI anyway. However, online reindex also act like offline so it's getting stuck somewhere while it's trying to index changes for that problematic repo I have mentioned. Result was the same when I delete whole index folder and run reindex to let Gerrit creates indexes from scratch. I also ran git gc in aggressive mode (takes around 2 hours to complete) but this did not help either. This repo, unfortunately, contains lots of binaries and its current size is around 14G.

My question is how can I get more info about reindex process ( --verbose in offline reindex did not really help to figure this out ), what is it doing and how can I see which exact change is causing this issue?

Did you provide enough heap and CPUs to the reindexing ?
 
I think so, 16 CPU and 24G memory and since it's a dedicated staging instance there was no other load at all. 

Well, the reindex is really putting extra pressure that you typically don’t have. Depending on the size of the repo and the complexity of the submit rules, the reindex may take lots of memory and CPU.



Did you observe Java gc activity during reindexing ?

If you meant taking a heap dump during the reindex and look inside, then my answer is no. 

No, you should run it with the verbose GC logging enabled:

 
On a huge repo with limited heap and CPUs you could overload the JVM so that it's mostly busy running Java gc.
On large repos you may need to reduce the number of reindexing threads to reduce the load on the JVM.

You can exclude proejcts when reindexing all changes, see gerrit.config option index.excludeProjectFromChangeReindex

But then this will lead that no change is visible in UI for that repository, right?

Sure, but you can reindex that later on, as Matthias suggested, with the command  'gerrit index changes-in-project’

 

You can then separately index the excluded project using the gerrit index changes-in-project command,
 
Yes, I also tried that, same result. I even let it run for 4-5 days but there was no new change has indexed (I was checking UI to see if some new changes are visible).

You should get a thread dump and see if there is any activity stuck, also check the error_log for errors.

I re-triggered changes-in-project for this specific project in the morning. Just after started, 14 changes showed up in the UI but since hours nothing "new" change is visible except those 14 changes. And when I check the java thread from time to time, it either spending time on:

- org.eclipse.jgit.diff.MyersDiff$MiddleEdit$EditPaths.calculate
or
- org.eclipse.jgit.diff.MyersDiff$MiddleEdit$BackwardEditPaths.snake

 Index-Batch-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@a1fcaa8[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@e7a1d8[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@e205fb4]]]running
at org.eclipse.jgit.diff.MyersDiff$MiddleEdit$BackwardEditPaths.snake(MyersDiff.java:480)
at org.eclipse.jgit.diff.MyersDiff$MiddleEdit$EditPaths.calculate(MyersDiff.java:398) 

 Index-Batch-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@a1fcaa8[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@e7a1d8[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@e205fb4]]]running
at org.eclipse.jgit.diff.MyersDiff$MiddleEdit$EditPaths.calculate(MyersDiff.java:377)
at org.eclipse.jgit.diff.MyersDiff$MiddleEdit.calculate(MyersDiff.java:231)

Nothing related in error_log. Also git fsck shows no error. I'm not that good at analyzing java threads, however, I assume this is still related to repository itself since it is spending the whole time on jgit diff stuff. I will also try to analyze heap dump that is taken while the index process is running. Sounds like, I need to focus on somehow curing the repository.

Matthias Sohn

unread,
Apr 22, 2024, 2:42:35 PM4/22/24
to Hakan AVCI, Repo and Gerrit Discussion
Can you provide the complete stacktrace of these threads ?

Ensure that the diff caches in gerrit are persistent and large enough.
If these caches weren't persistent or too small you may spend a lot
of time on computing diffs from scratch.
 

Hakan AVCI

unread,
Apr 23, 2024, 4:00:42 AM4/23/24
to Repo and Gerrit Discussion
Sure, and sorry for putting it as normal text, I couldn't find an option post it as a code block.
"Index-Batch-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@a1fcaa8[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@e7a1d8[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@e205fb4]]]running"
at org.eclipse.jgit.diff.MyersDiff$MiddleEdit$BackwardEditPaths.snake(MyersDiff.java:480)
at org.eclipse.jgit.diff.MyersDiff$MiddleEdit$EditPaths.calculate(MyersDiff.java:398)
at org.eclipse.jgit.diff.MyersDiff$MiddleEdit.calculate(MyersDiff.java:231)
at org.eclipse.jgit.diff.MyersDiff.calculateEdits(MyersDiff.java:158)
at org.eclipse.jgit.diff.MyersDiff.calculateEdits(MyersDiff.java:135)
at org.eclipse.jgit.diff.MyersDiff.<init>(MyersDiff.java:118)
at org.eclipse.jgit.diff.MyersDiff$1.diffNonCommon(MyersDiff.java:89)
at org.eclipse.jgit.diff.HistogramDiff$State.diffReplace(HistogramDiff.java:150)
at org.eclipse.jgit.diff.HistogramDiff$State.diffRegion(HistogramDiff.java:127)
at org.eclipse.jgit.diff.HistogramDiff.diffNonCommon(HistogramDiff.java:106)
at org.eclipse.jgit.diff.LowLevelDiffAlgorithm.diffNonCommon(LowLevelDiffAlgorithm.java:29)
at org.eclipse.jgit.diff.DiffAlgorithm.diff(DiffAlgorithm.java:93)
at org.eclipse.jgit.merge.MergeAlgorithm.merge(MergeAlgorithm.java:162)
at org.eclipse.jgit.merge.ResolveMerger.contentMerge(ResolveMerger.java:1458)
at org.eclipse.jgit.merge.ResolveMerger.processEntry(ResolveMerger.java:1277)
at org.eclipse.jgit.merge.ResolveMerger.mergeTreeWalk(ResolveMerger.java:1896)
at org.eclipse.jgit.merge.ResolveMerger.mergeTrees(ResolveMerger.java:1846)
at org.eclipse.jgit.merge.ResolveMerger.mergeImpl(ResolveMerger.java:910)
at org.eclipse.jgit.merge.Merger.merge(Merger.java:233)
at org.eclipse.jgit.merge.Merger.merge(Merger.java:186)
at org.eclipse.jgit.merge.ThreeWayMerger.merge(ThreeWayMerger.java:96)
at com.google.gerrit.server.git.PureRevertCache$Loader.load(PureRevertCache.java:186)
at com.google.gerrit.server.git.PureRevertCache$Loader.load(PureRevertCache.java:139)
at com.google.gerrit.server.cache.h2.H2CacheImpl$Loader.load(H2CacheImpl.java:283)
at com.google.gerrit.server.cache.h2.H2CacheImpl$Loader.load(H2CacheImpl.java:259)
at com.github.benmanes.caffeine.guava.CaffeinatedGuavaLoadingCache$SingleLoader.load(CaffeinatedGuavaLoadingCache.java:136)
at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:141)
at com.github.benmanes.caffeine.cache.LocalLoadingCache$$Lambda$371/0x00000008404c5040.apply(Unknown Source)
at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
at com.github.benmanes.caffeine.cache.LocalCache$$Lambda$980/0x0000000840c75440.apply(Unknown Source)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2413)
at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$981/0x0000000840c74c40.apply(Unknown Source)
at java.util.concurrent.ConcurrentHashMap.compute(java.base@11.0.20.1/ConcurrentHashMap.java:1908)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2411)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2394)
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:54)
at com.github.benmanes.caffeine.guava.CaffeinatedGuavaLoadingCache.get(CaffeinatedGuavaLoadingCache.java:59)
at com.google.gerrit.server.cache.h2.H2CacheImpl.get(H2CacheImpl.java:135)
at com.google.gerrit.server.git.PureRevertCache.isPureRevert(PureRevertCache.java:120)
at com.google.gerrit.server.git.PureRevertCache.isPureRevert(PureRevertCache.java:101)
at com.google.gerrit.server.change.PureRevert.get(PureRevert.java:46)
at com.google.gerrit.server.query.change.ChangeData.isPureRevert(ChangeData.java:1401)
at com.google.gerrit.server.index.change.ChangeField.lambda$static$34(ChangeField.java:575)
at com.google.gerrit.server.index.change.ChangeField$$Lambda$256/0x00000008401ef040.get(Unknown Source)
at com.google.gerrit.index.IndexedField.get(IndexedField.java:434)
at com.google.gerrit.index.IndexedField$SearchSpec.get(IndexedField.java:150)
at com.google.gerrit.index.Schema.fieldValues(Schema.java:263)
at com.google.gerrit.index.Schema.lambda$buildFields$0(Schema.java:295)
at com.google.gerrit.index.Schema$$Lambda$1579/0x0000000840ef5840.apply(Unknown Source)
at java.util.stream.ReferencePipeline$3$1.accept(java.base@11.0.20.1/ReferencePipeline.java:195)
at com.google.common.collect.CollectSpliterators$1WithCharacteristics.lambda$forEachRemaining$1(CollectSpliterators.java:67)
at com.google.common.collect.CollectSpliterators$1WithCharacteristics$$Lambda$904/0x0000000840bf1040.accept(Unknown Source)
at java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(java.base@11.0.20.1/Streams.java:104)
at com.google.common.collect.CollectSpliterators$1WithCharacteristics.forEachRemaining(CollectSpliterators.java:67)
at java.util.stream.AbstractPipeline.copyInto(java.base@11.0.20.1/AbstractPipeline.java:484)
at com.google.gerrit.index.Schema.buildFields(Schema.java:297)
at com.google.gerrit.lucene.AbstractLuceneIndex.toDocument(AbstractLuceneIndex.java:334)
at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:217)
at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:103)
at com.google.gerrit.server.index.change.ChangeIndexer.indexImpl(ChangeIndexer.java:234)
at com.google.gerrit.server.index.change.ChangeIndexer.doIndex(ChangeIndexer.java:197)
at com.google.gerrit.server.index.change.ChangeIndexer.index(ChangeIndexer.java:193)
at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.index(AllChangesIndexer.java:250)
at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.lambda$call$1(AllChangesIndexer.java:239)
at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer$$Lambda$1570/0x0000000840eee840.accept(Unknown Source)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(java.base@11.0.20.1/ForEachOps.java:183)
at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.call(AllChangesIndexer.java:239)
at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.call(AllChangesIndexer.java:209)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.20.1/Executors.java:515)
at java.util.concurrent.FutureTask.run(java.base@11.0.20.1/FutureTask.java:264)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.20.1/ScheduledThreadPoolExecutor.java:304)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:675)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.20.1/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.20.1/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.20.1/Thread.java:829)
"Index-Batch-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@a1fcaa8[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@e7a1d8[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@e205fb4]]]running"
at org.eclipse.jgit.diff.MyersDiff$MiddleEdit$EditPaths.calculate(MyersDiff.java:377)
at org.eclipse.jgit.diff.MyersDiff$MiddleEdit.calculate(MyersDiff.java:231)
at org.eclipse.jgit.diff.MyersDiff.calculateEdits(MyersDiff.java:158)
at org.eclipse.jgit.diff.MyersDiff.calculateEdits(MyersDiff.java:135)
at org.eclipse.jgit.diff.MyersDiff.<init>(MyersDiff.java:118)
at org.eclipse.jgit.diff.MyersDiff$1.diffNonCommon(MyersDiff.java:89)
at org.eclipse.jgit.diff.HistogramDiff$State.diffReplace(HistogramDiff.java:150)
at org.eclipse.jgit.diff.HistogramDiff$State.diffRegion(HistogramDiff.java:127)
at org.eclipse.jgit.diff.HistogramDiff.diffNonCommon(HistogramDiff.java:106)
at org.eclipse.jgit.diff.LowLevelDiffAlgorithm.diffNonCommon(LowLevelDiffAlgorithm.java:29)
at org.eclipse.jgit.diff.DiffAlgorithm.diff(DiffAlgorithm.java:93)
at org.eclipse.jgit.merge.MergeAlgorithm.merge(MergeAlgorithm.java:162)
at org.eclipse.jgit.merge.ResolveMerger.contentMerge(ResolveMerger.java:1458)
at org.eclipse.jgit.merge.ResolveMerger.processEntry(ResolveMerger.java:1277)
at org.eclipse.jgit.merge.ResolveMerger.mergeTreeWalk(ResolveMerger.java:1896)
at org.eclipse.jgit.merge.ResolveMerger.mergeTrees(ResolveMerger.java:1846)
at org.eclipse.jgit.merge.ResolveMerger.mergeImpl(ResolveMerger.java:910)
at org.eclipse.jgit.merge.Merger.merge(Merger.java:233)
at org.eclipse.jgit.merge.Merger.merge(Merger.java:186)
at org.eclipse.jgit.merge.ThreeWayMerger.merge(ThreeWayMerger.java:96)
at com.google.gerrit.server.git.PureRevertCache$Loader.load(PureRevertCache.java:186)
at com.google.gerrit.server.git.PureRevertCache$Loader.load(PureRevertCache.java:139)
at com.google.gerrit.server.cache.h2.H2CacheImpl$Loader.load(H2CacheImpl.java:283)
at com.google.gerrit.server.cache.h2.H2CacheImpl$Loader.load(H2CacheImpl.java:259)
at com.github.benmanes.caffeine.guava.CaffeinatedGuavaLoadingCache$SingleLoader.load(CaffeinatedGuavaLoadingCache.java:136)
at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:141)
at com.github.benmanes.caffeine.cache.LocalLoadingCache$$Lambda$371/0x00000008404c5040.apply(Unknown Source)
at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:139)
at com.github.benmanes.caffeine.cache.LocalCache$$Lambda$980/0x0000000840c75440.apply(Unknown Source)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2413)
at com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda$981/0x0000000840c74c40.apply(Unknown Source)
at java.util.concurrent.ConcurrentHashMap.compute(java.base@11.0.20.1/ConcurrentHashMap.java:1908)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2411)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2394)
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:54)
at com.github.benmanes.caffeine.guava.CaffeinatedGuavaLoadingCache.get(CaffeinatedGuavaLoadingCache.java:59)
at com.google.gerrit.server.cache.h2.H2CacheImpl.get(H2CacheImpl.java:135)
at com.google.gerrit.server.git.PureRevertCache.isPureRevert(PureRevertCache.java:120)
at com.google.gerrit.server.git.PureRevertCache.isPureRevert(PureRevertCache.java:101)
at com.google.gerrit.server.change.PureRevert.get(PureRevert.java:46)
at com.google.gerrit.server.query.change.ChangeData.isPureRevert(ChangeData.java:1401)
at com.google.gerrit.server.index.change.ChangeField.lambda$static$34(ChangeField.java:575)
at com.google.gerrit.server.index.change.ChangeField$$Lambda$256/0x00000008401ef040.get(Unknown Source)
at com.google.gerrit.index.IndexedField.get(IndexedField.java:434)
at com.google.gerrit.index.IndexedField$SearchSpec.get(IndexedField.java:150)
at com.google.gerrit.index.Schema.fieldValues(Schema.java:263)
at com.google.gerrit.index.Schema.lambda$buildFields$0(Schema.java:295)
at com.google.gerrit.index.Schema$$Lambda$1579/0x0000000840ef5840.apply(Unknown Source)
at java.util.stream.ReferencePipeline$3$1.accept(java.base@11.0.20.1/ReferencePipeline.java:195)
at com.google.common.collect.CollectSpliterators$1WithCharacteristics.lambda$forEachRemaining$1(CollectSpliterators.java:67)
at com.google.common.collect.CollectSpliterators$1WithCharacteristics$$Lambda$904/0x0000000840bf1040.accept(Unknown Source)
at java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(java.base@11.0.20.1/Streams.java:104)
at com.google.common.collect.CollectSpliterators$1WithCharacteristics.forEachRemaining(CollectSpliterators.java:67)
at java.util.stream.AbstractPipeline.copyInto(java.base@11.0.20.1/AbstractPipeline.java:484)
at com.google.gerrit.index.Schema.buildFields(Schema.java:297)
at com.google.gerrit.lucene.AbstractLuceneIndex.toDocument(AbstractLuceneIndex.java:334)
at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:217)
at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:103)
at com.google.gerrit.server.index.change.ChangeIndexer.indexImpl(ChangeIndexer.java:234)
at com.google.gerrit.server.index.change.ChangeIndexer.doIndex(ChangeIndexer.java:197)
at com.google.gerrit.server.index.change.ChangeIndexer.index(ChangeIndexer.java:193)
at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.index(AllChangesIndexer.java:250)
at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.lambda$call$1(AllChangesIndexer.java:239)
at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer$$Lambda$1570/0x0000000840eee840.accept(Unknown Source)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(java.base@11.0.20.1/ForEachOps.java:183)
at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.call(AllChangesIndexer.java:239)
at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectSliceIndexer.call(AllChangesIndexer.java:209)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.20.1/Executors.java:515)
at java.util.concurrent.FutureTask.run(java.base@11.0.20.1/FutureTask.java:264)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.20.1/ScheduledThreadPoolExecutor.java:304)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:675)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.20.1/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.20.1/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.20.1/Thread.java:829)


Ensure that the diff caches in gerrit are persistent and large enough.
 
It's VM/docker based installation and cache directory is persisted. Also, I'm using default values in gerrit.config.

Hakan AVCI

unread,
Apr 30, 2024, 3:57:16 AM4/30/24
to Repo and Gerrit Discussion
I would like to update/summarize this thread for the people who might get benefit from it. 

In my case, 7 hours~ after running "gerrit gc --show-progress --aggressive repo_name", it did the trick and reindex went well. Running "git gc" on the server for this bare repo did not help like I mentioned before, so I guess jgit might be doing some magic. But please make sure about --aggressive mode before use it, like reading what is it doing and even maybe taking a backup of the repo. 

Another note from my operation, there were some issues for some repos (unfortunately I dont have exact error messages anymore) to complete copy-approval process in version 3.5.2. So, first I upgraded the instance to 3.5.6 (issues that I faced in 3.5.2 was fixed in 3.5.6) then ran copy-approval %100 successfully and then I switched to 3.8.5 (we have decided to go for latest minor version of 3.8). Maybe copy-approval issues in 3.5.2 has been mentioned in some other threads but I guess worth to mention here as well.

Thanks for your all input and your precious time, also for making Gerrit is better!

Luca Milanesio

unread,
Apr 30, 2024, 10:28:38 AM4/30/24
to Repo and Gerrit Discussion, Luca Milanesio, Hakan AVCI
Thanks for sharing your feedback on the closure of this issue.

On 30 Apr 2024, at 08:57, Hakan AVCI <hakana...@gmail.com> wrote:

I would like to update/summarize this thread for the people who might get benefit from it. 

In my case, 7 hours~ after running "gerrit gc --show-progress --aggressive repo_name", it did the trick and reindex went well. Running "git gc" on the server for this bare repo did not help like I mentioned before, so I guess jgit might be doing some magic.

Well, it’s not really *magic* but a different way of repacking I guess.

CGit just doesn’t have any knowledge of how Gerrit works and treats all refs in the same way, creating a single packfile and one bitmap associated, containing everything. Thats’ not really useful for Gerrit though, where you clearly need to separate the merged commits from the open changes.

Gerrit does GC differently and JGit knows about it and behaves the best interest of Gerrit :-)

But please make sure about --aggressive mode before use it, like reading what is it doing and even maybe taking a backup of the repo. 

The “--aggressive” helps if you believe your delta-chains are outdated, otherwise isn’t very useful and it would just make the GC process a lot slower.

Another note from my operation, there were some issues for some repos (unfortunately I dont have exact error messages anymore) to complete copy-approval process in version 3.5.2. So, first I upgraded the instance to 3.5.6 (issues that I faced in 3.5.2 was fixed in 3.5.6) then ran copy-approval %100 successfully and then I switched to 3.8.5 (we have decided to go for latest minor version of 3.8). Maybe copy-approval issues in 3.5.2 has been mentioned in some other threads but I guess worth to mention here as well.

You’re possibly referring to the situation where a Label configuration was changed and then all of a sudden you need to run a copy-approval on a change with a label out of range.

@Nasser merged a workaround on stable-3.5 (see [1]) and you can run a clean build from the latest stable-3.5 if you would like to use it.
Another workaround is just to tweak the repository configuration and just allow that label.

HTH

Luca.


Reply all
Reply to author
Forward
0 new messages