Gerrit 2.10 issues

[Hugo Arès]

Since we upgraded to Gerrit 2.10, we have many critical issues: projects disappearing from the project list, failed push operations, online reindexer aborted.

After some investigation, all these issues seems to have the same root cause, a behaviour change(regression?) introduced in JGit.

Here is an example of a stacktrace found in the logs:

ERROR: Exception caught while accessing pack file /.../somproject.git/objects/pack/pack-f62368113a4b555ed10d09fd44ba8a5587f87e80.pack, the pack file might be corrupt
java.io.IOException: Stale file handle
        at java.io.RandomAccessFile.readBytes0(Native Method)
        at java.io.RandomAccessFile.readBytes(RandomAccessFile.java:350)
        at java.io.RandomAccessFile.read(RandomAccessFile.java:385)
        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444)
        at org.eclipse.jgit.internal.storage.file.PackFile.read(PackFile.java:645)
        at org.eclipse.jgit.internal.storage.file.WindowCache.load(WindowCache.java:289)
        at org.eclipse.jgit.internal.storage.file.WindowCache.getOrLoad(WindowCache.java:368)
        at org.eclipse.jgit.internal.storage.file.WindowCache.get(WindowCache.java:179)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.pin(WindowCursor.java:354)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.copy(WindowCursor.java:226)
        at org.eclipse.jgit.internal.storage.file.PackFile.readFully(PackFile.java:556)
        at org.eclipse.jgit.internal.storage.file.PackFile.load(PackFile.java:714)
        at org.eclipse.jgit.internal.storage.file.PackFile.get(PackFile.java:257)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:414)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:383)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:375)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:145)
        at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229)
        at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:840)
        at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:753)
        at com.google.gerrit.server.git.VersionedMetaData.load(VersionedMetaData.java:117)
        at com.google.gerrit.server.git.VersionedMetaData.load(VersionedMetaData.java:94)
        at com.google.gerrit.server.project.ProjectCacheImpl$Loader.load(ProjectCacheImpl.java:294)
        at com.google.gerrit.server.project.ProjectCacheImpl$Loader.load(ProjectCacheImpl.java:278)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
        at com.google.gerrit.server.project.ProjectCacheImpl.checkedGet(ProjectCacheImpl.java:142)
        at com.google.gerrit.server.project.ProjectCacheImpl.get(ProjectCacheImpl.java:129)
        at com.google.gerrit.server.project.ListProjects.display(ListProjects.java:266)
        at com.google.gerrit.sshd.commands.ListProjectsCommand$1.run(ListProjectsCommand.java:49)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:442)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:368)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

We gc our repositories using native "git gc" so when Gerrit(JGit) tries to access pack files, it's possible that the file was deleted by the git gc and we get this stacktrace. We worked around the issue by stopping gc. This setup used to work on Gerrit 2.9 so looking at the changes that were made between JGit 3.4.2 and 3.6.2, I found this change:

27ee3342136a588adbc1eee4b333179d8f6f1aa7 Don't remove pack from pack list for problems which could be transient

If I understand, before this change, any IOException(like this stale file handle) used to consider the pack invalid, remove it from the list but now, it keeps it in the list (and log this stacktrace) assuming that the error is transient.

Anyone have an idea how I should fix this issue? Which issue is that commit fixing? As a temporary solution, I was thinking of reverting this commit but I need a proper solution?

I also have a minor issue with this commit:
9b86ebb4f605619d4af14d6260a338c38496b492 Log reason for ignoring pack when IOException occurred

This will print an error message for any pack file that is corrupted (used to handle them silently) in the std.err (without any way to silence them), which lead us to think that there was a lot of new error in gerrit 2.10 but in fact, JGit is just more verbose.

[Luca Milanesio]

Hi Hugo,

have you traced them to the Gerrit Issue Tracker as critical issues?

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Saša Živkov]

Anyone have an idea how I should fix this issue with this logic?

Except seeing this exception again and again in the error_log, what is the real issue?

 

Which issue is that commit fixing?

If I remember well this change was done for a different problem which also manifested as projects

getting disappeared. If a pack file was temporarily unreadable, because of a transient problem, but 

otherwise not corrupt then, before this change, it was permanently removed from the pack list for this

repository. If the removed pack contained the commit which was the tip of the refs/meta/config branch

then the permissions of that project couldn't be read any more, the read permission couldn't be checked

and the project effectively disappeared.

 

As a temporary solution, I was thinking of reverting this commit but I need a proper solution?

I also have a minor issue with this commit:
9b86ebb4f605619d4af14d6260a338c38496b492 Log reason for ignoring pack when IOException occurred

This will print an error message for any pack file that is corrupted (used to handle them silently) in the std.err (without any way to silence them), which lead us to think that there was a lot of new error in gerrit 2.10 but in fact, JGit is just more verbose.

--

[Matthias Sohn]

On Tue, Mar 24, 2015 at 5:43 PM, Hugo Arès <hugo...@ericsson.com> wrote:

As explained in the commit message of this change https://git.eclipse.org/r/#/c/39685

this change was done due to another problem reported on the Gerrit list.

JGit removed some pack due to a transient IOException which led to MissingObjectExceptions. 

The only way to recover from the erroneously removed pack was restarting the server which

is pretty bad.

Looks like there are other non-transient IOExceptions which JGit should detect and remove

the corresponding pack from the pack list. At the moment this is done for CorruptObjectException

and PackInvalidException in ObjectDirectory.handlePackError()

http://git.eclipse.org/c/jgit/jgit.git/tree/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/ObjectDirectory.java?id=v3.6.2.201501210735-r#n553

You could try to add IOException with error message "Stale file handle" to the first branch of this if statement.

Though relying on the message text looks brittle, this may depend on the chosen Java implementation, version

and maybe even platform since the exception originates in native code.

 

I also have a minor issue with this commit:
9b86ebb4f605619d4af14d6260a338c38496b492 Log reason for ignoring pack when IOException occurred

This will print an error message for any pack file that is corrupted (used to handle them silently) in the std.err (without any way to silence them), which lead us to think that there was a lot of new error in gerrit 2.10 but in fact, JGit is just more verbose.

this was done in order to get a chance to analyze such problems which turned out to be impossible when these exceptions

are silently ignored. In 4.0.0 we use proper logging using slf4j instead of printing to System.err. We couldn't introduce this

on short notice since new 3rd party dependencies need an IP review at Eclipse which takes a while. 

-Matthias

[Matthias Sohn]

are you using NFS or a local filesystem to store the git repositories ?

Please file a bug for JGit in Bugzilla so that other applications using JGit can also track this problem

https://bugs.eclipse.org/bugs/enter_bug.cgi?product=JGit

-Matthias 

[Hugo Arès]

We are using NFS. Git GC is running on another machine than Gerrit so this is why we get this "Stale file handle" when gc removed the pack and Gerrit tries to access it.

Here is the JGit bug[1].

I saw that change [2] added a special handling for FileNotFoundException that remove the pack from the list so I think that an IOException with message "Stale file handle" should also be treated as a file not found. I know that is brittle but I do not see any other way.

[1]https://bugs.eclipse.org/bugs/show_bug.cgi?id=462868
[2]https://git.eclipse.org/r/#/c/40503/

[Jan Kundrát]

On Wednesday, 25 March 2015 14:18:33 CET, Hugo Arès wrote:
> I saw that change [2] added a special handling for FileNotFoundException
> that remove the pack from the list so I think that an IOException with
> message "Stale file handle" should also be treated as a file not found. I
> know that is brittle but I do not see any other way.

To offer some alternatives, this could help a bit (but keep in mind that I
have no idea how reasonable it is to use JNI in this context):

http://stackoverflow.com/questions/6767382/how-to-get-a-meaningful-message-for-failed-calls-to-java-file-objects-mkdir-re

To me, it feels a bit nicer than parsing human-readable and possibly
localized strings.

With kind regards,
Jan

--
Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

[Matthias Sohn]

On Wed, Mar 25, 2015 at 2:18 PM, Hugo Arès <hugo...@ericsson.com> wrote:

We are using NFS. Git GC is running on another machine than Gerrit so this is why we get this "Stale file handle" when gc removed the pack and Gerrit tries to access it.

Here is the JGit bug[1].

I saw that change [2] added a special handling for FileNotFoundException that remove the pack from the list so I think that an IOException with message "Stale file handle" should also be treated as a file not found. I know that is brittle but I do not see any other way.

but how can removing a pack from the pack list fix the problem that you see projects disappearing ?

I guess this is caused by another effect we also observed when running multiple jgit processes on

the same git repos on NFS. 

We introduced a new configuration option core.trustfolderstat in3.6 in

0fc8b05a71054311c6290f6408386f7c48409724 (https://git.eclipse.org/r/#/c/36058/)

to workaround this problem. If this option is set to false JGit will not rely on the modification

timestamp of the folder containing the pack files to detect if the list of packfiles has changed

but instead always list the contained files. In our case this helped to fix the problem that

one jgit process didn't see new packfiles introduced by another jgit process.

-Matthias

[Hugo Arès]

Lets me step back and start over so you can understand how I ended up looking into that commit.

We upgraded from Gerrit 2.9.4 to 2.10 (which upgraded JGit from 3.4.2 to 3.6.2) and the users reported the 2 following issues:
-I cannot see my project: we fixed this one by restarting Gerrit (we had to, flush-cahes did not help),see stacktrace [1]
-I cannot push: we fixed some by running git repack and others were fixed when we had to restart for the previous issue, see stactraces [2] and [3].

During the upgrade, we also had an issue, the online reindexed failed(see stack trace [4] and [5]), we disabled our garbage collection and restarted Gerrit, the online reindexer completed successfully this time.

The only thing we changed during the upgrade is Gerrit, we have the same setup; storage, GC cron, Git version use by GC cron. Looking at the logs, every time there is an issue, there is a stacktrace about a missing commit following a stacktrace about an IOException accessing the pack file which lead me (maybe wrongfully) to think that https://git.eclipse.org/r/#/c/39685 could have caused the problems since it changed the way it handles IOException.

Those issues were only reported on our busiest repositories, which happen to be GC more frequently than others (every 2 hours). The current situation is that we reduced the frequecy we run gc(to once a day in during the non-working hours) and we do not see the issues anymore.

In the repositories where we had issue, we configured the core.trustfolderstat option and it look like we no longer have the IOException "Stale file hande" but we have the FileNotFoundException which should be fixed by https://git.eclipse.org/r/#/c/40503/. Should this change be in JGit 3.6, its only in 3.7 for now.

Hugo

[1]
[2015-03-23 07:38:07,749] WARN com.google.gerrit.server.project.ProjectCacheImpl : Cannot read project <project_name>
java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 67b30f728830954ff823df1ea5fe69e81a04d7cd
        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
        at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)
        at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2346)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318)

        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
        at com.google.gerrit.server.project.ProjectCacheImpl.checkedGet(ProjectCacheImpl.java:142)
        at com.google.gerrit.server.project.ProjectCacheImpl.get(ProjectCacheImpl.java:129)
        at com.google.gerrit.server.project.ListProjects.display(ListProjects.java:266)
        at com.google.gerrit.sshd.commands.ListProjectsCommand$1.run(ListProjectsCommand.java:49)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:442)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:368)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 67b30f728830954ff823df1ea5fe69e81a04d7cd
        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:148)

        at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229)
        at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:840)
        at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:753)
        at com.google.gerrit.server.git.VersionedMetaData.load(VersionedMetaData.java:117)
        at com.google.gerrit.server.git.VersionedMetaData.load(VersionedMetaData.java:94)
        at com.google.gerrit.server.project.ProjectCacheImpl$Loader.load(ProjectCacheImpl.java:294)
        at com.google.gerrit.server.project.ProjectCacheImpl$Loader.load(ProjectCacheImpl.java:278)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)

        ... 18 more

[2]
ERROR: Exception caught while accessing pack file <path>/<project_name>.git/objects/pack/pack-863def7a513379ce9d1ef8ce024df62743f69100.pack, the pack file might be corrupt

java.io.IOException: Stale file handle
        at java.io.RandomAccessFile.readBytes0(Native Method)
        at java.io.RandomAccessFile.readBytes(RandomAccessFile.java:350)
        at java.io.RandomAccessFile.read(RandomAccessFile.java:385)
        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444)
        at org.eclipse.jgit.internal.storage.file.PackFile.read(PackFile.java:645)
        at org.eclipse.jgit.internal.storage.file.WindowCache.load(WindowCache.java:289)
        at org.eclipse.jgit.internal.storage.file.WindowCache.getOrLoad(WindowCache.java:368)
        at org.eclipse.jgit.internal.storage.file.WindowCache.get(WindowCache.java:179)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.pin(WindowCursor.java:354)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.copy(WindowCursor.java:226)
        at org.eclipse.jgit.internal.storage.file.PackFile.readFully(PackFile.java:556)
        at org.eclipse.jgit.internal.storage.file.PackFile.load(PackFile.java:714)
        at org.eclipse.jgit.internal.storage.file.PackFile.get(PackFile.java:257)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:414)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:383)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:375)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:145)
        at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229)
        at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:840)

        at org.eclipse.jgit.transport.BaseReceivePack.checkConnectivity(BaseReceivePack.java:1110)
        at org.eclipse.jgit.transport.BaseReceivePack.receivePackAndCheckConnectivity(BaseReceivePack.java:884)
        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:192)
        at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:161)
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:115)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)
        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)

        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:442)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:368)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

[3]
[2015-03-21 11:36:23,273] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user <user> account 1019628) during git-receive-pack '<project_name>.git'
com.google.gerrit.sshd.BaseCommand$Failure: fatal: Unpack error, check server log
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:176)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)
        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)

        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:442)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:368)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

Caused by: java.io.IOException: Unpack error on project "<project_name>":
  AdvertiseRefsHook: org.eclipse.jgit.transport.AdvertiseRefsHookChain@403e65c3class org.eclipse.jgit.transport.AdvertiseRefsHookChain

        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:175)
        ... 12 more
Caused by: org.eclipse.jgit.errors.UnpackException: Exception while parsing pack stream
        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:246)
        at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:161)
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:115)
        ... 12 more
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 36ee45e2d3d2e3b25e8fb0789beb10c2babce519
        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:148)

        at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229)
        at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:840)

        at org.eclipse.jgit.transport.BaseReceivePack.checkConnectivity(BaseReceivePack.java:1110)
        at org.eclipse.jgit.transport.BaseReceivePack.receivePackAndCheckConnectivity(BaseReceivePack.java:884)
        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:192)
        ... 14 more

[4]
ERROR: Exception caught while accessing pack file /<path><project_name>.git/objects/pack/pack-27ae30fb4364740fb7878b5253af33215134d2cf.pack, the pack file might be corrupt

java.io.IOException: Stale file handle
        at java.io.RandomAccessFile.readBytes0(Native Method)
        at java.io.RandomAccessFile.readBytes(RandomAccessFile.java:350)
        at java.io.RandomAccessFile.read(RandomAccessFile.java:385)
        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444)
        at org.eclipse.jgit.internal.storage.file.PackFile.read(PackFile.java:645)
        at org.eclipse.jgit.internal.storage.file.WindowCache.load(WindowCache.java:289)
        at org.eclipse.jgit.internal.storage.file.WindowCache.getOrLoad(WindowCache.java:368)
        at org.eclipse.jgit.internal.storage.file.WindowCache.get(WindowCache.java:179)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.pin(WindowCursor.java:354)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.copy(WindowCursor.java:226)
        at org.eclipse.jgit.internal.storage.file.PackFile.readFully(PackFile.java:556)
        at org.eclipse.jgit.internal.storage.file.PackFile.load(PackFile.java:714)
        at org.eclipse.jgit.internal.storage.file.PackFile.get(PackFile.java:257)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:414)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:383)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:375)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:145)
        at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229)
        at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:840)

        at com.google.gerrit.server.index.ChangeBatchIndexer$ProjectIndexer.call(ChangeBatchIndexer.java:295)
        at com.google.gerrit.server.index.ChangeBatchIndexer$3.call(ChangeBatchIndexer.java:242)
        at com.google.gerrit.server.index.ChangeBatchIndexer$3.call(ChangeBatchIndexer.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
        at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58)
        at com.google.gerrit.server.index.ChangeBatchIndexer.indexAll(ChangeBatchIndexer.java:156)
        at com.google.gerrit.lucene.OnlineReindexer.reindex(OnlineReindexer.java:79)
        at com.google.gerrit.lucene.OnlineReindexer.access$000(OnlineReindexer.java:33)
        at com.google.gerrit.lucene.OnlineReindexer$1.run(OnlineReindexer.java:62)
[5]
[2015-03-21 06:09:57,773] ERROR com.google.gerrit.server.index.ChangeBatchIndexer : Failed to index project<project_name>
java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 9aae3b3c6e95d8ec29b000097e930533cdfc7487
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:188)
        at com.google.gerrit.server.index.ChangeBatchIndexer$1.run(ChangeBatchIndexer.java:164)
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
        at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
        at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:101)
        at com.google.common.util.concurrent.ListenableFutureTask.addListener(ListenableFutureTask.java:83)
        at com.google.gerrit.server.index.ChangeBatchIndexer.indexAll(ChangeBatchIndexer.java:160)
        at com.google.gerrit.lucene.OnlineReindexer.reindex(OnlineReindexer.java:79)
        at com.google.gerrit.lucene.OnlineReindexer.access$000(OnlineReindexer.java:33)
        at com.google.gerrit.lucene.OnlineReindexer$1.run(OnlineReindexer.java:62)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 9aae3b3c6e95d8ec29b000097e930533cdfc7487
        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:148)

        at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229)
        at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:840)

        at com.google.gerrit.server.index.ChangeBatchIndexer$ProjectIndexer.call(ChangeBatchIndexer.java:295)
        at com.google.gerrit.server.index.ChangeBatchIndexer$3.call(ChangeBatchIndexer.java:242)
        at com.google.gerrit.server.index.ChangeBatchIndexer$3.call(ChangeBatchIndexer.java:226)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
        at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58)
        at com.google.gerrit.server.index.ChangeBatchIndexer.indexAll(ChangeBatchIndexer.java:156)
        ... 3 more

[Saša Živkov]

On Wed, Mar 25, 2015 at 8:59 PM, Hugo Arès <hugo...@ericsson.com> wrote:

Lets me step back and start over so you can understand how I ended up looking into that commit.

We upgraded from Gerrit 2.9.4 to 2.10 (which upgraded JGit from 3.4.2 to 3.6.2) and the users reported the 2 following issues:
-I cannot see my project: we fixed this one by restarting Gerrit (we had to, flush-cahes did not help),see stacktrace [1]

I bet the 67b30f728830954ff823df1ea5fe69e81a04d7cd was the tip of the refs/meta/config branch in that repository?

The JGit change 39685 was done to fix exactly this issue :-|

-I cannot push: we fixed some by running git repack and others were fixed when we had to restart for the previous issue, see stactraces [2] and [3].

During the upgrade, we also had an issue, the online reindexed failed(see stack trace [4] and [5]), we disabled our garbage collection and restarted Gerrit, the online reindexer completed successfully this time.

The only thing we changed during the upgrade is Gerrit, we have the same setup; storage, GC cron, Git version use by GC cron. Looking at the logs, every time there is an issue, there is a stacktrace about a missing commit following a stacktrace about an IOException accessing the pack file which lead me (maybe wrongfully) to think that https://git.eclipse.org/r/#/c/39685 could have caused the problems since it changed the way it handles IOException.

Those issues were only reported on our busiest repositories, which happen to be GC more frequently than others (every 2 hours). The current situation is that we reduced the frequecy we run gc(to once a day in during the non-working hours) and we do not see the issues anymore.

In the repositories where we had issue, we configured the core.trustfolderstat option and it look like we no longer have the IOException "Stale file hande" but we have the FileNotFoundException which should be fixed by https://git.eclipse.org/r/#/c/40503/. Should this change be in JGit 3.6, its only in 3.7 for now.

The JGit version in stable-2.10 (3.7.0.201502260915-r.58-g65c379e) contains the change 40503 which means that

2.10.2 will also contains it.

[Selberg, Sven]

It looks like the same issue we suffered from, at least the missing projects.

It came down to pack file naming confusion between Git and JGit.

Git created new pack files with the same name as the previous pack file, JGit didn’t like it at all.

We solved it by upgrading Git version.

Gustaf explained this in the thread [1].

 

/Sven

 

[1] https://groups.google.com/forum/#!searchin/repo-discuss/jgit/repo-discuss/Qdmbl-YZ4NU/VmLKVk4CnkEJ

[Hugo Arès]

Finally, the issue(missing projects) was really caused by a change in JGit. The project disappears if there is an error while reading the pack file containing the tip of the refs/meta/config branch.

In our case, the error while reading the pack file is "stale file handle" which only happens if the repo is on NFS and GC is ran from another machine. It means that Gerrit/JGit is trying to read a pack file that was deleted or replaced. I submitted 2 changes to fix the issue[1][2] which will be in JGit 4.0.0.

Then we upgraded to jgit 3.7.0.201502260915-r.58-g65c379e(same version as Gerrit 2.10.2) +[1][2]  and we had the memory leak issue[3].

To stabilize our production, we reverted to JGit 3.6.2  +

ae4b72d50e84f9af7978075d4919ad4d2b1d018a Remove pack from list when file handle is stale
d3a73c1407f8936cb45516da8d3a2d4dd43c6379 Lower log level to warn for handled pack errors
57644f23a1481cb6be0e3e9515b62ff1be028f78 Provide more details in exceptions thrown when packfile is invalid
a8743df19c2574bc3481fa5f811499b5960951a8 Add log4j and slf4j-log4j bridge to jgit feature
c18694e0d1c122a2f95537fd9254fb075d2e0bc2 Use slf4j to log instead of printing to System.err
f5936405a3a66b821f716e551de6ee4c1c33ca0b If a pack isn't found on disk remove it from pack list
954306be814846c33371b7b0edadc85ebe1c8fe5 Make MyersDiff interruptible

Hugo

[1]https://git.eclipse.org/r/#/c/45641/
[2]https://git.eclipse.org/r/#/c/45649/
[3]https://groups.google.com/forum/#!topic/repo-discuss/RRQT_xCqz4o

[Dave Castagna (Motorola Mobility)]

I can't access that link. What Git version did you move to please?

[Dave Castagna (Motorola Mobility)]

Sorry. That question was for Sven.

[Dave Castagna (Motorola Mobility)]

[David Pursehouse]

Try this:

https://groups.google.com/d/msg/repo-discuss/Qdmbl-YZ4NU/VmLKVk4CnkEJ

On Tue, Nov 24, 2015 at 6:22 PM Dave Castagna (Motorola Mobility) <cast...@motorola.com> wrote:

Sorry.  That question was for Sven.

[David Castagna]

It still seems unclear which Git version I need.

1.9+?

Thanks for replying.

[Bassem Rabil]

According to the commit information for the fix below, you should be fine with git 1.9+.

Regards

Bassem

Author: Jeff King <peff@.net>  2013-12-05 21:28:07

Committer: Junio C Hamano <gitster@.com>  2013-12-06 00:40:11

Parent: ab1900a36ef7fcfe872ff9d0803f9442e356c2f0 (Appease Sun Studio by renaming "tmpfile")

Child:  f06a5e607dde266884db4a99b70fbee09d5c5efc (Merge branch 'jk/sha1write-void')

Branches: remotes/origin/maint, remotes/origin/master, remotes/origin/next, remotes/origin/pu

Follows: v1.8.4.1

Precedes: v1.9-rc0