Facing too many files open issues and Gerrit is unresponsive

[choudhar...@gmail.com]

Hi

We are using Gerrit 3.3.4 verison and for some repository we are facing the too many files open exceptions and due to this Gerrit is unresponsive we have also increased the number of files open to max but this probelm is not resolved. Can someone please guide us what exactly is this issues and how to resolve the same.

[2022-02-02T13:37:18.387-06:00] [SSH git-upload-pack /reponame (username)] ERROR org.eclipse.jgit.internal.storage.file.ObjectDirectory : Failed to access pack file /var/lib/gerrit/review_site/git/reponame/objects/pack/pack-f4503bb4caa2f8e6820eea5f540af016c4e98af9.pack, caught 1 consecutive errors while trying to access this pack.
java.io.FileNotFoundException: /var/lib/gerrit/review_site/git/reponame.git/objects/pack/pack-f4503bb4caa2f8e6820eea5f540af016c4e98af9.pack (Too many open files)
        at java.base/java.io.RandomAccessFile.open0(Native Method)
        at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
        at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
        at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
        at org.eclipse.jgit.internal.storage.file.PackFile.doOpen(PackFile.java:658)
        at org.eclipse.jgit.internal.storage.file.PackFile.beginWindowCache(PackFile.java:638)
        at org.eclipse.jgit.internal.storage.file.WindowCache.load(WindowCache.java:511)
        at org.eclipse.jgit.internal.storage.file.WindowCache.getOrLoad(WindowCache.java:603)
        at org.eclipse.jgit.internal.storage.file.WindowCache.get(WindowCache.java:386)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.pin(WindowCursor.java:327)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.copy(WindowCursor.java:226)
        at org.eclipse.jgit.internal.storage.file.PackFile.readFully(PackFile.java:614)
        at org.eclipse.jgit.internal.storage.file.PackFile.representation(PackFile.java:1085)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.selectObjectRepresentation(ObjectDirectory.java:593)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.selectObjectRepresentation(ObjectDirectory.java:584)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.selectObjectRepresentation(WindowCursor.java:177)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.searchForReuse(PackWriter.java:1343)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.searchForReuse(PackWriter.java:1317)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1178)
        at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2312)
        at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2143)
        at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:1077)
        at org.eclipse.jgit.transport.UploadPack.uploadWithExceptionPropagation(UploadPack.java:834)
        at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:748)
        at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:101)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:108)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:73)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:488)
        at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:103)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

[Martin Fick]

On 2022-02-02 12:50, choudhar...@gmail.com wrote:
>
> We are using Gerrit 3.3.4 verison and for some repository we are
> facing the too many files open exceptions and due to this Gerrit is
> unresponsive we have also increased the number of files open to max
> but this probelm is not resolved. Can someone please guide us what
> exactly is this issues and how to resolve the same.
>
> [2022-02-02T13:37:18.387-06:00] [SSH git-upload-pack /reponame
> (username)] ERROR
> org.eclipse.jgit.internal.storage.file.ObjectDirectory : Failed to
> access pack file
> /var/lib/gerrit/review_site/git/reponame/objects/pack/pack-f4503bb4caa2f8e6820eea5f540af016c4e98af9.pack,

How many files are in this directory? Do you run git gc on your repos
regularly?

-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation