How to interrupt Gerrit garbage collector

473 views

Skip to first unread message

Tatiana Girod

unread,

Apr 28, 2017, 1:10:49 PM4/28/17

to Repo and Gerrit Discussion

We are seeing some strange behavior with the Gerrit garbage collector: an attempt to interrupt the running garbage collection task seemingly succeeds, but the actual garbage collection continues nevertheless.
We have currently Gerrit version 2.13.5.
The garbage collection is scheduled to run via the gerrit gc command:

$ ssh -p 29418 review.example.com gerrit gc --all

The gerrit show-queue shows the gc task running:

$ ssh -p 29418 review.example.com gerrit show-queue -w 
Task     State        StartTime         Command
------------------------------------------------------------------------------
91b7ba87              15:00:01.896      gerrit gc --all (user)

When we interrupt it with the Gerrit kill command, the "gerrit gc" task disappears from the show-queue command output:

$ ssh -p 29418 review.example.com kill 91b7ba87

$ ssh -p 29418 review.example.com gerrit show-queue -w 
Task     State        StartTime         Command
------------------------------------------------------------------------------

But it looks like the actual garbage collection still continues after that, since the gc_log keeps growing showing more and more processed Gerrit repositories:

[2017-04-28 16:45:07,196] INFO  : [ABC] gc config: gc.aggressive=false; 
[2017-04-28 16:45:07,196] INFO  : [ABC] pack config: maxDeltaDepth=50, deltaSearchWindowSize=10, deltaSearchMemoryLimit=0, deltaCacheSize=52428800, deltaCacheLimit=100, compressionLevel=-1, indexVersion=2, bigFileThreshold=52428800, threads=0, reuseDeltas=true, reuseObjects=true, deltaCompress=true, buildBitmaps=true, bitmapContiguousCommitCount=100, bitmapRecentCommitCount=20000, bitmapRecentCommitSpan=100, bitmapDistantCommitSpan=5000, bitmapExcessiveBranchCount=100, bitmapInactiveBranchAge=90
[2017-04-28 16:45:07,205] INFO  : [ABC] before: sizeOfPackedObjects=862272, sizeOfLooseObjects=0, numberOfPackedObjects=3774, numberOfPackFiles=2, numberOfPackedRefs=2, numberOfLooseRefs=1, numberOfLooseObjects=0
[2017-04-28 16:45:07,398] INFO  : [ABC] after:  sizeOfPackedObjects=862272, sizeOfLooseObjects=0, numberOfPackedObjects=3774, numberOfPackFiles=2, numberOfPackedRefs=2, numberOfLooseRefs=1, numberOfLooseObjects=0

And new pack files are being created under the <basePath>/REPOSITORY.git/objects/pack.

How can one actually interrupt the Gerrit garbage collector?

Matthias Sohn

unread,

Jul 7, 2017, 10:31:55 AM7/7/17

to Tatiana Girod, Repo and Gerrit Discussion

I tried your scenario with gerrit 2.13.5 and found that killing the gc --all command

interrupts GC and produces the following output:

$ ssh -p 29418 admin@localhost gerrit gc --all

error: garbage collection for project "eclipse/eclipse.jdt.core" failed

error: garbage collection for project "eclipse/eclipse.jdt.ui" failed

when running these 2 commands in a different terminal

$ ssh -p 29418 admin@localhost gerrit show-queue

Task State StartTime Command

------------------------------------------------------------------------------

bba24fe3 16:28:03.467 gerrit gc --all (admin)

5b92338f 23:00:00.002 16:27:36.918 Log File Compressor

------------------------------------------------------------------------------

2 tasks

$ ssh -p 29418 admin@localhost kill bba24fe3

in addition the following error is logged in gerrit error_log

[2017-07-07 16:28:13,134] [SSH gerrit gc --all (admin)] ERROR com.google.gerrit.server.git.GarbageCollection : [eclipse/eclipse.jdt.core]

org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed.

at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:192)

at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:97)

at com.google.gerrit.sshd.commands.GarbageCollectionCommand.runGC(GarbageCollectionCommand.java:90)

at com.google.gerrit.sshd.commands.GarbageCollectionCommand.run(GarbageCollectionCommand.java:66)

at com.google.gerrit.sshd.SshCommand$1.run(SshCommand.java:35)

at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:442)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:417)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.nio.channels.ClosedChannelException

at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110)

at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:379)

at org.eclipse.jgit.internal.storage.file.GC.writePack(GC.java:827)

at org.eclipse.jgit.internal.storage.file.GC.repack(GC.java:597)

at org.eclipse.jgit.internal.storage.file.GC.gc(GC.java:176)

at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:175)

... 13 more

[2017-07-07 16:28:17,642] [SSH gerrit gc --all (admin)] ERROR com.google.gerrit.server.git.GarbageCollection : [eclipse/eclipse.jdt.ui]