[3.3.4][GC] Exception caught while accessing pack file
447 views
Skip to first unread message
elzoc...@gmail.com
May 22, 2022, 12:48:54 PM5/22/22
to Repo and Gerrit Discussion
Hi all,
We upgraded from 3.3.10 to Gerrit 3.4.4 a couple of weeks back.
On Sunday PM we have the git gc running (initiated by gerrit via [gc] in gerrit.config)
Since the upgrade, for a few of the projects, we have errors such as the one below.
We upgraded from 3.3.10 to Gerrit 3.4.4 a couple of weeks back.
On Sunday PM we have the git gc running (initiated by gerrit via [gc] in gerrit.config)
Since the upgrade, for a few of the projects, we have errors such as the one below.
I listed all the pack files of that project before the GC started, pack pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx was present (and created yesterday) before GC started.
Re-running Gc on that project a few minutes later does not pop new errors in the error_log.
Running git fsck on that project (after gc) shows 60 dangling commits and 2 dangling tag, no errors.
No problem afaik on the projects except for those errors.
Those errors were not present on Gerrit 3.3.x
Are those errors "normal" ?
Thanks,
Cedric.
ls /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/ | grep cbf5a1ab49b39fdb91e8cc146be310d8c3852b59
-r--r--r-- 1 gerrit2 gerrit2 364161 May 21 13:47 pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.pack
-r--r--r-- 1 gerrit2 gerrit2 2220 May 21 13:47 pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx
cat /gerrit/logs/error_log | grep 'Exception caught while accessing pack file' | grep path/pathagain/andthen/project.git | wc -l
60
gc.log:
[2022-05-22 17:01:21,914] INFO : [path/pathagain/andthen/project] gc config: gc.aggressive=false;
[2022-05-22 17:01:21,914] INFO : [path/pathagain/andthen/project] pack config: maxDeltaDepth=50, deltaSearchWindowSize=10, deltaSearchMemoryLimit=0, deltaCacheSize=52428800, deltaCacheLimit=100, compressionLevel=-1, indexVersion=2, bigFileThreshold=52428800, threads=0, reuseDeltas=true, reuseObjects=true, deltaCompress=true, buildBitmaps=true, bitmapContiguousCommitCount=100, bitmapRecentCommitCount=20000, bitmapRecentCommitSpan=100, bitmapDistantCommitSpan=5000, bitmapExcessiveBranchCount=100, bitmapInactiveBranchAge=90, singlePack=false
[2022-05-22 17:01:25,274] INFO : [path/pathagain/andthen/project] before: numberOfPackedRefs=28494, numberOfPackFiles=64, sizeOfLooseObjects=5370906, numberOfLooseObjects=2512, numberOfPackedObjects=386478, numberOfLooseRefs=83, sizeOfPackedObjects=303702313
[2022-05-22 17:02:26,467] INFO : [path/pathagain/andthen/project] after: numberOfPackedRefs=28576, numberOfPackFiles=2, sizeOfLooseObjects=817531, numberOfLooseObjects=263, numberOfPackedObjects=386929, numberOfLooseRefs=1, sizeOfPackedObjects=291971282
error.log
[2022-05-22T17:02:00.358+01:00] [WorkQueue-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@d3c931b[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@3064eddd[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@3cb8492e]]]] ERROR org.eclipse.jgit.internal.storage.file.PackDirectory : Exception caught while accessing pack file /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.pack, the pack file might be corrupt. Caught 1 consecutive errors while trying to read this pack.
java.io.IOException: Unreadable pack index: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:70)
at org.eclipse.jgit.internal.storage.file.Pack.idx(Pack.java:161)
at org.eclipse.jgit.internal.storage.file.Pack.get(Pack.java:273)
at org.eclipse.jgit.internal.storage.file.PackDirectory.open(PackDirectory.java:211)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:390)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:354)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObjectWithoutRestoring(ObjectDirectory.java:345)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:330)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:212)
at org.eclipse.jgit.internal.storage.file.GC.loosen(GC.java:294)
at org.eclipse.jgit.internal.storage.file.GC.deleteOldPacks(GC.java:348)
at org.eclipse.jgit.internal.storage.file.GC.repack(GC.java:862)
at org.eclipse.jgit.internal.storage.file.GC.doGc(GC.java:270)
at org.eclipse.jgit.internal.storage.file.GC.gc(GC.java:221)
at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:179)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:99)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:71)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:67)
at com.google.gerrit.server.git.GarbageCollectionRunner.run(GarbageCollectionRunner.java:64)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.FileNotFoundException: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at org.eclipse.jgit.util.io.SilentFileInputStream.<init>(SilentFileInputStream.java:31)
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:65)
... 27 more
[2022-05-22T17:02:00.360+01:00] [WorkQueue-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@d3c931b[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@3064eddd[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@3cb8492e]]]] ERROR org.eclipse.jgit.internal.storage.file.PackDirectory : Exception caught while accessing pack file /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-df5110ae359eef73fd544c067ffad087da5b939c.pack, the pack file might be corrupt. Caught 1 consecutive errors while trying to read this pack.
java.io.IOException: Unreadable pack index: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-df5110ae359eef73fd544c067ffad087da5b939c.idx
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:70)
at org.eclipse.jgit.internal.storage.file.Pack.idx(Pack.java:161)
at org.eclipse.jgit.internal.storage.file.Pack.get(Pack.java:273)
at org.eclipse.jgit.internal.storage.file.PackDirectory.open(PackDirectory.java:211)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:390)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:354)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObjectWithoutRestoring(ObjectDirectory.java:345)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:330)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:212)
at org.eclipse.jgit.internal.storage.file.GC.loosen(GC.java:294)
at org.eclipse.jgit.internal.storage.file.GC.deleteOldPacks(GC.java:348)
at org.eclipse.jgit.internal.storage.file.GC.repack(GC.java:862)
at org.eclipse.jgit.internal.storage.file.GC.doGc(GC.java:270)
at org.eclipse.jgit.internal.storage.file.GC.gc(GC.java:221)
at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:179)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:99)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:71)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:67)
at com.google.gerrit.server.git.GarbageCollectionRunner.run(GarbageCollectionRunner.java:64)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.FileNotFoundException: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-df5110ae359eef73fd544c067ffad087da5b939c.idx (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at org.eclipse.jgit.util.io.SilentFileInputStream.<init>(SilentFileInputStream.java:31)
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:65)
... 27 more
[2022-05-22T17:02:00.541+01:00] [WorkQueue-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@d3c931b[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@3064eddd[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@3cb8492e]]]] WARN org.eclipse.jgit.internal.storage.file.PackDirectory : Pack file /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.pack is corrupt, removing it from pack list
org.eclipse.jgit.errors.PackInvalidException: Pack file invalid: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.pack
at org.eclipse.jgit.internal.storage.file.Pack.idx(Pack.java:155)
at org.eclipse.jgit.internal.storage.file.Pack.get(Pack.java:273)
at org.eclipse.jgit.internal.storage.file.PackDirectory.open(PackDirectory.java:211)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:390)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:354)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObjectWithoutRestoring(ObjectDirectory.java:345)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:330)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:212)
at org.eclipse.jgit.internal.storage.file.GC.loosen(GC.java:294)
at org.eclipse.jgit.internal.storage.file.GC.deleteOldPacks(GC.java:348)
at org.eclipse.jgit.internal.storage.file.GC.repack(GC.java:862)
at org.eclipse.jgit.internal.storage.file.GC.doGc(GC.java:270)
at org.eclipse.jgit.internal.storage.file.GC.gc(GC.java:221)
at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:179)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:99)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:71)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:67)
at com.google.gerrit.server.git.GarbageCollectionRunner.run(GarbageCollectionRunner.java:64)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Unreadable pack index: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:70)
at org.eclipse.jgit.internal.storage.file.Pack.idx(Pack.java:161)
... 26 more
Caused by: java.io.FileNotFoundException: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at org.eclipse.jgit.util.io.SilentFileInputStream.<init>(SilentFileInputStream.java:31)
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:65)
... 27 more
[2022-05-22T17:02:00.544+01:00] [WorkQueue-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@d3c931b[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@3064eddd[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@3cb8492e]]]] WARN org.eclipse.jgit.internal.storage.file.PackDirectory : Pack file /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.pack is corrupt, removing it from pack list
org.eclipse.jgit.errors.PackInvalidException: Pack file invalid: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.pack
at org.eclipse.jgit.internal.storage.file.Pack.idx(Pack.java:155)
at org.eclipse.jgit.internal.storage.file.Pack.get(Pack.java:273)
at org.eclipse.jgit.internal.storage.file.PackDirectory.open(PackDirectory.java:211)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:390)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:354)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObjectWithoutRestoring(ObjectDirectory.java:345)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:330)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:212)
at org.eclipse.jgit.internal.storage.file.GC.loosen(GC.java:294)
at org.eclipse.jgit.internal.storage.file.GC.deleteOldPacks(GC.java:348)
at org.eclipse.jgit.internal.storage.file.GC.repack(GC.java:862)
at org.eclipse.jgit.internal.storage.file.GC.doGc(GC.java:270)
at org.eclipse.jgit.internal.storage.file.GC.gc(GC.java:221)
at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:179)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:99)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:71)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:67)
at com.google.gerrit.server.git.GarbageCollectionRunner.run(GarbageCollectionRunner.java:64)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Unreadable pack index: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:70)
at org.eclipse.jgit.internal.storage.file.Pack.idx(Pack.java:161)
... 26 more
Caused by: java.io.FileNotFoundException: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at org.eclipse.jgit.util.io.SilentFileInputStream.<init>(SilentFileInputStream.java:31)
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:65)
... 27 more
Re-running Gc on that project a few minutes later does not pop new errors in the error_log.
Running git fsck on that project (after gc) shows 60 dangling commits and 2 dangling tag, no errors.
No problem afaik on the projects except for those errors.
Those errors were not present on Gerrit 3.3.x
Are those errors "normal" ?
Thanks,
Cedric.
ls /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/ | grep cbf5a1ab49b39fdb91e8cc146be310d8c3852b59
-r--r--r-- 1 gerrit2 gerrit2 364161 May 21 13:47 pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.pack
-r--r--r-- 1 gerrit2 gerrit2 2220 May 21 13:47 pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx
cat /gerrit/logs/error_log | grep 'Exception caught while accessing pack file' | grep path/pathagain/andthen/project.git | wc -l
60
gc.log:
[2022-05-22 17:01:21,914] INFO : [path/pathagain/andthen/project] gc config: gc.aggressive=false;
[2022-05-22 17:01:21,914] INFO : [path/pathagain/andthen/project] pack config: maxDeltaDepth=50, deltaSearchWindowSize=10, deltaSearchMemoryLimit=0, deltaCacheSize=52428800, deltaCacheLimit=100, compressionLevel=-1, indexVersion=2, bigFileThreshold=52428800, threads=0, reuseDeltas=true, reuseObjects=true, deltaCompress=true, buildBitmaps=true, bitmapContiguousCommitCount=100, bitmapRecentCommitCount=20000, bitmapRecentCommitSpan=100, bitmapDistantCommitSpan=5000, bitmapExcessiveBranchCount=100, bitmapInactiveBranchAge=90, singlePack=false
[2022-05-22 17:01:25,274] INFO : [path/pathagain/andthen/project] before: numberOfPackedRefs=28494, numberOfPackFiles=64, sizeOfLooseObjects=5370906, numberOfLooseObjects=2512, numberOfPackedObjects=386478, numberOfLooseRefs=83, sizeOfPackedObjects=303702313
[2022-05-22 17:02:26,467] INFO : [path/pathagain/andthen/project] after: numberOfPackedRefs=28576, numberOfPackFiles=2, sizeOfLooseObjects=817531, numberOfLooseObjects=263, numberOfPackedObjects=386929, numberOfLooseRefs=1, sizeOfPackedObjects=291971282
error.log
[2022-05-22T17:02:00.358+01:00] [WorkQueue-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@d3c931b[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@3064eddd[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@3cb8492e]]]] ERROR org.eclipse.jgit.internal.storage.file.PackDirectory : Exception caught while accessing pack file /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.pack, the pack file might be corrupt. Caught 1 consecutive errors while trying to read this pack.
java.io.IOException: Unreadable pack index: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:70)
at org.eclipse.jgit.internal.storage.file.Pack.idx(Pack.java:161)
at org.eclipse.jgit.internal.storage.file.Pack.get(Pack.java:273)
at org.eclipse.jgit.internal.storage.file.PackDirectory.open(PackDirectory.java:211)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:390)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:354)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObjectWithoutRestoring(ObjectDirectory.java:345)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:330)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:212)
at org.eclipse.jgit.internal.storage.file.GC.loosen(GC.java:294)
at org.eclipse.jgit.internal.storage.file.GC.deleteOldPacks(GC.java:348)
at org.eclipse.jgit.internal.storage.file.GC.repack(GC.java:862)
at org.eclipse.jgit.internal.storage.file.GC.doGc(GC.java:270)
at org.eclipse.jgit.internal.storage.file.GC.gc(GC.java:221)
at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:179)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:99)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:71)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:67)
at com.google.gerrit.server.git.GarbageCollectionRunner.run(GarbageCollectionRunner.java:64)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.FileNotFoundException: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at org.eclipse.jgit.util.io.SilentFileInputStream.<init>(SilentFileInputStream.java:31)
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:65)
... 27 more
[2022-05-22T17:02:00.360+01:00] [WorkQueue-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@d3c931b[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@3064eddd[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@3cb8492e]]]] ERROR org.eclipse.jgit.internal.storage.file.PackDirectory : Exception caught while accessing pack file /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-df5110ae359eef73fd544c067ffad087da5b939c.pack, the pack file might be corrupt. Caught 1 consecutive errors while trying to read this pack.
java.io.IOException: Unreadable pack index: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-df5110ae359eef73fd544c067ffad087da5b939c.idx
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:70)
at org.eclipse.jgit.internal.storage.file.Pack.idx(Pack.java:161)
at org.eclipse.jgit.internal.storage.file.Pack.get(Pack.java:273)
at org.eclipse.jgit.internal.storage.file.PackDirectory.open(PackDirectory.java:211)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:390)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:354)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObjectWithoutRestoring(ObjectDirectory.java:345)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:330)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:212)
at org.eclipse.jgit.internal.storage.file.GC.loosen(GC.java:294)
at org.eclipse.jgit.internal.storage.file.GC.deleteOldPacks(GC.java:348)
at org.eclipse.jgit.internal.storage.file.GC.repack(GC.java:862)
at org.eclipse.jgit.internal.storage.file.GC.doGc(GC.java:270)
at org.eclipse.jgit.internal.storage.file.GC.gc(GC.java:221)
at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:179)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:99)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:71)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:67)
at com.google.gerrit.server.git.GarbageCollectionRunner.run(GarbageCollectionRunner.java:64)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.FileNotFoundException: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-df5110ae359eef73fd544c067ffad087da5b939c.idx (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at org.eclipse.jgit.util.io.SilentFileInputStream.<init>(SilentFileInputStream.java:31)
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:65)
... 27 more
[2022-05-22T17:02:00.541+01:00] [WorkQueue-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@d3c931b[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@3064eddd[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@3cb8492e]]]] WARN org.eclipse.jgit.internal.storage.file.PackDirectory : Pack file /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.pack is corrupt, removing it from pack list
org.eclipse.jgit.errors.PackInvalidException: Pack file invalid: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.pack
at org.eclipse.jgit.internal.storage.file.Pack.idx(Pack.java:155)
at org.eclipse.jgit.internal.storage.file.Pack.get(Pack.java:273)
at org.eclipse.jgit.internal.storage.file.PackDirectory.open(PackDirectory.java:211)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:390)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:354)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObjectWithoutRestoring(ObjectDirectory.java:345)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:330)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:212)
at org.eclipse.jgit.internal.storage.file.GC.loosen(GC.java:294)
at org.eclipse.jgit.internal.storage.file.GC.deleteOldPacks(GC.java:348)
at org.eclipse.jgit.internal.storage.file.GC.repack(GC.java:862)
at org.eclipse.jgit.internal.storage.file.GC.doGc(GC.java:270)
at org.eclipse.jgit.internal.storage.file.GC.gc(GC.java:221)
at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:179)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:99)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:71)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:67)
at com.google.gerrit.server.git.GarbageCollectionRunner.run(GarbageCollectionRunner.java:64)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Unreadable pack index: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:70)
at org.eclipse.jgit.internal.storage.file.Pack.idx(Pack.java:161)
... 26 more
Caused by: java.io.FileNotFoundException: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at org.eclipse.jgit.util.io.SilentFileInputStream.<init>(SilentFileInputStream.java:31)
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:65)
... 27 more
[2022-05-22T17:02:00.544+01:00] [WorkQueue-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@d3c931b[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@3064eddd[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@3cb8492e]]]] WARN org.eclipse.jgit.internal.storage.file.PackDirectory : Pack file /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.pack is corrupt, removing it from pack list
org.eclipse.jgit.errors.PackInvalidException: Pack file invalid: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.pack
at org.eclipse.jgit.internal.storage.file.Pack.idx(Pack.java:155)
at org.eclipse.jgit.internal.storage.file.Pack.get(Pack.java:273)
at org.eclipse.jgit.internal.storage.file.PackDirectory.open(PackDirectory.java:211)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:390)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:354)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObjectWithoutRestoring(ObjectDirectory.java:345)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:330)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:212)
at org.eclipse.jgit.internal.storage.file.GC.loosen(GC.java:294)
at org.eclipse.jgit.internal.storage.file.GC.deleteOldPacks(GC.java:348)
at org.eclipse.jgit.internal.storage.file.GC.repack(GC.java:862)
at org.eclipse.jgit.internal.storage.file.GC.doGc(GC.java:270)
at org.eclipse.jgit.internal.storage.file.GC.gc(GC.java:221)
at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:179)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:99)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:71)
at com.google.gerrit.server.git.GarbageCollection.run(GarbageCollection.java:67)
at com.google.gerrit.server.git.GarbageCollectionRunner.run(GarbageCollectionRunner.java:64)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Unreadable pack index: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:70)
at org.eclipse.jgit.internal.storage.file.Pack.idx(Pack.java:161)
... 26 more
Caused by: java.io.FileNotFoundException: /gerrit/shared/git/path/pathagain/andthen/project.git/objects/pack/pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at org.eclipse.jgit.util.io.SilentFileInputStream.<init>(SilentFileInputStream.java:31)
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:65)
... 27 more
Luca Milanesio
May 22, 2022, 1:28:15 PM5/22/22
to Repo and Gerrit Discussion, Luca Milanesio, elzoc...@gmail.com
On 22 May 2022, at 17:48, elzoc...@gmail.com <elzoc...@gmail.com> wrote:Hi all,
We upgraded from 3.3.10 to Gerrit 3.4.4 a couple of weeks back.
On Sunday PM we have the git gc running (initiated by gerrit via [gc] in gerrit.config)
Since the upgrade, for a few of the projects, we have errors such as the one below.
What is your prune policy?
(pruneExpire and prunePackExpire in jgit.config or repo’s config)
I listed all the pack files of that project before the GC started, pack pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx was present (and created yesterday) before GC started.
Re-running Gc on that project a few minutes later does not pop new errors in the error_log.
Running git fsck on that project (after gc) shows 60 dangling commits and 2 dangling tag, no errors.
The GC repacked the project and that pack was removed.
No problem afaik on the projects except for those errors.
Those errors were not present on Gerrit 3.3.x
Possibly just JGit was discarding them silently?
Do you see any user-face error?
Are those errors "normal" ?
Depending on your prune policy, it *may* be a normal behaviour.
Luca.
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/fb1ed3f4-2dc6-4732-b8f9-1e394b1567d7n%40googlegroups.com.
elzoc...@gmail.com
May 22, 2022, 1:42:12 PM5/22/22
to Repo and Gerrit Discussion
Le dimanche 22 mai 2022 à 18:28:15 UTC+1, lucamilanesio a écrit :
On 22 May 2022, at 17:48, elzoc...@gmail.com <elzoc...@gmail.com> wrote:Hi all,
We upgraded from 3.3.10 to Gerrit 3.4.4 a couple of weeks back.
On Sunday PM we have the git gc running (initiated by gerrit via [gc] in gerrit.config)
Since the upgrade, for a few of the projects, we have errors such as the one below.What is your prune policy?(pruneExpire and prunePackExpire in jgit.config or repo’s config)
No prune policy set, so default. Should I set anything?
cat /gerrit/etc/jgit.config
[receive]
autogc = false
[protocol]
version = 2
[receive]
autogc = false
[protocol]
version = 2
cat /gerrit/shared/git/path/pathagain/andthen/project.git/config
[core]
repositoryformatversion = 0
filemode = true
bare = true
logallrefupdates = true
[core]
repositoryformatversion = 0
filemode = true
bare = true
logallrefupdates = true
I listed all the pack files of that project before the GC started, pack pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx was present (and created yesterday) before GC started.
Re-running Gc on that project a few minutes later does not pop new errors in the error_log.
Running git fsck on that project (after gc) shows 60 dangling commits and 2 dangling tag, no errors.The GC repacked the project and that pack was removed.
No problem afaik on the projects except for those errors.
Those errors were not present on Gerrit 3.3.xPossibly just JGit was discarding them silently?Do you see any user-face error?
No, as far as I know, repos are OK, I have no problem cloning, UI browsing, and there were plenty of pushes during the week, I have not been made aware of any problem.
Thanks again!
Luca Milanesio
May 22, 2022, 2:49:33 PM5/22/22
to Repo and Gerrit Discussion, Luca Milanesio, elzoc...@gmail.com
On 22 May 2022, at 18:42, elzoc...@gmail.com <elzoc...@gmail.com> wrote:Le dimanche 22 mai 2022 à 18:28:15 UTC+1, lucamilanesio a écrit :On 22 May 2022, at 17:48, elzoc...@gmail.com <elzoc...@gmail.com> wrote:Hi all,
We upgraded from 3.3.10 to Gerrit 3.4.4 a couple of weeks back.
On Sunday PM we have the git gc running (initiated by gerrit via [gc] in gerrit.config)
Since the upgrade, for a few of the projects, we have errors such as the one below.What is your prune policy?(pruneExpire and prunePackExpire in jgit.config or repo’s config)No prune policy set, so default.
Your settings would then be documented at [1] (see below).
If you are running GC less often than every hour, you would then risk to remove packfiles that may have been actually loaded and referred by JGit in its cache.
Hence the errors you saw in the logs.
Should I set anything?cat /gerrit/etc/jgit.config
[receive]
autogc = false
[protocol]
version = 2
How often are you GCing the repos?
If you do GC less than every hour, then the default would suffice.
Otherwise, set the prune pack expire longer than your interval between GCs.
cat /gerrit/shared/git/path/pathagain/andthen/project.git/config
[core]
repositoryformatversion = 0
filemode = true
bare = true
logallrefupdates = trueI listed all the pack files of that project before the GC started, pack pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx was present (and created yesterday) before GC started.
Re-running Gc on that project a few minutes later does not pop new errors in the error_log.
Running git fsck on that project (after gc) shows 60 dangling commits and 2 dangling tag, no errors.The GC repacked the project and that pack was removed.
No problem afaik on the projects except for those errors.
Those errors were not present on Gerrit 3.3.xPossibly just JGit was discarding them silently?Do you see any user-face error?No, as far as I know, repos are OK, I have no problem cloning, UI browsing, and there were plenty of pushes during the week, I have not been made aware of any problem.
You can safe ignore those messages in the logs then.
Luca.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/00a8e169-c71a-4997-a52a-31f76cab6d69n%40googlegroups.com.
elzoc...@gmail.com
May 22, 2022, 3:27:16 PM5/22/22
to Repo and Gerrit Discussion
Le dimanche 22 mai 2022 à 19:49:33 UTC+1, lucamilanesio a écrit :
On 22 May 2022, at 18:42, elzoc...@gmail.com <elzoc...@gmail.com> wrote:Le dimanche 22 mai 2022 à 18:28:15 UTC+1, lucamilanesio a écrit :On 22 May 2022, at 17:48, elzoc...@gmail.com <elzoc...@gmail.com> wrote:Hi all,
We upgraded from 3.3.10 to Gerrit 3.4.4 a couple of weeks back.
On Sunday PM we have the git gc running (initiated by gerrit via [gc] in gerrit.config)
Since the upgrade, for a few of the projects, we have errors such as the one below.What is your prune policy?(pruneExpire and prunePackExpire in jgit.config or repo’s config)No prune policy set, so default.Your settings would then be documented at [1] (see below).If you are running GC less often than every hour, you would then risk to remove packfiles that may have been actually loaded and referred by JGit in its cache.Hence the errors you saw in the logs.Should I set anything?cat /gerrit/etc/jgit.config
[receive]
autogc = false
[protocol]
version = 2How often are you GCing the repos?
we are doing gc only once a week, we were doing it daily before but I had to make it weekly due to issue when main does GC, shared nfs readonly replicas have failing clones due to pack file gc-ed. I didn't want to have to monitor daily GC, and do daily restarts of replicas.
If you do GC less than every hour, then the default would suffice.Otherwise, set the prune pack expire longer than your interval between GCs.cat /gerrit/shared/git/path/pathagain/andthen/project.git/config
[core]
repositoryformatversion = 0
filemode = true
bare = true
logallrefupdates = trueI listed all the pack files of that project before the GC started, pack pack-cbf5a1ab49b39fdb91e8cc146be310d8c3852b59.idx was present (and created yesterday) before GC started.
Re-running Gc on that project a few minutes later does not pop new errors in the error_log.
Running git fsck on that project (after gc) shows 60 dangling commits and 2 dangling tag, no errors.The GC repacked the project and that pack was removed.
No problem afaik on the projects except for those errors.
Those errors were not present on Gerrit 3.3.xPossibly just JGit was discarding them silently?Do you see any user-face error?No, as far as I know, repos are OK, I have no problem cloning, UI browsing, and there were plenty of pushes during the week, I have not been made aware of any problem.You can safe ignore those messages in the logs then.
Thanks!
C.
Nasser Grainawi
May 31, 2022, 7:11:47 PM5/31/22
to elzoc...@gmail.com, Repo and Gerrit Discussion
Le dimanche 22 mai 2022 à 19:49:33 UTC+1, lucamilanesio a écrit :On 22 May 2022, at 18:42, elzoc...@gmail.com <elzoc...@gmail.com> wrote:Le dimanche 22 mai 2022 à 18:28:15 UTC+1, lucamilanesio a écrit :On 22 May 2022, at 17:48, elzoc...@gmail.com <elzoc...@gmail.com> wrote:Hi all,
We upgraded from 3.3.10 to Gerrit 3.4.4 a couple of weeks back.
On Sunday PM we have the git gc running (initiated by gerrit via [gc] in gerrit.config)
Since the upgrade, for a few of the projects, we have errors such as the one below.What is your prune policy?(pruneExpire and prunePackExpire in jgit.config or repo’s config)No prune policy set, so default.Your settings would then be documented at [1] (see below).If you are running GC less often than every hour, you would then risk to remove packfiles that may have been actually loaded and referred by JGit in its cache.Hence the errors you saw in the logs.Should I set anything?cat /gerrit/etc/jgit.config
[receive]
autogc = false
[protocol]
version = 2How often are you GCing the repos?we are doing gc only once a week, we were doing it daily before but I had to make it weekly due to issue when main does GC, shared nfs readonly replicas have failing clones due to pack file gc-ed. I didn't want to have to monitor daily GC, and do daily restarts of replicas.
If you use JGit's preserved pack functionality, it should avoid that issue with the read-only replicas. I don't think anyone has added config to gerrit's gc to enable that functionality though, so you'd need to run the JGit CLI yourself or add that ability to Gerrit.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/d95ab09c-6c9b-45d5-b63b-3295fef0b95an%40googlegroups.com.
Musab Shakeel
Nov 21, 2023, 11:45:12 AM11/21/23
to Repo and Gerrit Discussion
@luca based on your answer, are A and B correct:
1. gc.prunePackExpire=1hr gc.interval=2hr
2. gc.prunePackExpire=2hr gc.interval=1hr
1. gc.prunePackExpire=1hr gc.interval=2hr
2. gc.prunePackExpire=2hr gc.interval=1hr
A) 2 is the appropriate configuration.
B) If you use use the configuration in 1, you will get "corrupt packfile" errors as indicated in the error.log posted by the OP (original poster). However, these errors are safe to ignore and do not cause any user-facing errors, just messy logs?
B) If you use use the configuration in 1, you will get "corrupt packfile" errors as indicated in the error.log posted by the OP (original poster). However, these errors are safe to ignore and do not cause any user-facing errors, just messy logs?
?
Thanks!
Luca Milanesio
Nov 21, 2023, 2:51:12 PM11/21/23
to Repo and Gerrit Discussion, Luca Milanesio, Musab Shakeel
On 21 Nov 2023, at 16:02, 'Musab Shakeel' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:@luca based on your answer, are A and B correct:
1. gc.prunePackExpire=1hr gc.interval=2hr
2. gc.prunePackExpire=2hr gc.interval=1hrA) 2 is the appropriate configuration.
B) If you use use the configuration in 1, you will get "corrupt packfile" errors as indicated in the error.log posted by the OP (original poster). However, these errors are safe to ignore and do not cause any user-facing errors, just messy logs?
If you prune the packs that existed just before the GC cycle, you would risk some clone to fail because they are pointing to non-existent files.
The suggestion is to keep them for at least one cycle.
So, if you are doing GC every 1h, you keep them for at least 1h.
HTH
Luca.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/3e5a6bfd-d619-49b2-ad59-2eac0bc4be64n%40googlegroups.com.
Musab Shakeel
Nov 21, 2023, 3:08:38 PM11/21/23
to Repo and Gerrit Discussion
I see.
1. So the following would be an appropriate configuration: gc.prunePackExpire=2hr gc.interval=1hr ?
2. Gerrit docs here [1] cover some jgit config options but not all. Is there a reason why the gc.prunePackExpire and other options are not in the docs? Would it make sense if we added a pointer to the actual jgit config docs [2] in [1]?
3. What is the difference between the gerrit.config [core] options vs jgit.config [core] options? Seems like you could specify for example core.packedGitWindowSize in both - gerrit.config [core] or jgit.config [core]?
Luca Milanesio
Nov 21, 2023, 3:18:46 PM11/21/23
to Musab Shakeel, Luca Milanesio, Repo and Gerrit Discussion
Hi Musab,
On 21 Nov 2023, at 20:08, 'Musab Shakeel' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:I see.1. So the following would be an appropriate configuration: gc.prunePackExpire=2hr gc.interval=1hr ?
My suggestion was: "So, if you are doing GC every 1h, you keep them for at least 1h.”, I believe 2 > 1
2. Gerrit docs here [1] cover some jgit config options but not all. Is there a reason why the gc.prunePackExpire and other options are not in the docs?
I believe all the JGit options are in the docs. If any of them are missing, feel free to contribute a fix.
The JGit doc is at [3], not [2].
Would it make sense if we added a pointer to the actual jgit config docs [2] in [1]?
The JGit documentation is pointed by the project homepage already, but feel free to contribute a fix to Gerrit documentation with an extra pointer.
3. What is the difference between the gerrit.config [core] options vs jgit.config [core] options? Seems like you could specify for example core.packedGitWindowSize in both - gerrit.config [core] or jgit.config [core]?
That is quite messy I’m afraid:
- jgit.config => used by Gerrit for reading the JGit config when running the JGit core
- gerrit.config => the ‘core’ section is used to set some of the JGit config, overriding what is in jgit.config
- .gitconfig => used by JGit when running outside Gerrit
I know it is confusing :-(
Luca.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/b2e6085c-ca4c-42ae-beb0-87c367f0a526n%40googlegroups.com.
Luca Milanesio
Nov 21, 2023, 3:24:12 PM11/21/23
to Repo and Gerrit Discussion, Luca Milanesio, Musab Shakeel
On 21 Nov 2023, at 20:18, Luca Milanesio <luca.mi...@gmail.com> wrote:Hi Musab,On 21 Nov 2023, at 20:08, 'Musab Shakeel' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:I see.1. So the following would be an appropriate configuration: gc.prunePackExpire=2hr gc.interval=1hr ?My suggestion was: "So, if you are doing GC every 1h, you keep them for at least 1h.”, I believe 2 > 12. Gerrit docs here [1] cover some jgit config options but not all. Is there a reason why the gc.prunePackExpire and other options are not in the docs?I believe all the JGit options are in the docs. If any of them are missing, feel free to contribute a fix.The JGit doc is at [3], not [2].
The changes from [2] to [3] happened just yesterday :-)
Luca.
Reply all
Reply to author
Forward
0 new messages