Locked resources holding up fetches

[luke....@hpe.com]

We are running Gerrit version 2.15.4 and recently have been having issues with slow fetches that seem to do nothing at all for minutes, at times. While monitoring the queue I noticed something abnormal: A user's fetch was taking significantly longer than normal, and subsequent user fetches behind it were also not making progress. As soon as that initial fetch finished, the others followed suit very quickly. This led me to do some thread dumps to see what was going on, which got me to the below thread.

The highlighted locked resource below was blocking all other user fetches - The other user git-upload-pack threads were blocked on that resource. My question is (hopefully someone can provide insight), what is that resource that is being locked, and why is it being locked for minutes at a time, thus blocking everyone else? The stack trace shows it is stuck building a TagSet. If that is related to tags, we do have a hefty amount of tags in our repo, which we can (and probably should) offload to a different remote.

It may be beneficial to give some info about our environment. We have about 600-700 active developers on a single large project. A fresh clone of the repo is 11G, while the actual git directory for that project on our Gerrit server is 20G, and we run gerrit gc daily. Most fetches of individual refs take between 10-20 seconds, while at times they seem to take minutes, presumably because of the above.

SSH git-upload-pack /<project> (<user>)" #618 prio=1 os_prio=0 tid=0x00007f1c6400a800 nid=0x163e9 runnable [0x00007f1ab6cee000]

java.lang.Thread.State: RUNNABLE

at java.util.zip.Inflater.reset(Native Method)

at java.util.zip.Inflater.reset(Inflater.java:352)

- locked <0x00007f20e9581328> (a java.util.zip.ZStreamRef)

at org.eclipse.jgit.internal.storage.file.WindowCursor.prepareInflater(WindowCursor.java:334)

at org.eclipse.jgit.internal.storage.file.WindowCursor.inflate(WindowCursor.java:300)

at org.eclipse.jgit.internal.storage.file.PackFile.decompress(PackFile.java:357)

at org.eclipse.jgit.internal.storage.file.PackFile.load(PackFile.java:788)

at org.eclipse.jgit.internal.storage.file.PackFile.get(PackFile.java:275)

at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:481)

at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:439)

at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:430)

at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:159)

at org.eclipse.jgit.revwalk.RevWalk.getCachedBytes(RevWalk.java:903)

at org.eclipse.jgit.revwalk.RevCommit.parseHeaders(RevCommit.java:155)

at org.eclipse.jgit.revwalk.PendingGenerator.next(PendingGenerator.java:147)

at org.eclipse.jgit.revwalk.RevWalk.next(RevWalk.java:435)

at com.google.gerrit.server.git.TagSet.build(TagSet.java:179)

at com.google.gerrit.server.git.TagSetHolder.rebuild(TagSetHolder.java:90)

- locked <0x00007f1eebe69a00> (a java.lang.Object)

at com.google.gerrit.server.git.TagSetHolder.matcher(TagSetHolder.java:56)

at com.google.gerrit.server.git.VisibleRefFilter.filter(VisibleRefFilter.java:204)

at com.google.gerrit.server.git.VisibleRefFilter.filter(VisibleRefFilter.java:254)

at com.google.gerrit.server.git.VisibleRefFilter.getAdvertisedRefs(VisibleRefFilter.java:243)

at org.eclipse.jgit.transport.AbstractAdvertiseRefsHook.advertiseRefs(AbstractAdvertiseRefsHook.java:63)

at org.eclipse.jgit.transport.UploadPack.sendAdvertisedRefs(UploadPack.java:896)

at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:726)

at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:673)

at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:77)

at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:102)

at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)

at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:67)

at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:453)

- locked <0x00007f21332abd50> (a com.google.gerrit.sshd.BaseCommand$TaskThunk)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:558)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Thanks,

Luke

[Martin Fick]

On Wednesday, February 6, 2019 12:45:09 PM MST luke....@hpe.com wrote:
> We are running Gerrit version 2.15.4 and recently have been having issues
> with slow fetches that seem to do nothing at all for minutes, at times.
> While monitoring the queue I noticed something abnormal: A user's fetch was
> taking significantly longer than normal, and subsequent user fetches behind
> it were also not making progress. As soon as that initial fetch finished,
> the others followed suit very quickly.

How many ssh threads do you have configured?

> The highlighted locked resource below was blocking all other user fetches -
> The other user git-upload-pack threads were blocked on that resource. My

> question is (hopefully someone can provide insight), *what is that resource

> that is being locked, and why is it being locked for minutes at a time,

> thus blocking everyone else?* The stack trace shows it is stuck building a

> TagSet. If that is related to tags, we do have a hefty amount of tags in
> our repo, which we can (and probably should) offload to a different remote.

How many tags/refs in your repo?

> SSH git-upload-pack /<project> (<user>)" #618 prio=1 os_prio=0
> tid=0x00007f1c6400a800 nid=0x163e9 runnable [0x00007f1ab6cee000]
> java.lang.Thread.State: RUNNABLE

Not stuck, it is running, doing work. Does a second stack trace show a
similar location?


-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

[Matthias Sohn]

TagSetHolder rebuilds the TagCache for a repository in a synchronized block. This cache was introduced in [1]. 

If the cache "git_tags" [2] is too small to hold all the tags your repository has this could probably cause this symptom.

If rebuilding this cache is slow e.g. because your cache is too small and hence the server has to do a

lot of IO this could be another reason why this lock is held for a long time.

• how large is the Java heap ?
• how large is core.packedGitLimit [3] ?
• how many refs / branches / tags does your repository have ?
• how is the cache "git_tags" configured ?
• what's the output of git count-objects -v ?

[1] https://gerrit-review.googlesource.com/c/gerrit/+/24054

[2] https://gerrit-documentation.storage.googleapis.com/Documentation/2.15.4/config-gerrit.html#cache_names

[3] https://gerrit-documentation.storage.googleapis.com/Documentation/2.15.4/config-gerrit.html#core.packedGitLimit

-Matthias

[luke....@hpe.com]

Matthias -  

• how large is the Java heap ?
• how large is core.packedGitLimit [3] ?
• how many refs / branches / tags does your repository have ?
• how is the cache "git_tags" configured ?
• what's the output of git count-objects -v ?

•  55G
  
• 25G
• ~370k refs/~400/ ~600
• The "git_tags" cache is not configured at all - so whatever is set by default, which the documentation doesn't say
• Looking at show-caches It appears to be at 17.95m - and is getting a 99% mem hit ratio, and 100% disk hit ratio
• Executed from the Gerrit server:

                  count: 11014

                  size: 52132

                  in-pack: 8269231

                  packs: 256

                  size-pack: 11909878

                  prune-packable: 114

                  garbage: 2

                  size-garbage: 0

Martin -

How many ssh threads do you have configured?

64 ssh threads, 32 batchThreads

Not stuck, it is running, doing work.  Does a second stack trace show a 
similar location?

Yes, poor choice of words on my part :) But yes, I took about 4 thread dumps, each between 5-10 seconds apart, each dump showed similar stack traces

Thank you for your help!

Luke

[Martin Fick]

On Wednesday, February 6, 2019 2:20:52 PM MST luke....@hpe.com wrote:
> > - how large is the Java heap ?

> > - how large is core.packedGitLimit [3] ?
>
> - 55G
> - 25G

How much RAM on the server?

> > - how many refs / branches / tags does your repository have ?
> - ~370k refs/~400/ ~600

400 branches and 600 tags is not a lot. 370 refs is significant.

We have projects with over 30K tags so I don't think the number of your tags
is problematic. We have projects with over 500Krefs, and that does slow many
things down, but an ls-remote on those projects is still very fast (a few
seconds)

> > - what's the output of git count-objects -v ?
>
> - Executed from the Gerrit server:
>
> count: 11014

Not small, repacking needed?

> size: 52132
> in-pack: 8269231

~8M objects significant

> packs: 256

We notice major slowdowns around 200-300 packs, I would not be surprised if
this is affecting you. You may need to repack more often.

> size-pack: 11909878
> prune-packable: 114
> garbage: 2
> size-garbage: 0

...
> >> A fresh clone the repo is 11G, while the actual git directory for that

> >> project on our Gerrit server is 20G, and we run gerrit gc daily.
>

> 64 ssh threads, 32 batchThreads

That's a lot of threads for a large repo and not very big heap. If we pretend
for a moment that a clone of your repo use 20G, and you get 64 clones at a
time, you don't have anywhere near enough heap for that. Are you having java
GC issues?

> > Not stuck, it is running, doing work. Does a second stack trace show a
> > similar location?
>
> Yes, poor choice of words on my part :) But yes, I took about 4 thread
> dumps, each between 5-10 seconds apart, each dump showed similar stack
> traces

...
> >> at org.eclipse.jgit.revwalk.RevWalk.next(RevWalk.java:435)

Do those dumps also show the RevWalk.next line?

If you have 370K refs, 8M and >200 pack files, I am not too surprised to see
the RevWalk being slow,

[luke....@hpe.com]

How much RAM on the server?

 

 96G

We notice major slowdowns around 200-300 packs, I would not be surprised if 
this is affecting you. You may need to repack more often.

 

This question has been brought up countless times in our org - how *should* we be garbage collecting/repacking/etc.? I've gone through countless threads and have come up with no clear answers. Should we be ignoring gerrit gc altogether and instead doing our own implementation of git gc/git repack -abdf/git prune? Does Gerrit's use of bitmaps collide with the bitmaps generated by git repack?

I *thought* jgit's gc did a repack at the end for you, but git's and jgit's implementations seem to be very different at certain things. I would love to hear how others are running garbage collection, whether it's a combination of gerrit gc and git repacks, or no gerrit gc at all, or other things I'm missing.

That's a lot of threads for a large repo and not very big heap. If we pretend 
for a moment that a clone of your repo use 20G, and you get 64 clones at a 
time, you don't have anywhere near enough heap for that. Are you having java 
GC issues?

This is a fair point, though we don't nearly have that much traffic. Most of our fetches are very shallow, and we point most traffic to cache servers, rather than having everyone ping the server directly. What kind of java GC issues would you expect us to have? Generally gerrit runs great, save for what I'm emailing about here (for now. If there's things we should change, I'm more than happy to do it before they turn into ugly problems).

Do those dumps also show the RevWalk.next line? 

 

They do. :)

-Luke 

[Sven Selberg]

On Thursday, February 7, 2019 at 2:05:07 AM UTC+1, luke....@hpe.com wrote:

How much RAM on the server?

 

 96G

We notice major slowdowns around 200-300 packs, I would not be surprised if 
this is affecting you. You may need to repack more often.

 

This question has been brought up countless times in our org - how *should* we be garbage collecting/repacking/etc.? I've gone through countless threads and have come up with no clear answers. Should we be ignoring gerrit gc altogether and instead doing our own implementation of git gc/git repack -abdf/git prune? Does Gerrit's use of bitmaps collide with the bitmaps generated by git repack?

I *thought* jgit's gc did a repack at the end for you, but git's and jgit's implementations seem to be very different at certain things. I would love to hear how others are running garbage collection, whether it's a combination of gerrit gc and git repacks, or no gerrit gc at all, or other things I'm missing.

We run git ["repack -a -d -b", "pack-refs --all", "prune"] on all repositories weekly.

[Matthias Sohn]

On Thu, Feb 7, 2019 at 5:00 PM Sven Selberg <sven.s...@axis.com> wrote:

On Thursday, February 7, 2019 at 2:05:07 AM UTC+1, luke....@hpe.com wrote:

How much RAM on the server?

 

 96G

We notice major slowdowns around 200-300 packs, I would not be surprised if 
this is affecting you. You may need to repack more often.

 

This question has been brought up countless times in our org - how *should* we be garbage collecting/repacking/etc.? I've gone through countless threads and have come up with no clear answers. Should we be ignoring gerrit gc altogether and instead doing our own implementation of git gc/git repack -abdf/git prune? Does Gerrit's use of bitmaps collide with the bitmaps generated by git repack?

I *thought* jgit's gc did a repack at the end for you, but git's and jgit's implementations seem to be very different at certain things. I would love to hear how others are running garbage collection, whether it's a combination of gerrit gc and git repacks, or no gerrit gc at all, or other things I'm missing.

We run git ["repack -a -d -b", "pack-refs --all", "prune"] on all repositories weekly.

we run gc-all.sh [1] on small and medium sized repositories. For the large ones we run it every 3 hours.

[1] https://gerrit.googlesource.com/k8s-gerrit/+/master/container-images/git-gc/tools/gc-all.sh

 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Martin Fick]

On Wednesday, February 6, 2019 5:05:06 PM MST luke....@hpe.com wrote:
> > > > - how large is the Java heap ?
> > >

> > > - 55G

> > How much RAM on the server?
>
> 96G

A reasonable amount given the heap above (leaves room for stack, FS caching,
and other stuff such as repacking on the server).

> We notice major slowdowns around 200-300 packs, I would not be surprised if
> > this is affecting you. You may need to repack more often.
>
> This question has been brought up countless times in our org - how *should*
> we be garbage collecting/repacking/etc.? I've gone through countless
> threads and have come up with no clear answers. Should we be ignoring
> gerrit gc altogether and instead doing our own implementation of git gc/git
> repack -abdf/git prune?

I suspect most heavy users of Gerrit do not use the Gerrit gc, we run git
manually for better control (see details below) and for distribution across
multiple servers (using NFS).

> Does Gerrit's use of bitmaps collide with the bitmaps generated by git
repack?

They should not collide. They should be compatible, but they will have
different content. I believe jgit (Gerrit gcs implementation) optimizes for
regular clones (i.e. it does not put the refs/changes/* in the main pack with
bitmaps), where as git gc might.

> I *thought* jgit's gc did a repack at the end for you, but git's and jgit's
> implementations seem to be very different at certain things.

Yes they are different, each has their dis/advantages.

> I would *love* to hear how others are running garbage collection

We run it almost hourly (on very busy servers). But the time frequency is not
important, what is important is how many packfiles and loose objects
accumulate before you run it. Every push to a Gerrit server results in a
packfile. If you know on average how many pushes you get per time period, I
would aim to repack way before there are 200 packfiles.

We use git plumbing commands directly. We run:

git reflog expire --expire never --expire-unreachable never

git pack-refs --all

git prune --expire="1 week ago"

And for repacking itself we use:

git-exproll.sh -r 5 --gc-cmd repack -- -d -k -p -B --window-memory 2g

git-exproll is a magic script which keeps larger packs around until there are
enough smaller packs to repack along with it (it targets an exponential pack
size distribution). It is in the gerrit source tree as ./contrib/git-
exproll.sh The switches above make it use:

git repack -d -k -p -B --window-memory 2g

For some repos like our kernels, we add --window 250 --depth 50

The -k and -p above are custom switches of ours which "preserves" packs in a
special "preserved" directory for one extra gc round to avoid stale NFS file
handles. Those options are not in upstream git gc, but they are in jgit!

Note: on some servers where we have patched jgit to be able to "recover" the
preserved packs when needed (https://git.eclipse.org/r/#/c/122288/), we use
the -a switch instead of -B.

> whether it's a combination of gerrit gc and git repacks, or no gerrit gc at
> all, or other things I'm missing.

It likely does not make sense to use both git gc and gerrit gc, they may
interfere with each other. You can use jgit gc manually externally to Gerrit
if you prefer the packfiles that it makes, or anything else it does better
(preserving packfiles for NFS).

> That's a lot of threads for a large repo and not very big heap. If we
>
> > pretend
> > for a moment that a clone of your repo use 20G, and you get 64 clones at a
> > time, you don't have anywhere near enough heap for that. Are you having
> > java
> > GC issues?
>
> This is a fair point, though we don't nearly have that much traffic.

Then you probably don't benefit from that many threads, and are exposing
yourself to the risk of resource exhaustion. We have super busy servers with
way more RAM/heap and we about half that thread count to avoid system
overload. As our repos grow, we find that we need to reduce the count to avoid
java gc issues.

> Most of our fetches are very shallow, and we point most traffic to cache
> servers, rather than having everyone ping the server directly.

Sure. It sounds like you might finally be hitting an issue with your config?

> What kind of java GC issues would you expect us to have? Generally gerrit
> runs great, save for what I'm emailing about here (for now. If there's
> things we should change, I'm more than happy to do it before they turn into
> ugly problems).

When your heap exhausts, java gc will start to run heavily and potentially
lock up the host or Gerrit process and/or use tons of CPU,

[luke....@hpe.com]

This is all great information, thank you all. However, we hit a bit of a snag today when trying to implement some of this and I'd appreciate if anyone has any input on what happened.

We decided we need to stop running gerrit gc, and start running some version of gc/repack/etc. multiple times a day. To start this, we went ahead and ran a repack on our live server (after testing it on our test server) as packs started to increase. Things seemed fine until someone pointed out that their change no longer was reachable in the Gerrit UI. This led us to find that 6 refs somehow got corrupted during the repack and we had 6 instances of something like this:

error:refs/changes/XX/XXXXX/meta does not point to a valid object!

Each change that got corrupted was updated while the repack was in progress.

The only thing I can think here is that we've been running gerrit gc for a long time (jgit), so running git repack caused pack files to be constructed differently than the jgit pack files. This was fine for almost everything, except those 6 changes that received updates (new patchsets) while the refs were being repacked in a different format. Am I on the right track here? Or does anyone have other insights?

We were able to recover 4 of the missing/bad meta refs, and had to update-ref -d the other two. After that the repo was in a happy state.

-Luke

[Martin Fick]

On Thursday, February 7, 2019 3:41:21 PM MST luke....@hpe.com wrote:
> This is all great information, thank you all. However, we hit a bit of a
> snag today when trying to implement some of this and I'd appreciate if
> anyone has any input on what happened.
>
> We decided we need to stop running gerrit gc, and start running some
> version of gc/repack/etc. multiple times a day. To start this, we went
> ahead and ran a repack on our live server (after testing it on our test
> server) as packs started to increase.

What commands and options did you run?

> Things seemed fine until someone
> pointed out that their change no longer was reachable in the Gerrit UI.
> This led us to find that 6 refs somehow got corrupted during the repack and
> we had 6 instances of something like this:
>
> error:refs/changes/XX/XXXXX/meta does not point to a valid object!
>
> Each change that got corrupted was updated while the repack was in progress.
>
> The only thing I can think here is that we've been running gerrit gc for a
> long time (jgit), so running git repack caused pack files to be constructed
> differently than the jgit pack files. This was fine for almost everything,
> except those 6 changes that received updates (new patchsets) while the refs
> were being repacked in a different format. Am I on the right track here? Or
> does anyone have other insights?

I doubt you have bad pack-files. More likely you have missing objects because
of pruning. If new objects were being written during repacking and were
unreferenced, with the wrong repacking options they may have gotten pruned,

[luke....@hpe.com]

What commands and options did you run? 

git repack -abdf, followed by git prune and git pack-refs --all

The errors then showed up when running git pack-refs --all. So you're probably right that pruning was triggered the errors, but you're suggesting our repack options were the true cause?

-Luke

[luke....@hpe.com]

Or, was it the fact that I didn't add options to git prune, and I should have added --expire="1 week ago"? That seems more reasonable.

-Luke

[Martin Fick]

On Friday, February 8, 2019 10:47:22 AM MST luke....@hpe.com wrote:
> > What commands and options did you run?
>
> git repack -abdf, followed by git prune and git pack-refs --all

I believe using -a instead of -A is risky on a running server (-A has it own
risk, loose object explosions). From my understanding (man pages are very
confusing), -a will not include unreferenced (your potentially new objects)
into packs, and it will not "loosen" them either. I believe this effectively
deletes and unreferenced objects that are in packs.

> The errors then showed up when running git pack-refs --all. So you're
> probably right that pruning was triggered the errors, but you're suggesting
> our repack options were the true cause?

Pruning could also be the problem, if your unreferenced objects were loose and
not packed. So I think it depends on whether the unreferenced were loose or
packed,

[luke....@hpe.com]

I hate to bring this back up, but we just had a critical failure today. We've been running git pack-refs --all and git repack -Abd every 3 hours on our repos for the last month or so, with no problems. We aren't exactly sure what happened, but during the repack at 3 PM today, a whole bunch of pack files got corrupted (at least jgit seemed to think so) which caused jgit to remove a lot of packs from the pack list - resulting in losing history for a ~2 hour period, but only for changes/meta it seems. This means we lost all votes/comments that happened within that time period. The only thing I can think that would cause this is the repack.

That said, what can we do to prevent this? Do we need to implement similar custom switches like you have above to preserve pack files for another round of gc? Is there something else we can/need to be doing? I'd also love to hear if you have any idea why this problem occurred in the first place.

Thank you,

Luke

[Luca Milanesio]

On 28 Feb 2019, at 03:02, luke....@hpe.com wrote:

I hate to bring this back up, but we just had a critical failure today. We've been running git pack-refs --all and git repack -Abd every 3 hours on our repos for the last month or so, with no problems. We aren't exactly sure what happened, but during the repack at 3 PM today, a whole bunch of pack files got corrupted (at least jgit seemed to think so) which caused jgit to remove a lot of packs from the pack list - resulting in losing history for a ~2 hour period, but only for changes/meta it seems. This means we lost all votes/comments that happened within that time period. The only thing I can think that would cause this is the repack.

That seems exactly the issue I've raised on JGit:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=544199

What basically happens is that JGit finds that the checksum of the packfile does not correspond to the idex and thus flags the packfile as permanently invalid and then gets removed.

This is all "smoke and mirror" because actually everything is fine.

We are using a HA and multi-site config though, the healthcheck plugin detects the situation and the HAProxy redirects immediately the traffic to the other nodes and the "corrupted cache one" is not active until the cache is back to normal.

That said, what can we do to prevent this? Do we need to implement similar custom switches like you have above to preserve pack files for another round of gc? Is there something else we can/need to be doing? I'd also love to hear if you have any idea why this problem occurred in the first place.

Nothing you can do I'm afraid at the GC side, this is a JGit caching issue IMHO.

It gets triggered somehow by the underlying GC, but the GC isn't the problem.

Luca.

Thank you,

Luke

On Monday, February 11, 2019 at 9:00:26 AM UTC-8, MartinFick wrote:

On Friday, February 8, 2019 10:47:22 AM MST luke....@hpe.com wrote:
> > What commands and options did you run?
>
> git repack -abdf, followed by git prune and git pack-refs --all

I believe using -a instead of -A is risky on a running server (-A has it own
risk, loose object explosions). From my understanding (man pages are very
confusing), -a will not include unreferenced (your potentially new objects)
into packs, and it will not "loosen" them either. I believe this effectively
deletes and unreferenced objects that are in packs.


> The errors then showed up when running git pack-refs --all. So you're
> probably right that pruning was triggered the errors, but you're suggesting
> our repack options were the true cause?

Pruning could also be the problem, if your unreferenced objects were loose and
not packed. So I think it depends on whether the unreferenced were loose or
packed,

-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

[luke....@hpe.com]

Do you have any suggestions for temporary workarounds? We don't have an HA/multimaster setup, and this problem is deleting history, with no way for us to get it back. would it help to create a wrapper around git repack that saves the packfiles somewhere so we can retrieve them if jgit decides to delete them? Or will that not help because as soon as the "old" packfiles are pulled in, jgit will mark them as invalid again?

-Luke

To unsubscribe, email repo-disc...@googlegroups.com

[Luca Milanesio]

On 28 Feb 2019, at 17:54, luke....@hpe.com wrote:

Do you have any suggestions for temporary workarounds? We don't have an HA/multimaster setup, and this problem is deleting history, with no way for us to get it back.

Not sure that is correct: the fact that the in-memory JGit packlist is invalid, it means that the repository is misbehaving, true. 

In my case at least, the situation on the filesystem is fine, there is no history deletion.

would it help to create a wrapper around git repack that saves the packfiles somewhere so we can retrieve them if jgit decides to delete them?

The JGit's gc.prunepackexpire already does that, but that it is not the problem. Even if you have plenty of valid packfiles on disk (we keep 1 day of packfile history, using gc.prunepackexpire = 1.day.ago) but JGit in-memory list of packfiles is wrong, then you still have problems.

Or will that not help because as soon as the "old" packfiles are pulled in, jgit will mark them as invalid again?

No, I believe there is a genuine bug in JGit on the packlist synchronisation: instead of being centralised in a well defined single place, it is all scattered around the code on all the possible "failure scenarios".

In a nutshell, instead of saying "oh, one packfile isn't good, let's refresh the list", what JGit does is to remove the packfile in the in-memory list and then wait for stuff to fail, which is bad.

Then in all around the code, there are conditions that in case of failure, reload the pack list. I am guessing (speculations for now, need to do further analysis) that in some of places, that does not happen.

One more thing: the trustfolderstats flag behaviour could have an impact on the story here.

If you set "core.trustfolderstat" to true, JGit won't list the files in the underlying directory unless the folder stats are telling to do so. However, imagine that the list of pack files hasn't changed and, still, JGit has removed a "assumed invalid" packfile.

Even if JGit is calling the refresh packlist, that won't be executed just because the underlying directory hasn't changed.

In fact, we do see that the in-memory packlist that is self-recovering at the following JGit GC phase, where the folder stats are udpated.

Will do more code analysis and experiments today and provide update.

@Martin I believe you don't have this problem just because your JGit version is quite different. We started noticing this issue starting from v2.15.

Luca.

To unsubscribe, email repo-discuss...@googlegroups.com

[Luca Milanesio]

On 1 Mar 2019, at 07:54, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 28 Feb 2019, at 17:54, luke....@hpe.com wrote:

Do you have any suggestions for temporary workarounds? We don't have an HA/multimaster setup, and this problem is deleting history, with no way for us to get it back.

Not sure that is correct: the fact that the in-memory JGit packlist is invalid, it means that the repository is misbehaving, true. 

In my case at least, the situation on the filesystem is fine, there is no history deletion.

would it help to create a wrapper around git repack that saves the packfiles somewhere so we can retrieve them if jgit decides to delete them?

The JGit's gc.prunepackexpire already does that, but that it is not the problem. Even if you have plenty of valid packfiles on disk (we keep 1 day of packfile history, using gc.prunepackexpire = 1.day.ago) but JGit in-memory list of packfiles is wrong, then you still have problems.

Or will that not help because as soon as the "old" packfiles are pulled in, jgit will mark them as invalid again?

No, I believe there is a genuine bug in JGit on the packlist synchronisation: instead of being centralised in a well defined single place, it is all scattered around the code on all the possible "failure scenarios".

In a nutshell, instead of saying "oh, one packfile isn't good, let's refresh the list", what JGit does is to remove the packfile in the in-memory list and then wait for stuff to fail, which is bad.

Then in all around the code, there are conditions that in case of failure, reload the pack list. I am guessing (speculations for now, need to do further analysis) that in some of places, that does not happen.

One more thing: the trustfolderstats flag behaviour could have an impact on the story here.

If you set "core.trustfolderstat" to true, JGit won't list the files in the underlying directory unless the folder stats are telling to do so. However, imagine that the list of pack files hasn't changed and, still, JGit has removed a "assumed invalid" packfile.

Even if JGit is calling the refresh packlist, that won't be executed just because the underlying directory hasn't changed.

In fact, we do see that the in-memory packlist that is self-recovering at the following JGit GC phase, where the folder stats are udpated.

I've just added in crontab the following:

* * * * * touch /var/gerrit/git/All-Users.git/objects/pack/.update

Every minute, I'm updating the folderstats of the repository packs directory: if my hypothesis is correct (pack list not fixed because of trustfolderstat set to true), the above workaround would make sure that the list is refreshed at least every minute.

Luca.

[Martin Fick]

On Wednesday, February 27, 2019 7:02:30 PM MST luke....@hpe.com wrote:
> I hate to bring this back up, but we just had a critical failure today.
> We've been running git pack-refs --all and git repack -Abd every 3 hours on
> our repos for the last month or so, with no problems. We aren't exactly
> sure what happened, but during the repack at 3 PM today, a whole bunch of
> pack files got corrupted (at least jgit seemed to think so)

Where they really corrupted? Does git think so?

> which caused jgit to remove a lot of packs from the pack list

Just to clarify, how do you know this, messages in the logs? Or do you mean it
actually deleted them?

> - resulting in losing
> history for a ~2 hour period, but only for changes/meta it seems. This
> means we lost all votes/comments that happened within that time period. The
> only thing I can think that would cause this is the repack.

It sounds like you are saying they were removed from the internal list of
cached jgit files. If that is the case, it would be good to know more about
your specific setup. What filesystem are you using, what version of Gerrit/
jgit, and what does the stacktrace look like from the logs when it removed a
packfile from the list. The last part is essential if jgit is to get a fix
(assuming it needs one) for such a situation,

[luke....@hpe.com]

Not sure that is correct: the fact that the in-memory JGit packlist is invalid, it means that the repository is misbehaving, true. 

In my case at least, the situation on the filesystem is fine, there is no history deletion.

In our case we saw lost history in the changes/XX/XXXXX/meta branches. Any comments or votes that were placed in a two hour period were completely gone. Fetching the changes/XX/XXXXX/meta branches showed that the comments/votes that were placed in that timeframe were completely gone, like they never existed in the first place.

 The JGit's gc.prunepackexpire already does that, but that it is not the problem. Even if you have plenty of valid packfiles on disk (we keep 1 day of packfile history, using gc.prunepackexpire = 1.day.ago) but JGit in-memory list of packfiles is wrong, then you still have problems.

So, we don't use JGit for gc, we use git directly for gc, however, we don't run git prune --expire (we don't run git prune at all, actually. I'm assuming the git prune is similar to jgit prunepack). It seems unlikely our lack of using git prune --expire would cause our specific problem of packfiles being lost. I assume at the very least, that a gerrit restart would cause jgit to load the packfiles into memory again? A gerrit restart did not bring back missing comments/votes.

To unsubscribe, email repo-disc...@googlegroups.com

[Luca Milanesio]

On 1 Mar 2019, at 15:37, Martin Fick <mf...@codeaurora.org> wrote:

On Wednesday, February 27, 2019 7:02:30 PM MST luke....@hpe.com wrote:

I hate to bring this back up, but we just had a critical failure today.
We've been running git pack-refs --all and git repack -Abd every 3 hours on
our repos for the last month or so, with no problems. We aren't exactly
sure what happened, but during the repack at 3 PM today, a whole bunch of
pack files got corrupted (at least jgit seemed to think so)

Where they really corrupted? Does git think so?

The files were never corrupted on disk: when HAProxy detects the problem (we use the healthcheck plugin that does JGit-level checking) the traffic is redirected to another node and everything is working fine again.

However, JGit *believes* the packfile is corrupted and removes is from the pack list.

The problem is in the ObjectDirectory.openPackedObject(), it seems that the search for an object in the pack list is broken when one of the packs is failing for PackMismatchException.

See the code below with my comments.

Will attempt a fix and give it to you, Jonathan and Matthias for review.

However, see below a high-level description:

ObjectLoader openPackedObject(WindowCursor curs, AnyObjectId objectId) {
   PackList pList;
   do {
      SEARCH: for (;;) {
         pList = packList.get();
         for (PackFile p : pList.packs) {
            try {
               ObjectLoader ldr = p.get(curs, objectId); <== (0) Throws PackMismatchException because the packfile/checksum <> index/checksum
               p.resetTransientErrorCount();
               if (ldr != null)
                  return ldr;
            } catch (PackMismatchException e) { <== (1) When the pack/checksum <> idx/checksum
               // Pack was modified; refresh the entire pack list.
               if (searchPacksAgain(pList)) <== (2) It would refresh the pack-list, but there are no changes on the FS, so returns false
                  continue SEARCH; <== (3) The code doesn't come here, so the search is interrupted
            } catch (IOException e) {
               handlePackError(e, p);
            }
         }
         break SEARCH; <== (4) Search is interrupted
      }
   } while (searchPacksAgain(pList)); <== (5) Same as above, this returns false
   return null;
}

The in-memory Pack index checksum fails at (0), this is because the index (in memory) checksum does not correspond to the one on disk.

It seems that the code has always been like that in the past 10 years. I added more logs and it seems that happens quite often.

In theory this is managed by (1) and it should be recovered by searching the whole pack list again at (2).

However, that does nothing because this is not generated by a change on the filesystem but rather an in-memory mismatch. So instead of doing his job, returns false and doesn't reload anything.

The search instead of going through the rest of the packfiles, which would have solved the problem because JGit GC can preserve old packfiles (prunepackexpire = 1.day.ago), it is interrupted and the method returns null.

The thing is, as soon as a brand-new JGit GC starts, and the directory lastModifiedDate changes and the list of files is now different, the system self-recover and the whole pack list is reloaded again.

One fix that I am willing to upload will force to reload the whole list, unconditionally: what do you think?

which caused jgit to remove a lot of packs from the pack list

Just to clarify, how do you know this, messages in the logs? Or do you mean it
actually deleted them?

I added a lot message and saw that in the logs.

Then I cross-checked the file names with the file-system and they are still on disk.

"git verify-pack" says that it is all fine for each one of them.

- resulting in losing
history for a ~2 hour period, but only for changes/meta it seems. This
means we lost all votes/comments that happened within that time period. The
only thing I can think that would cause this is the repack.

It sounds like you are saying they were removed from the internal list of
cached jgit files. If that is the case, it would be good to know more about
your specific setup.

My one is very simple:

- JGit 5.1.5 / Gerrit v2.16.5

- Local FS (spinning disk) - ext4

What filesystem are you using, what version of Gerrit/
jgit, and what does the stacktrace look like from the logs when it removed a
packfile from the list. The last part is essential if jgit is to get a fix
(assuming it needs one) for such a situation,

org.eclipse.jgit.errors.PackMismatchException: Pack object count mismatch: pack e837a44b6d624b5a97cf7d91ad98dab21595299a index b48bcd6c3a9c677cbd28f9431a8c646d37518be8: /var/gerrit/git/All-Users.git/objects/pack/pack-f3b84ebd5c78f687d634b68b96dc10250d835f8d.pack

        at org.eclipse.jgit.internal.storage.file.PackFile.onOpenPack(PackFile.java:763)

        at org.eclipse.jgit.internal.storage.file.PackFile.doOpen(PackFile.java:651)

        at org.eclipse.jgit.internal.storage.file.PackFile.beginWindowCache(PackFile.java:631)

        at org.eclipse.jgit.internal.storage.file.WindowCache.load(WindowCache.java:295)

        at org.eclipse.jgit.internal.storage.file.WindowCache.getOrLoad(WindowCache.java:379)

        at org.eclipse.jgit.internal.storage.file.WindowCache.get(WindowCache.java:184)

        at org.eclipse.jgit.internal.storage.file.WindowCursor.pin(WindowCursor.java:360)

        at org.eclipse.jgit.internal.storage.file.WindowCursor.copy(WindowCursor.java:259)

        at org.eclipse.jgit.internal.storage.file.PackFile.readFully(PackFile.java:607)

        at org.eclipse.jgit.internal.storage.file.PackFile.load(PackFile.java:781)

        at org.eclipse.jgit.internal.storage.file.PackFile.get(PackFile.java:291)

        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:491)

        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:449)

        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:440)

        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:165)

        at org.eclipse.jgit.lib.ObjectDatabase.open(ObjectDatabase.java:176)

        at org.eclipse.jgit.lib.ObjectDatabase.open(ObjectDatabase.java:146)

        at org.eclipse.jgit.lib.Repository.open(Repository.java:349)

        at com.googlesource.gerrit.plugins.healthcheck.check.JGitHealthCheck.doCheck(JGitHealthCheck.java:48)

        at com.googlesource.gerrit.plugins.healthcheck.check.AbstractHealthCheck.lambda$run$0(AbstractHealthCheck.java:54)

        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)

        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)

        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

[2019-03-01 11:25:40,926] [pool-28-thread-3] WARN  org.eclipse.jgit.internal.storage.file.ObjectDirectory : Packfile /var/gerrit/git/All-Users.git/objects/pack/pack-f3b84ebd5c78f687d634b68b96dc10250d835f8d.pack is corrupted

org.eclipse.jgit.errors.PackInvalidException: Pack file invalid: /var/gerrit/git/All-Users.git/objects/pack/pack-f3b84ebd5c78f687d634b68b96dc10250d835f8d.pack

        at org.eclipse.jgit.internal.storage.file.PackFile.doOpen(PackFile.java:647)

        at org.eclipse.jgit.internal.storage.file.PackFile.beginWindowCache(PackFile.java:631)

        at org.eclipse.jgit.internal.storage.file.WindowCache.load(WindowCache.java:295)

        at org.eclipse.jgit.internal.storage.file.WindowCache.getOrLoad(WindowCache.java:379)

        at org.eclipse.jgit.internal.storage.file.WindowCache.get(WindowCache.java:184)

        at org.eclipse.jgit.internal.storage.file.WindowCursor.pin(WindowCursor.java:360)

        at org.eclipse.jgit.internal.storage.file.WindowCursor.copy(WindowCursor.java:259)

        at org.eclipse.jgit.internal.storage.file.PackFile.readFully(PackFile.java:607)

        at org.eclipse.jgit.internal.storage.file.PackFile.load(PackFile.java:781)

        at org.eclipse.jgit.internal.storage.file.PackFile.get(PackFile.java:291)

        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:491)

        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:449)

        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:440)

        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:165)

        at org.eclipse.jgit.lib.ObjectDatabase.open(ObjectDatabase.java:176)

        at org.eclipse.jgit.lib.ObjectDatabase.open(ObjectDatabase.java:146)

        at org.eclipse.jgit.lib.Repository.open(Repository.java:349)

        at com.googlesource.gerrit.plugins.healthcheck.check.JGitHealthCheck.doCheck(JGitHealthCheck.java:48)

        at com.googlesource.gerrit.plugins.healthcheck.check.AbstractHealthCheck.lambda$run$0(AbstractHealthCheck.java:54)

        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)

        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)

        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

[2019-03-01 11:25:40,926] [pool-28-thread-3] WARN  org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /var/gerrit/git/All-Users.git/objects/pack/pack-f3b84ebd5c78f687d634b68b96dc10250d835f8d.pack is corrupt, removing it from pack list

Luca

-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

[luke....@hpe.com]

Where they really corrupted? Does git think so?

I'm not sure if git thinks so or not, but the history is completely gone. This must have happened during some phase of our git repack which seems to have caused git to stop including packfiles as soon as the bug hit. I'm not sure if this means jgit deleted not only the in memory pack files, or the pack files in the filesystem as well, but it seems larger than just jgit cache was affected.

Just to clarify, how do you know this, messages in the logs? Or do you mean it 
actually deleted them?

Messages in the logs. And maybe also that they were actually deleted. Looking at the logs now it appears some .idx files were deleted somehow? See the stack trace below from the logs.

What filesystem are you using, what version of Gerrit/ 

jgit, and what does the stacktrace look like from the logs when it removed a 
packfile from the list.

Filesystem is ext4

Gerrit v2.16.4 - So I assume JGit version is 5.1.5.201812261915-r (according to the release notes)

There's a couple different errors/stack traces in the logs, and from multiple sources (many come from replication, but there are others from HTTP). Here's the common errors:

[2019-02-27 15:12:35,143] [ReplicateTo-XXX-3] WARN  org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /home/gerrit/review_site/git/XXX.git/objects/pack/pack-f5736cf43e69c023fde03607d38dda83ad5c66f1.pack is corrupt, removing it from pack list

[2019-02-27 15:12:35,194] [ReplicateTo-XXX-2] ERROR org.eclipse.jgit.internal.storage.file.ObjectDirectory : ERROR: Exception caught while accessing pack file /home/gerrit/review_site/git/XXX.git/objects/pack/pack-7ec6213fc057e36599f41c739d30fc16e9934cd2.pack, the pack file might be corrupt, {1}. Caught {2} consecutive errors while trying to read this pack.
java.io.IOException: Unreadable pack index: /home/gerrit/review_site/git/XXX.git/objects/pack/pack-7ec6213fc057e36599f41c739d30fc16e9934cd2.idx
     at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:102)
     at org.eclipse.jgit.internal.storage.file.PackFile.idx(PackFile.java:183)
     at org.eclipse.jgit.internal.storage.file.PackFile.get(PackFile.java:284)
     at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:486)
     at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:444)
     at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:435)
     at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:165)
     at org.eclipse.jgit.revwalk.RevWalk.getCachedBytes(RevWalk.java:933)
     at org.eclipse.jgit.revwalk.RevCommit.parseHeaders(RevCommit.java:159)
     at org.eclipse.jgit.revwalk.PendingGenerator.next(PendingGenerator.java:147)
     at org.eclipse.jgit.revwalk.RevWalk.next(RevWalk.java:444)
     at org.eclipse.jgit.revwalk.ObjectWalk.next(ObjectWalk.java:307)
     at org.eclipse.jgit.revwalk.BitmapWalker.findObjectsWalk(BitmapWalker.java:213)
     at org.eclipse.jgit.revwalk.BitmapWalker.findObjects(BitmapWalker.java:137)
     at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPackUsingBitmaps(PackWriter.java:2000)
     at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPack(PackWriter.java:1795)
     at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:914)
     at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:864)
     at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:786)
     at org.eclipse.jgit.transport.BasePackPushConnection.writePack(BasePackPushConnection.java:356)
     at org.eclipse.jgit.transport.BasePackPushConnection.doPush(BasePackPushConnection.java:219)
     at org.eclipse.jgit.transport.BasePackPushConnection.push(BasePackPushConnection.java:170)
     at org.eclipse.jgit.transport.PushProcess.execute(PushProcess.java:172)
     at org.eclipse.jgit.transport.Transport.push(Transport.java:1346)
     at org.eclipse.jgit.transport.Transport.push(Transport.java:1392)
     at com.googlesource.gerrit.plugins.replication.PushOne.pushVia(PushOne.java:452)
     at com.googlesource.gerrit.plugins.replication.PushOne.runImpl(PushOne.java:431)
     at com.googlesource.gerrit.plugins.replication.PushOne.runPushOperation(PushOne.java:316)
     at com.googlesource.gerrit.plugins.replication.PushOne.access$000(PushOne.java:82)
     at com.googlesource.gerrit.plugins.replication.PushOne$1.call(PushOne.java:281)
     at com.googlesource.gerrit.plugins.replication.PushOne$1.call(PushOne.java:278)
     at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:212)
     at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:191)
     at com.google.gerrit.server.git.PerThreadRequestScope$Propagator.lambda$scope$0(PerThreadRequestScope.java:73)
     at com.googlesource.gerrit.plugins.replication.PushOne.run(PushOne.java:285)
     at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:72)

     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

     at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:646)

     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
     at java.lang.Thread.run(Thread.java:748)

Caused by: java.io.FileNotFoundException: /home/gerrit/review_site/git/XXX.git/objects/pack/pack-7ec6213fc057e36599f41c739d30fc16e9934cd2.idx (No such file or directory)
     at java.io.FileInputStream.open0(Native Method)
     at java.io.FileInputStream.open(FileInputStream.java:195)
     at java.io.FileInputStream.<init>(FileInputStream.java:138)
     at org.eclipse.jgit.util.io.SilentFileInputStream.<init>(SilentFileInputStream.java:64)
     at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:98)
     ... 43 more

[Martin Fick]

On Friday, March 1, 2019 8:31:24 AM MST luke....@hpe.com wrote:
> > Where they really corrupted? Does git think so?

The answer to the above question would help direct the investigation based on
your specific situation.

> I'm not sure if git thinks so or not, but the history is completely gone.
> This must have happened during some phase of our git repack which seems to
> have caused git to stop including packfiles as soon as the bug hit. I'm not
> sure if this means jgit deleted not only the in memory pack files, or the
> pack files in the filesystem as well, but it seems larger than just jgit
> cache was affected.
>
> Just to clarify, how do you know this, messages in the logs? Or do you mean
> it actually deleted them?
>
> Messages in the logs. And maybe also that they were actually deleted.

I believe The only Gerrit operations that could delete files (even files it
thinks are corrupted), would be the jgit gc operations. If you are not running
jgit (Gerrit) gc, then any deletions can likely only have come from git
repacking.

> [2019-02-27 15:12:35,143] [ReplicateTo-XXX-3] WARN
> org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file
> /home/gerrit/review_site/git/XXX.git/objects/pack/pack-f5736cf43e69c023fde03
> 607d38dda83ad5c66f1.pack is corrupt, removing it from pack list

> [2019-02-27 15:12:35,194] [ReplicateTo-XXX-2] ERROR
> org.eclipse.jgit.internal.storage.file.ObjectDirectory : ERROR: Exception
> caught while accessing pack file
> /home/gerrit/review_site/git/XXX.git/objects/pack/pack-7ec6213fc057e36599f41
> c739d30fc16e9934cd2.pack, the pack file might be corrupt, {1}. Caught {2}
> consecutive errors while trying to read this pack.
> java.io.IOException: Unreadable pack index:
> /home/gerrit/review_site/git/XXX.git/objects/pack/pack-7ec6213fc057e36599f41
> c739d30fc16e9934cd2.idx at

I would use git to see if those files really are corrupt,

[luke....@hpe.com]

I would use git to see if those files really are corrupt

I would check, but they don't exist anymore. An additional repack must have gotten rid of them :(

-Luke

[luke....@hpe.com]

Update on this.. I checked logs from previous days - we've been getting those errors every single repack run. Every 3 hours we've been getting those errors, but we only lost history the one time. We stopped repacking as soon as we lost history, and just recently ran a gerrit gc with no errors.

At this point we're going to have to stop using git gc commands, which is unfortunate because we were liking the extra control.

-Luke

[Luca Milanesio]

On 1 Mar 2019, at 21:03, luke....@hpe.com wrote:

Update on this.. I checked logs from previous days - we've been getting those errors every single repack run. Every 3 hours we've been getting those errors, but we only lost history the one time. We stopped repacking as soon as we lost history, and just recently ran a gerrit gc with no errors.

Careful on the JGit GC settings though, especially the gc.prunepackexpire. By default is set to 1h (1.hour.ago) that means that if you a 'gerrit gc' less frequently than once an hour, the GC operation will get rid immediately of the old packfiles, creating potential "hiccups" to the in-memory pack list.

We are using JGit GC (see my thread) with gc.prunepackexpire set to 1.day.ago, and we are hitting a different problem also.

There are still things to be fixed then, even if you just rely on 'gerrit gc'.

Luca.

[luke....@hpe.com]

Can I change the 'gerrit gc' options by modifying the [gc] section of the gerrit.config? The documentation suggests you can only modify aggressive, start time, and interval. Can I also change prunepackexpire, and threads in the [gc] section?

-Luke

To unsubscribe, email repo-disc...@googlegroups.com

[Luca Milanesio]

On 1 Mar 2019, at 21:33, luke....@hpe.com wrote:

Can I change the 'gerrit gc' options by modifying the [gc] section of the gerrit.config? The documentation suggests you can only modify aggressive, start time, and interval. Can I also change prunepackexpire, and threads in the [gc] section?

Those are JGit settings, that must be set either in the 'config' file in the bare repo or, for global settings, in the ~/.gitconfig of the user running Gerrit.

If you defined them in the [gc] section of the gerrit.config, they will be ignored.

HTH

Luca.

To unsubscribe, email repo-discuss...@googlegroups.com

[Luca Milanesio]

@Martin, bingo !

I added more logs and I found that the problem *it is actually* a packfile name collision after a JGit GC.

Example, I saw it from the logs

[2019-03-02 00:21:33,865] [pool-28-thread-6] INFO  org.eclipse.jgit.internal.storage.file.PackFile : PackFile /var/gerrit/git/All-Users.git/objects/pack/pack-bbde47cbe659e7b6303fc7d2a9a0981bb1e46ffe.pack checksum from Index e69767ca0430f6a99c89d6f80fafd13609d71b72
The Packfile and the Index are loaded and aligned at 00:21:33

But if I look at the date / time on the filesystem now, it says 01:17 (exactly 2 mins after the last JGit GC started)

And see what happens just afterwards:

[2019-03-02 01:26:54,099] [pool-28-thread-2] ERROR org.eclipse.jgit.internal.storage.file.PackFile : Permanent problems opening packfile /var/gerrit/git/All-Users.git/objects/pack/pack-bbde47cbe659e7b6303fc7d2a9a0981bb1e46ffe.pack
org.eclipse.jgit.errors.PackMismatchException: Pack checksum mismatch detected for pack file /var/gerrit/git/All-Users.git/objects/pack/pack-bbde47cbe659e7b6303fc7d2a9a0981bb1e46ffe.pack: .pack has 35861936c9b2a41392b57ad65102772de5f71fbf whilst .idx has e69767ca0430f6a99c89d6f80fafd13609d71b72

The JGit bug is *exactly* where I thought and happens because JGit GC actually replaces a packfile, overwrite an old one because of a name collision.

The above code (call to searchPacksAgain(pList)) won't reload the in-memory list because it see that the list of names hasn't changed and you end up with Gerrit with an invalid packlist, because the most precious packfile you need (the one just generated by JGit GC) as been removed from the in-memory list.

Can you please double-check from your JGit understanding?

P.S. When the server gets into this situation, the immediate workaround is just to create a "fake" packfile in the directory, so that JGit believes the list of names is different and reloads the list of packfiles.

HTH

Luca.

[lucamilanesio]

I have added more insights to the issue:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=544199

@Luke: you're not safe with JGit either until this bug is resolved. The only workaround I've found is to add an extra "dummy fake" packfile (0 bytes) with a randeom name after the GC operation, so that you are that the list of packfiles is always different and you do not fall into this hole.

Reproducing the problem is straightforward:

a) Create a repo

b) Push one commit

c) Perform a JGit GC (no parameters)

d) Perform a JGit GC (with --aggressive)

You will see that between c) and d) the list of packfiles is the same, but they have different checksums. What Gerrit does is to consider the (only) packfile as corrupted, but removes it from the in-memory packlist and then any operation on it are returning a "Missing unknown" until you restart Gerrit.

This bug *needs urgent* fixing, as I am hearing a few people having issues, apart from us :-)

Luca.

 

HTH

Luca.

To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[thomasmu...@yahoo.com]

Does this bug only affect 2.16? Or 2.15 too?

[Luca Milanesio]

On 2 Mar 2019, at 22:30, thomasmulhall410 via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

Does this bug only affect 2.16? Or 2.15 too?

I believe it has been there for a while, need to track down that piece of code in JGit when it was introduced.

It impacts a lot more Gerrit with NoteDb, just because you would need to GC some repos (e.g. All-Users) a lot more often.

I am testing a fix on GerritHub.io as we speak.

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[thomasmu...@yahoo.com]

Thank you! I've opened a task and submitted a patch at the wmf to have this disabled (gc using jgit).

[lucamilanesio]

I was wrong: the code *is supposed* to work in that condition as well, Shawn designed that properly and had a specific comment on this condition :-)

However, there were some later changes that have introduced un-protected race conditions on the concurrent update of the packlist in memory, and *that* broke the mechanism.

I have now a specific unit-test that demonstrates the problem and I am going to:

a) Apply the test to different versions of JGit

b) Minimise the fix to the *sole* race condition, without modifying the overall logic of reload of the pack list

@Martin: because you are running a fork of a much older version of JGit, you possibly do not have this problem in production.

@Luke: if you are running a more recent version of JGit / Gerrit, you have this problem, as I do on GerritHub.io. *DO NOT* make concurrent GC yet whilst production traffic is active on Gerrit until I have submitted my fix.

Luca.

 

To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[David Pursehouse]

Please upload the unit test for review, even if the fix isn't ready yet.

 

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[lucamilanesio]

Done:

 https://git.eclipse.org/r/138119

 

 

To unsubscribe, email repo-disc...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[lucamilanesio]

And the fix on stable-4.9 at:

https://git.eclipse.org/r/c/138120/

P.S. The fix is super-duper-simple and can be applied to an earlier version of JGit also, where this problem occurs. I can do that tomorrow, now it is a bit late for the UK :-) 

[lucamilanesio]

We have found a second problem that is associated with this bug, and that is on stable-4.5.

That means, *any* Gerrit >= v2.13 are impacted and may have packlist corruption if they do GC (JGit gc, Gerrit gc or native-gc) whilst there is *read* or *write* traffic to the repo.

Even though we do not officially support v2.13 anymore (our policy is the latest 2 releases, v2.15 and v2.16), I believe the community deserves a fix anyway.

Luca.

[lucamilanesio]

This is the second fix:

https://git.eclipse.org/r/c/138163/

[luke....@hpe.com]

When can we expect a fix/gerrit release? Delaying gc for even a day has large performance impacts for us. We attempted running gc offline, but that wasn't viable since the "enumerating objects" phase of the repack was taking forever (almost 2 hours, by my calculations) because we have so many changes come in within a 24 hour period. Shutting the server down for >10 minutes is not something we can do since we're a global company and there's no time where everyone is offline.

-Luke

To unsubscribe, email repo-disc...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[luca.mi...@gmail.com]

Sent from my iPhone

On 6 Mar 2019, at 17:21, luke....@hpe.com wrote:

When can we expect a fix/gerrit release?

It may take some time, because needs to be reviewed and merged on JGIt first, then releases and finally included in Gerrit.

You can alternatively make a local build on your dev environment, test and deploy it.

Delaying gc for even a day has large performance impacts for us. We attempted running gc offline, but that wasn't viable since the "enumerating objects" phase of the repack was taking forever (almost 2 hours, by my calculations) because we have so many changes come in within a 24 hour period. Shutting the server down for >10 minutes is not something we can do since we're a global company and there's no time where everyone is offline.

How did you guys managed until now?

If your adoption is so widespread, that means that somehow it was working before, or am I mistaken?

Luca

To unsubscribe, email repo-discuss...@googlegroups.com

[luke....@hpe.com]

You can alternatively make a local build on your dev environment, test and deploy it.

Is there documentation/instructions on how to do this? Specifically how to incorporate the local jgit build into gerrit?

 

How did you guys managed until now?

If your adoption is so widespread, that means that somehow it was working before, or am I mistaken?

So, we have been getting these error messages in the logs since 02/09/2019. Two important things happened on that date: 

1. Gerrit was upgraded from 2.15.7 to 2.16.4.

2. We switched from running 'gerrit gc' once daily to running git plumbing commands directly - every 6 hours (then later switched to every 3 hours). We did attempt the git plumbing commands on 2/6 (which is why this thread started in the first place), and we didn't see any errors in the logs for that day. That tells me the gerrit upgrade (not using git instead of jgit) is the reason the error messages are now showing up.

From what I can see, it happens at the end of the repack. When the repack consolidates smaller packfiles into larger ones, jgit seems to throw errors specifically for the packfiles that were consolidated, and then more errors are thrown for unreadable .idx files corresponding to those packfiles, which sounds "fine" to me, since those files were indeed removed by gc. Once gc is finished, everything is fine (most of the time).

There was the one instance where running a repack seemingly deleted history in the refs/changes/meta branches, and since then we've switched from running direct git commands every 3 hours to running gerrit gc once per day. We have not seen any issues this week aside from the errors in the logs. That's where our problem lies here. We absolutely *have* to repack, but we're seemingly rolling the dice when we do it live, and we can't afford to take down the servers because the repack takes ~1-2 hours. Our trade-off is - we run it during low traffic times.

We will stop running gc for now and monitor performance over the next couple days. If we notice significant slow downs we're stuck at that point to either: roll the dice and run gc, or try pointing everyone to the cache servers as best we can to minimize direct fetches from Gerrit (not viable for jenkins builds though, unfortunately).

-Luke

[Luca Milanesio]

On 6 Mar 2019, at 19:27, luke....@hpe.com wrote:

You can alternatively make a local build on your dev environment, test and deploy it.

Is there documentation/instructions on how to do this? Specifically how to incorporate the local jgit build into gerrit?

For building Gerrit with bazel:

https://gerrit-review.googlesource.com/Documentation/dev-bazel.html

For incorporating a local JGit source tree, just edit the lig/jgit/jgit.gzl and it is self-explanatory (look at the LOCAL_JGIT_REPO variable)

 

How did you guys managed until now?

If your adoption is so widespread, that means that somehow it was working before, or am I mistaken?

So, we have been getting these error messages in the logs since 02/09/2019. Two important things happened on that date: 

1. Gerrit was upgraded from 2.15.7 to 2.16.4.

The JGit bug was introduced back in 2.13.x, so you had it already in 2.15.7.

2. We switched from running 'gerrit gc' once daily to running git plumbing commands directly - every 6 hours (then later switched to every 3 hours).

The 'gerrit gc' has full support for bitmaps generation, whilst I'm not sure with the 'git plumbing', it depends on the version.

We did attempt the git plumbing commands on 2/6 (which is why this thread started in the first place), and we didn't see any errors in the logs for that day. That tells me the gerrit upgrade (not using git instead of jgit) is the reason the error messages are now showing up.

In my experience, the upgrade to v2.16 (compared to v2.15) has shown a huge improvement in the performance, not a regression.

Did you perform a ReviewDb to NoteDb migration contextual to the upgrade? Or were you on v2.15 / NoteDb already?

From what I can see, it happens at the end of the repack. When the repack consolidates smaller packfiles into larger ones, jgit seems to throw errors specifically for the packfiles that were consolidated, and then more errors are thrown for unreadable .idx files corresponding to those packfiles, which sounds "fine" to me, since those files were indeed removed by gc. Once gc is finished, everything is fine (most of the time).

You need to preserve existing packfiles during GC (gc.prunepackexpire settings), otherwise you generate "hiccups" to the JGit cache.

If you do during the night, without significant traffic, not an issue at all.

Otherwise, you may experience issues, and loose data also if you are particularly unlucky.

There was the one instance where running a repack seemingly deleted history in the refs/changes/meta branches, and since then we've switched from running direct git commands every 3 hours to running gerrit gc once per day. We have not seen any issues this week aside from the errors in the logs. That's where our problem lies here. We absolutely *have* to repack, but we're seemingly rolling the dice when we do it live, and we can't afford to take down the servers because the repack takes ~1-2 hours. Our trade-off is - we run it during low traffic times.

Now that the problem has been fully identified, workarounds are possible also.

In a nutshell, the in-memory packlist becomes "incomplete" and the flagged with the last update timestamp on the directory.

When that happens, just create a fake packfile on objects/packs (it will get ignored anyway, but will retrigger the scan) and the inconsistency goes away.

You may sporadically have builds failing during the hiccup though.

We will stop running gc for now and monitor performance over the next couple days. If we notice significant slow downs we're stuck at that point to either: roll the dice and run gc, or try pointing everyone to the cache servers as best we can to minimize direct fetches from Gerrit (not viable for jenkins builds though, unfortunately).

Yes, work on the data you have: extract metrics and see when and how the GC can help you with.

In my experience, a "well tuned GC policy" makes a Gerrit server fast and effective. At times, too frequent GCs won't help or could even be the cause of the slowdowns, because of the CPU and Heap overhead generated.

HTH

Luca.

[luke....@hpe.com]

The 'gerrit gc' has full support for bitmaps generation, whilst I'm not sure with the 'git plumbing', it depends on the version.

We're running git 2.20.1, and it does have support for bitmap generation, which we made sure to include in our repack options.

In my experience, the upgrade to v2.16 (compared to v2.15) has shown a huge improvement in the performance, not a regression.

Did you perform a ReviewDb to NoteDb migration contextual to the upgrade? Or were you on v2.15 / NoteDb already?

A huge improvement in the performance of what, exactly? Because I will say that since upgrading we have seen much faster fetches when the repo gets "messy" (large number of loose objects/packfiles). However we never received these "pack file is corrupt, removing it from pack list" errors until the 2.16 upgrade. I know you said the bug was has been there since 2.15, but I began seeing those errors immediately after the upgrade, and never before it.

And yes, we were on NoteDb/2.15.7 prior to the 2.16 upgrade. I believe we did the NoteDb migration/2.15 upgrade in August of 2018.

You need to preserve existing packfiles during GC (gc.prunepackexpire settings), otherwise you generate "hiccups" to the JGit cache.

Looking up documentation on gitconfig files, I see no references to gc.prunepackexpire, so I assume this is a JGit setting. I went ahead and added the following to our gerrit user's .gitconfig anyways:

[gc]

           prunepackexpire = 1.day.ago

As you suggested, and we are still seeing errors. I know JGit is respecting the options in this file, because it's using the threads and memory that we have told it to, but it doesn't seem like JGit is respecting that specific variable, because it doesn't show up in the gc logs as a setting that JGit is using for gc. Am I doing something wrong here?

I also would like to say THANK YOU so much for the help, to everyone here. Many of you have helped me gain very valuable knowledge both for me and for my organization, and (in this case specifically) helped me have some level of understanding in an area with a whole lot of question marks. I *sincerely* appreciate the time you have taken to help solve this issue, and to help guide me in the right direction. THANK YOU!!

-Luke

[David Pursehouse]

On Thu, Mar 7, 2019 at 9:29 AM <luke....@hpe.com> wrote:

The 'gerrit gc' has full support for bitmaps generation, whilst I'm not sure with the 'git plumbing', it depends on the version.

We're running git 2.20.1, and it does have support for bitmap generation, which we made sure to include in our repack options.

In my experience, the upgrade to v2.16 (compared to v2.15) has shown a huge improvement in the performance, not a regression.

Did you perform a ReviewDb to NoteDb migration contextual to the upgrade? Or were you on v2.15 / NoteDb already?

A huge improvement in the performance of what, exactly? Because I will say that since upgrading we have seen much faster fetches when the repo gets "messy" (large number of loose objects/packfiles). However we never received these "pack file is corrupt, removing it from pack list" errors until the 2.16 upgrade. I know you said the bug was has been there since 2.15, but I began seeing those errors immediately after the upgrade, and never before it.

And yes, we were on NoteDb/2.15.7 prior to the 2.16 upgrade. I believe we did the NoteDb migration/2.15 upgrade in August of 2018.

You need to preserve existing packfiles during GC (gc.prunepackexpire settings), otherwise you generate "hiccups" to the JGit cache.

Looking up documentation on gitconfig files, I see no references to gc.prunepackexpire, so I assume this is a JGit setting. I went ahead and added the following to our gerrit user's .gitconfig anyways:

[gc]

           prunepackexpire = 1.day.ago

The gc.prunepackexpire is a jgit setting [1] and it gets read by the GC implementation here [2].  Looking at that code I'm not sure if it needs to be set per repo, or if it'll fall back to the user or global config.  The default value is "1.hour.ago".

[1] https://github.com/eclipse/jgit/blob/master/org.eclipse.jgit/src/org/eclipse/jgit/lib/ConfigConstants.java#L327

[2] https://github.com/eclipse/jgit/blob/master/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/GC.java#L707

[3] https://github.com/eclipse/jgit/blob/master/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/GC.java#L139 

 

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Luca Milanesio]

On 7 Mar 2019, at 00:29, luke....@hpe.com wrote:

The 'gerrit gc' has full support for bitmaps generation, whilst I'm not sure with the 'git plumbing', it depends on the version.

We're running git 2.20.1, and it does have support for bitmap generation, which we made sure to include in our repack options.

In my experience, the upgrade to v2.16 (compared to v2.15) has shown a huge improvement in the performance, not a regression.

Did you perform a ReviewDb to NoteDb migration contextual to the upgrade? Or were you on v2.15 / NoteDb already?

A huge improvement in the performance of what, exactly?

Starting from v2.16 you can back the project lists by Lucene, which has two advantages:

- It's *a lot* faster in searches than scanning the full list of projects

- Occupies *a lot less* JVM heap than loading the project state from JGit and parsing it in memory

We do collect metrics on a regular basis from our production systems, and we did see a *HUGE* improvement in terms of less heap utilisation, less CPU overhead, less frequent GCs and reduction in the overall latency.

If things are different for your system, I would recommend a complete system health check by some Gerrit experts.

I know GerritForge offers Enterprise Support and health checks, in addition to consultancy to setup HA configurations, multi-master and multi-site.

Because I will say that since upgrading we have seen much faster fetches when the repo gets "messy" (large number of loose objects/packfiles). However we never received these "pack file is corrupt, removing it from pack list" errors until the 2.16 upgrade.

I did write a specific unit-test to reproduce the problem *all the times* and I can confirm that happens in all JGit releases from stable-4.5 onwards, that means, Gerrit v2.13.x or later.

I know you said the bug was has been there since 2.15, but I began seeing those errors immediately after the upgrade, and never before it.

And yes, we were on NoteDb/2.15.7 prior to the 2.16 upgrade. I believe we did the NoteDb migration/2.15 upgrade in August of 2018.

By crunching all the logs from before 2.15, during 2.15 and after the upgrade to 2.16, you would find the metrics that will tell you what changed.

There is a *huge* value in all Gerrit logs and often that is not leveraged enough to understand what's happening under the hood.

You need to preserve existing packfiles during GC (gc.prunepackexpire settings), otherwise you generate "hiccups" to the JGit cache.

Looking up documentation on gitconfig files, I see no references to gc.prunepackexpire, so I assume this is a JGit setting. I went ahead and added the following to our gerrit user's .gitconfig anyways:

[gc]

           prunepackexpire = 1.day.ago

As you suggested, and we are still seeing errors.

Yes, you need the fix that is under review anyway.

There were two genuine bugs introduced over the years that contributed to the issue to arise.

I know JGit is respecting the options in this file, because it's using the threads and memory that we have told it to, but it doesn't seem like JGit is respecting that specific variable, because it doesn't show up in the gc logs as a setting that JGit is using for gc. Am I doing something wrong here?

Have you restarted Gerrit? (if you are performing Gerrit's GC)

I also would like to say THANK YOU so much for the help, to everyone here. Many of you have helped me gain very valuable knowledge both for me and for my organization, and (in this case specifically) helped me have some level of understanding in an area with a whole lot of question marks. I *sincerely* appreciate the time you have taken to help solve this issue, and to help guide me in the right direction. THANK YOU!!

No problem, and if you need a little more help, ask the Gerrit experts.

HTH

Luca.

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Luke Engle]

Have you restarted Gerrit? (if you are performing Gerrit's GC)

Setting 'prunepackexpire = 1.day.ago' in the repo config file worked, but setting it at the user gitconfig file did not. However, setting threads/windowMemory at the user level worked as expected.

-Luke 

       <span

[luke....@hpe.com]

Turns out we still get the below errors with prunepackexpire = 1.day.ago (running gerrit gc twice daily):

ERROR org.eclipse.jgit.internal.storage.file.ObjectDirectory : ERROR: Exception caught while accessing pack        file /home/gerrit/review_site/git/<repo>.git/objects/pack/pack-d0aa96eefcf553452ce591d0021c24e1a7353869.pack, the pack file might be corrupt, {1}.            Caught {2} consecutive errors while trying to read this pack.

14987 java.io.IOException: Unreadable pack index: /home/gerrit/review_site/git/halon.git/objects/pack/pack-d0aa96eefcf553452ce591d0021c24e1a7353869.idx

It appears certain tasks hit this error when they're doing a tree walk while GC is running, and it deletes > 1 day old pack files while the task is trying to read the pack file. This normally happens with Index-Batch and replication tasks. This doesn't exactly make sense to me, though, because I would think that a repack would move all pack files to 1 larger pack file, while keeping the old ones, which would mean no tasks would ever need to access the "old" pack files because they are no longer referenced since the creation of the single pack file? Unless the tasks need to iterate over *every* pack file regardless.

It doesn't matter how long we wait to delete pack files, it always seems to have this "problem". Are those errors not something we should worry about? Or is there a way we can do one of the following:

• Make tasks restart the tree walk if the idx/pack files change (get "corrupted") underneath them
• Move old pack files to a different directory, where tasks will look if the idx/pack files change (get "corrupted") underneath them
• These old pack files could be removed prior to the next gc run

-Luke

[luca.mi...@gmail.com]

Sent from my iPhone

On 13 Mar 2019, at 17:20, luke....@hpe.com wrote:

Turns out we still get the below errors with prunepackexpire = 1.day.ago (running gerrit gc twice daily):

The fix hasn’t been merged in Gerrit yet, have you only done the prunepackexpire change? Or have you deployed a patched JGIt in your Gerrit.war?

Luca

--

[luke....@hpe.com]

The fix hasn’t been merged in Gerrit yet, have you only done the prunepackexpire change? Or have you deployed a patched JGIt in your Gerrit.war?

Ah, I was under the impression you were only seeing the "...pack is corrupt, removing it from pack list" warnings, and not seeing the "Unreadable pack index" errors. In my case, the pack files removed from the pack list *really* aren't there anymore, because they've been removed by gc. Will your jgit fix also resolve the issue I'm seeing? They appear to be different, and it doesn't make sense to me that reloading the pack list will fix the problem I'm seeing, since the pack files *really* aren't there, and therefore the pack list reload won't bring them back. There's also a decent chance I'm misunderstanding things :)

-Luke

On Wednesday, March 13, 2019 at 10:28:34 AM UTC-7, lucamilanesio wrote:

Sent from my iPhone

On 13 Mar 2019, at 17:20, luke....@hpe.com wrote:

Turns out we still gt the below errors with prunepackexpire = 1.day.ago (running gerrit gc twice daily):

[luke....@hpe.com]

We are still getting the errors after upgrading Gerrit to 2.16.7 :( I'd like to think the errors we're seeing are unimportant, since the pack files that Gerrit is complaining about are "old" and should already be consolidated to a new pack file, but I'm afraid of losing history again, and therefore I'm afraid of going back to a normal gc schedule of multiple times per day (which we should do, given our daily activity. Any ideas?

-Luke

For more options, visit <a href="https://groups.google.com/d/optout" t

[Luca Milanesio]

On 24 Mar 2019, at 18:55, luke....@hpe.com wrote:

We are still getting the errors after upgrading Gerrit to 2.16.7 :( I'd like to think the errors we're seeing are unimportant, since the pack files that Gerrit is complaining about are "old" and should already be consolidated to a new pack file, but I'm afraid of losing history again, and therefore I'm afraid of going back to a normal gc schedule of multiple times per day (which we should do, given our daily activity. Any ideas?

Just to clarify, the message of "pack is corrupt, removing it from pack list" will continue to exist even after the patch: the "corrupt" wording is misleading, it really means that the in-memory cache is stale.

So, don't worry, that is only a false alarm.

The problem was really what happened *after* that message, a series of "missing unknown" for me ... and that problem after the fix is 100% gone.

What do you see instead as error messages *after* the message?

Luca.

For more options, visit https://groups.google.com/d/optout.

[luke....@hpe.com]

We actually get errors *before* seeing the "pack is corrupt, removing it from pack list" message. The errors we're seeing are:

...Exception caught while accessing pack file <pack file>, the pack file might be corrupt. Caught 1 consecutive errors while trying to read this pack.

java.io.IOException: Unreadable pack index <corresponding idx file>

We get a whole bunch of those errors, as well as the "pack is corrupt, removing it from pack list" messages. However, it seems "fine" that tasks find these idx files are unreadable, because they literally don't exist anymore, GC deleted them. I *do* have prunepackexpire set to 2 days ago, and I've been running 'gerrit gc' once per day. But each time gc runs and deletes all the 2+ day old pack files, various Gerrit tasks seem to be trying to read from them concurrently, and we get the above errors.

I don't know if this is a problem or not. I'd like to think it isn't, now that we set prunepackexpire=2.days.ago. I'd *like* to think that jgit gc will take all the pack files, and consolidate them into one single pack, but just keep the older pack files around, in which case, removing or corrupting the 2 day old packs wouldn't possibly cause any problems since we still have the data in the one large pack file. However, I don't know if it actually works like that :)

-Luke

[Luca Milanesio]

On 25 Mar 2019, at 00:56, luke....@hpe.com wrote:

We actually get errors *before* seeing the "pack is corrupt, removing it from pack list" message. The errors we're seeing are:

...Exception caught while accessing pack file <pack file>, the pack file might be corrupt. Caught 1 consecutive errors while trying to read this pack.

java.io.IOException: Unreadable pack index <corresponding idx file>

That's just logging saying that JGit can't read that pack, but it isn't a problem until you see a user-related error and a user-related stack trace associated with it.

We get a whole bunch of those errors, as well as the "pack is corrupt, removing it from pack list" messages. However, it seems "fine" that tasks find these idx files are unreadable, because they literally don't exist anymore, GC deleted them. I *do* have prunepackexpire set to 2 days ago, and I've been running 'gerrit gc' once per day.

Yes, whenever an expired pack is removed, you'll see errors like the one above. However, because there is a newer pack available, the objects are still found.

But each time gc runs and deletes all the 2+ day old pack files, various Gerrit tasks seem to be trying to read from them concurrently, and we get the above errors.

I don't know if this is a problem or not.

Until you see a user-related stack-trace, it isn't a problem.

I'd like to think it isn't, now that we set prunepackexpire=2.days.ago. I'd *like* to think that jgit gc will take all the pack files, and consolidate them into one single pack, but just keep the older pack files around, in which case, removing or corrupting the 2 day old packs wouldn't possibly cause any problems since we still have the data in the one large pack file.

Yes, that's the case.

[Luca Milanesio]

On 25 Mar 2019, at 02:18, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 25 Mar 2019, at 00:56, luke....@hpe.com wrote:

We actually get errors *before* seeing the "pack is corrupt, removing it from pack list" message. The errors we're seeing are:

...Exception caught while accessing pack file <pack file>, the pack file might be corrupt. Caught 1 consecutive errors while trying to read this pack.

java.io.IOException: Unreadable pack index <corresponding idx file>

That's just logging saying that JGit can't read that pack, but it isn't a problem until you see a user-related error and a user-related stack trace associated with it.

We get a whole bunch of those errors, as well as the "pack is corrupt, removing it from pack list" messages. However, it seems "fine" that tasks find these idx files are unreadable, because they literally don't exist anymore, GC deleted them. I *do* have prunepackexpire set to 2 days ago, and I've been running 'gerrit gc' once per day.

Yes, whenever an expired pack is removed, you'll see errors like the one above. However, because there is a newer pack available, the objects are still found.

But each time gc runs and deletes all the 2+ day old pack files, various Gerrit tasks seem to be trying to read from them concurrently, and we get the above errors.

I don't know if this is a problem or not.

Until you see a user-related stack-trace, it isn't a problem.

I'd like to think it isn't, now that we set prunepackexpire=2.days.ago. I'd *like* to think that jgit gc will take all the pack files, and consolidate them into one single pack, but just keep the older pack files around, in which case, removing or corrupting the 2 day old packs wouldn't possibly cause any problems since we still have the data in the one large pack file.

Yes, that's the case.

However, I don't know if it actually works like that :)

-Luke

On Sunday, March 24, 2019 at 2:27:43 PM UTC-7, lucamilanesio wrote:

On 24 Mar 2019, at 18:55, luke....@hpe.com wrote:

We are still getting the errors after upgrading Gerrit to 2.16.7 :( I'd like to think the errors we're seeing are unimportant, since the pack files that Gerrit is complaining about are "old" and should already be consolidated to a new pack file, but I'm afraid of losing history again, and therefore I'm afraid of going back to a normal gc schedule of multiple times per day (which we should do, given our daily activity. Any ideas?

Just to clarify, the message of "pack is corrupt, removing it from pack list" will continue to exist even after the patch: the "corrupt" wording is misleading, it really means that the in-memory cache is stale.

So, don't worry, that is only a false alarm.

The problem was really what happened *after* that message, a series of "missing unknown" for me ... and that problem after the fix is 100% gone.

What do you see instead as error messages *after* the message?

FYI, I believe I've found *another* bug with JGit and concurrent reads, different than the one fixed in v2.16.7: we had a 15' inconsistency again a few days ago :-(

Adding more diagnostic to have more light on it.

Luca.

[Luke Engle]

That's good to hear, thank you for clearing that up. What does it mean if we get a user related error? Is that related to the issues you've been seeing which were fixed with the latest jgit patch?

-Luke

You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/v-2YGmmeKE4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.

[luke....@hpe.com]

Hi Luca,

Was this issue every fully resolved? Since we began running jgit gc outside of gerrit (i.e. not using Gerrit gc) we've been seeing this same issue with some of our smaller, less active repos. We are currently on 2.16.7, is the issue fully resolved in 2.16.8?

Thanks,

Luke

On Sunday, March 24, 2019 at 7:38:39 PM UTC-7, lucamilanesio wrote:

On 25 Mar 2019, at 02:18, Luca Milanesio <luca.m...@gmail.com> wrote:

Luca.

<di

[Luca Milanesio]

On 3 May 2019, at 18:52, luke....@hpe.com wrote:

Hi Luca,

Was this issue every fully resolved? Since we began running jgit gc outside of gerrit (i.e. not using Gerrit gc) we've been seeing this same issue with some of our smaller, less active repos. We are currently on 2.16.7, is the issue fully resolved in 2.16.8?

Yes, it is ... however we are debating some performance issues observed after the fix.

See the full discussion at:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=546891

In a nutshell, there is a 2.5s slowdown of Git pushes.

Luca.

[luke....@hpe.com]

Hm, okay. Thanks for the update! I think we'll stick with bandaids for now until that is resolved. I'd rather include some workarounds for the corruption issue than make pushes 2.5s slower for everyone all the time.

Thanks,

Luke

On Friday, May 3, 2019 at 11:08:35 AM UTC-7, lucamilanesio wrote:

Thanks,

Luca.

<div clas

[Luke Engle]

Any update on this? It looks like the review has gone stale.. I need to upgrade to 2.16.8, but don't want to include the 2.5s slowdown, and I'd also rather not fork jgit. Any way this can be prioritized?

Thanks,

Luke

[Luca Milanesio]

On 21 May 2019, at 16:24, Luke Engle <luke....@hpe.com> wrote:

Any update on this? It looks like the review has gone stale.. I need to upgrade to 2.16.8, but don't want to include the 2.5s slowdown, and I'd also rather not fork jgit. Any way this can be prioritized?

Hi Luke,

the change series is https://git.eclipse.org/r/c/141843/ and is getting addressed on JGit stable-5.2.

Gerrit v2.16 though is on stable-5.1 branch, so you'll need to upgrade to v3.0 to get the fix.

P.S. If you are looking for well defined SLA and Support Levels, there are commercial services available from WANDisco, CollabNet and GerritForge. With regards to OpenSource, there are no well-defined ETAs for fixes like this.

HTH

Luca.

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/a0e69e83-d8bf-41fb-8784-b2a86c4abedc%40googlegroups.com.

[Matthias Sohn]

I can move this patch series to stable-5.1 if this helps for 2.16.
Could someone review it ?

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/6954ABCB-D32D-4617-AFF3-BDF5EFA6FC4C%40gmail.com.

[Luca Milanesio]

On 22 May 2019, at 06:53, Matthias Sohn <matthi...@gmail.com> wrote:

I can move this patch series to stable-5.1 if this helps for 2.16.
Could someone review it ?

It the conflicts aren't too many, that would be useful :-)

Happy to review it, but I don't have any +2 power on JGit ;-)

Luca.

[Matthias Sohn]

moved the series to stable-5.1: https://git.eclipse.org/r/#/c/141843/

[Doug Luedtke]

Is this considered fix now?

[Matthias Sohn]

Still working on the series, this is more complex than expected.

But I think we are coming closer to a solution.

On Mon, Jul 15, 2019 at 6:04 PM Doug Luedtke <douglas...@gmail.com> wrote:

Is this considered fix now?

--

--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/5b65366e-541b-4f27-adc2-58111aee233d%40googlegroups.com.

[Doug Luedtke]

Thank you, Matthias Sohn,

Is there a better summary available about this problem? This thread started as a locked thread, then found a problem with pack files getting deleted, and now the linked ticket has grown due to finding more problems. I don't want to misunderstand what is happening here.

I'm finally getting to plan/test an upgrade from 2.14.18 to the newest 2.16.x available. If there is a chance of data loss from daily activities from a lot of users/bots, then postponing the implementation date for the upgrade would be preferred. If anyone can summarize or point me to the correct documentation I would really appreciate it.

[Luke Engle]

Hi Doug,

The issue I was seeing with repo corruption was due to running git gc (separate from Gerrit) with nothing in place to preserve pack files before gc went and removed them. This caused issues with in flight changes in Gerrit while GC was running. This was due to our lack of understanding the proper way of garbage collecting repos on a live server in Gerrit, not due to any bug in jgit.

AFAIK you should not see repo corruption due the the jgit bug discussed in this thread and will not experience repo corruption if you have a proper gc process.

Thanks,

Luke

--

--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---

You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/v-2YGmmeKE4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/e6fd5325-b6d3-401a-a381-00bb3f40e433%40googlegroups.com.

[Matthias Sohn]

On Mon, Jul 15, 2019 at 10:31 PM Luke Engle <luke.e...@gmail.com> wrote:

Hi Doug,

The issue I was seeing with repo corruption was due to running git gc (separate from Gerrit) with nothing in place to preserve pack files before gc went and removed them. This caused issues with in flight changes in Gerrit while GC was running. This was due to our lack of understanding the proper way of garbage collecting repos on a live server in Gerrit, not due to any bug in jgit.

AFAIK you should not see repo corruption due the the jgit bug discussed in this thread and will not experience repo corruption if you have a proper gc process.

that's correct, the JGit version we are currently fixing is not yet used by any Gerrit release

since we discovered the issue when we tried to update Gerrit to use it.

-Matthias 

[Doug Luedtke]

Thank you, Luke and Matthias!

[Raj Ravi]

Hi Mathias/Luca,

Does this issue is fixed under 2.15.17, As i can see the 2.15.17 release notes says JGIT is upgraded into JGit to 4.11.9.201909030838-r.

Regards,

Raj Ravi

On Tuesday, July 16, 2019 at 7:43:30 PM UTC+5:30, Doug Luedtke wrote:

Thank you, Luke and Matthias!

[doug.r...@wandisco.com]

Matthias:

On Monday, July 15, 2019 at 12:26:40 PM UTC-4 Matthias Sohn wrote:

Still working on the series, this is more complex than expected.

But I think we are coming closer to a solution.

I know this is "old history" now but trying to get some closure.  It appears you backported the fixes to 5.1 and that they made it into a Gerrit 2.16 release?  Can you confirm?

Also, is it safe to use, say standalone jgit 5.1.latest on 2.16?  What about older releases like 2.13, 2.14, 2.15?  There's still a lot of those out there and it would be good to know we can use a fixed version to do standalone jgit gc on older versions (clearly unsupported) of Gerrit.

What about even newer versions of standalone jgit like 5.9.latest on the older Gerrit versions?

Cheers.

Doug