JGIT gc outside of gerrit

[luke....@hpe.com]

Hi,

What is the best method for running JGIT gc outside of gerrit (so, not running 'gerrit gc')? I don't want gc using gerrit heap space and I don't want to use git gc because git gc doesn't have a way to preserve pack files for X hours/days.

Does jgit provide a way of running jgit gc via the command line?

Thanks,

Luke

[Eric Peterson]

My apologies for taking this on a tangent already, but...

Is it safe to run "git gc" or a JGit gc on a repo that Gerrit is managing while the server is live? My guess would have been "no", but I'd love to be wrong.

—Eric

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eric Peterson]

Btw it looks like there is a jgit CLI tool, but "gc" isn't one of the included subcommands:

https://wiki.eclipse.org/JGit/User_Guide

—Eric

[Luca Milanesio]

I believe there was already a long discussion on the mailing list on this topic (see [1]).

The short answer is NO, unless you take some precautions:

1. The machine that is running Gerrit has enough spare RAM and CPU

2. You are running GC externally using JGit CLI

3. You configure a "reasonable" pack-expire policies at JGit level

4. You have installed the latest patch-releases of Gerrit (from v2.13 up to v2.15)

5. You have on the filesystem enough disk-space available (around 2x the size of the repos)

With v2.16, I am testing the latest patch in production and, if all goes well, I'll release v2.16.8 around the beginning of next week.

If you don't take the necessary 1. to 5. precautions,  *very bad things* may happen :-( form pack-list corruption, to the complete loss of Git objects in flight.

HTH

Luca.

[1] https://groups.google.com/forum/#!searchin/repo-discuss/GC$20concurrently%7Csort:date/repo-discuss/v-2YGmmeKE4/IA4EemP1AgAJ

[Jonathan Nieder]

> *very bad things* may happen :-(

Can you please link to a bug about this?

[Luca Milanesio]

On 24 Apr 2019, at 22:25, Jonathan Nieder <j...@google.com> wrote:

> *very bad things* may happen :-(

Can you please link to a bug about this?

Not one, but many bugs :-)

The most recent one that I fixed was:

https://bugs.chromium.org/p/gerrit/issues/detail?id=10562

However, if you just look for "Missing Unknown" in the issue tracker, you will find many issues related to concurrency of Git (or JGit) GC and normal Gerrit operation.

The culprit was in some recent (well, not so recent, from JGit stable-4.5) refactoring that caused the temporary corruption of the in-memory pack-list.

I fixed the issue already in JGit and thanks to David Pursehouse and Matthias Sohn, it has then been merged upstream until JGit master.

I know that Hugo mentioned a couple of years ago during the Gerrit User Summit 2017 in London that they have even lost in-flight commits :-(

HTH

Luca.

[Jonathan Nieder]

> However, if you just look for "Missing Unknown" in the issue tracker, you will find many issues related to concurrency of Git (or JGit) GC and normal Gerrit operation.

By this you mean https://bugs.eclipse.org/bugs/buglist.cgi?quicksearch=%22missing%20unknown%22?

Mind filing or pointing to one that is reproducible? Many of those in the bug tracker don't appear to have any connection to GC.

More generally, I would like to make a habit of always mentioning a relevant bug whenever providing instructions for working around it. Otherwise it's difficult to make progress on fixing the underlying bugs.

Thanks,

Jonathan

[Luca Milanesio]

On 24 Apr 2019, at 22:38, Jonathan Nieder <j...@google.com> wrote:

> However, if you just look for "Missing Unknown" in the issue tracker, you will find many issues related to concurrency of Git (or JGit) GC and normal Gerrit operation.

By this you mean https://bugs.eclipse.org/bugs/buglist.cgi?quicksearch=%22missing%20unknown%22?

Mind filing or pointing to one that is reproducible? Many of those in the bug tracker don't appear to have any connection to GC.

More generally, I would like to make a habit of always mentioning a relevant bug whenever providing instructions for working around it. Otherwise it's difficult to make progress on fixing the underlying bugs.

The *most specific bug* has been filed on the JGit issue tracker, because the problem isn't isolated to Gerrit but to any JGit-based implementation.

https://bugs.eclipse.org/bugs/show_bug.cgi?id=544199

If you follow the discussion thread, you will see all the extensive details of the problem.

I believe that setups that have an older version of JGit (e.g. the one that Martin uses) do not have the same issues.

Luca.

[Jonathan Nieder]

> https://bugs.eclipse.org/bugs/show_bug.cgi?id=544199

Thanks much. That explains precaution #4 (using a new enough version of JGit and Gerrit) but I am still wondering about precautions 1-3 and 5.

Are they about the same bug?

Jonathan

[Luca Milanesio]

On 24 Apr 2019, at 22:45, Jonathan Nieder <j...@google.com> wrote:

> https://bugs.eclipse.org/bugs/show_bug.cgi?id=544199

Thanks much. That explains precaution #4 (using a new enough version of JGit and Gerrit) but I am still wondering about precautions 1-3 and 5.

Are they about the same bug?

Nope.

1. The machine that is running Gerrit has enough spare RAM and CPU

This is not a bug: if you run a GC on a machine that isn't capable of allocating enough RAM or have enough spare-CPU cycles, you will just kill your server.

It's more *common sense* rather than a bug, but it's worth taking into consideration anyway.

That's the reason why we do GC on the failover node, sharing the repos over NFS.

2. You are running GC externally using JGit CLI

This is not a bug again, but common sense: if you run GC inside Gerrit, the operation is going to trigger the load and invalidation of many in-memory JGit objects and thus putting a lot of stress on the JVM head and triggering lots of JVM GC. Again, you could kill the Gerrit JVM with that.

3. You configure a "reasonable" pack-expire policies at JGit level

This isn't a Gerrit or JGit-specific concept, see:

https://git-scm.com/docs/git-gc

If you don't keep the old objects and packfiles during and after the GC cycle, again you cause the full invalidation of the in-memory cache altogether, causing significant trouble to a running production server.

In our case, a Server without a properly loaded JGit cache can take up to 20-30s to serve an incoming call.

We use NoteDb where a lot of data is store on the repo itself.

5. You have on the filesystem enough disk-space available (around 2x the size of the repos)

Again, common sense. During the GC you double (at least) the disk space used by the repo, because you have the old and the new packfiles present at the same time.

Luca.

[Jonathan Nieder]

Wonderful, thanks for the explanations.

If any of these produce corruption or other problematic symptoms beyond e.g. filling up disk space, please do file a bug. We want this code to do the right thing automatically --- it should respect the user's intentions, instead of requiring the user to carefully respect its needs. :)

Thanks,

Jonathan

[luke....@hpe.com]

> You are running GC externally using JGit CLI

With this precaution, are you saying 'gerrit gc' can/will cause problems (performance related) if ran while the Gerrit server is live/active?

Thanks,

Luke

[1] https://groups.google.com/forum/#!searchin/repo-discuss/GC$20concurrently%7Csort:date/repo-discuss/v-2YGmmeKE4/IA4EemP1AgAJ

—Eric

To unsubscribe, email re...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
To unsubscribe, email re...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
To unsubscribe, email rep...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

[Luca Milanesio]

On 24 Apr 2019, at 23:10, Jonathan Nieder <j...@google.com> wrote:

Wonderful, thanks for the explanations.

If any of these produce corruption or other problematic symptoms beyond e.g. filling up disk space, please do file a bug. We want this code to do the right thing automatically

Yes, that would be ideal.

However, at times, the willingness to be "super-automatic" creates more trouble than help :-(

As an example, the receive.autogc feature in the recent versions of JGit caused significant trouble.

--- it should respect the user's intentions, instead of requiring the user to carefully respect its needs. :)

In the case of JGit I believe it is *very very difficult* to understand the user's intention, just because it is used in so many different use-cases and platforms.

Going back to the example of the receive.autogc, the aim was (I believe) to avoid a local Git repo managed by JGit to grow in the number of packfiles and loose objects when used in a client-based environment.

(e.g. Eclipse). However, the result was that the mechanism triggered itself in a different context (Gerrit server) and inferred the wrong need.

In my experience of 10 years help people with Git and Gerrit, with complex systems, it is best to clearly expose what the software does and it requires with good documentation, training material and clear configuration.

It is very difficult to make *everything* automatic, because the use-cases and the environments are so different that one size never fits all of them.

[Eric Peterson]

On Apr 24, 2019, at 5:53 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:

1. The machine that is running Gerrit has enough spare RAM and CPU

This is not a bug: if you run a GC on a machine that isn't capable of allocating enough RAM or have enough spare-CPU cycles, you will just kill your server.

It's more *common sense* rather than a bug, but it's worth taking into consideration anyway.

That's the reason why we do GC on the failover node, sharing the repos over NFS.

2. You are running GC externally using JGit CLI

This is not a bug again, but common sense: if you run GC inside Gerrit, the operation is going to trigger the load and invalidation of many in-memory JGit objects and thus putting a lot of stress on the JVM head and triggering lots of JVM GC. Again, you could kill the Gerrit JVM with that.

Luca,

What about the idea of running regular "git gc" on a failover node that has access to the repos over NFS? Would that be expected to work?

—Eric

[Luca Milanesio]

On 24 Apr 2019, at 23:27, luke....@hpe.com wrote:

> You are running GC externally using JGit CLI

With this precaution, are you saying 'gerrit gc' can/will cause problems (performance related) if ran while the Gerrit server is live/active?

It depends on:

a) Size of the repo you are GC-ing

b) Spare CPU cycles on the machine

c) Spare JVM heap

If you are GC-ing a small repo, on a server with low CPU utilisation and plenty of JVM heap available, then the answer is no, it won't cause any problem.

But if you are GC-ing a large repo (e.g. 1M objects, 10 GB size), on a heavy loaded machine (70-80% average CPU utilisation) and near full JVM heap (less than 10GB available) ... then you could just kill your Gerrit JVM.

Luca.

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[Luca Milanesio]

I believe Martin does exactly that, but he is running a different version of Gerrit.

We never use 'git gc' because the support for bitmap generation is incomplete and the "optimized repo" wouldn't be so fast compared to the bitmap generated by 'jgit gc'.

HTH

Luca.

[Martin Fick]

On Wednesday, April 24, 2019 11:41:15 PM MDT Luca Milanesio wrote:
> > On 24 Apr 2019, at 23:28, Eric Peterson <epet...@interactivebrokers.com>
wrote:
> >> On Apr 24, 2019, at 5:53 PM, Luca Milanesio <luca.mi...@gmail.com

<mailto:luca.mi...@gmail.com>> wrote:
> >
> > What about the idea of running regular "git gc" on a failover node that
> > has access to the repos over NFS? Would that be expected to work?

> I believe Martin does exactly that, but he is running a different version of
> Gerrit.

So far on any existing version of Gerrit, if you run git or jgit gc on another
NFS node and have NFS file/dir attribute caching turned on, you may see
occasional ref rewinds. This is because refs unlike objects, are mutable and
their content gets cached, usually for 1 min, but I believe we see times where
it is longer (over 6mins with a 1mins cache setting). I believe it is possible
to have a more catastrophic failure, such as the packed-refs file being lost
(so many lost refs), but I believe that is rare.

We had patches uploaded to jgit to fix this, but they had severe performance
issues (likely equivalent to turning off NFS attribute caching). We still
would like to work on better solutions to fix this, I believe it is possible.

> We never use 'git gc' because the support for bitmap generation is
> incomplete and the "optimized repo" wouldn't be so fast compared to the
> bitmap generated by 'jgit gc'.

Can you elaborate on this?

-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

[Luca Milanesio]

I believe that Jonathan knows the bitmap internals of C-Git vs. J-Git better than me :-)

Luca.

[Doug Robinson]

Luca:

I've let this sit a while and I understand that there are still parts moving about.  I was hoping things to settle out.  Not sure they have but I'll ask anyway:

     What you might consider the "best" version of JGit standalone to be used to GC repositories on Gerrit?  Especially for folks running older Gerrit versions such as 2.13?

Cheers.

Doug