Appropriate use/configuration of Gerrit

[Larry Gadallah]

Hello all:

We have a new Gerrit installation with some relatively complex
permissions settings, and we have been able to create a very simple
project and verify that the permissions configuration works, and that
code review and commits to refs/heads/* work. The problem we have run
into is that when we try to checkout a large project (like AOSP) via
repo and ssh, there are errors, and eventually Gerrit locks up to the
point where Tomcat has to be restarted to revive it (a single Java
process is taking 100% CPU, no messages about memory shortages can be
found). The user sees messages like this during the repo sync:

fatal: internal server error
fatal: The remote end hung up unexpectedly
fatal: protocol error: bad pack header

The Gerrit logs contain messages like this:

com.google.gerrit.server.ssh.BaseCommand - Internal server error
(user <name> account <number>) during git-upload-pack '/foo.git'

Once these errors appear, the Gerrit web UI seems to become
unresponsive and both Gerrit and Tomcat have to be kicked to get
things going again.

Similar problems have been reported here as noted in the following
threads:

http://groups.google.com/group/repo-discuss/browse_thread/thread/6eb1d0a73da8ed70

http://groups.google.com/group/repo-discuss/browse_thread/thread/0d137c9e55e55542

http://groups.google.com/group/repo-discuss/browse_thread/thread/97d4e1f92f5ca477

http://groups.google.com/group/repo-discuss/browse_thread/thread/80ecf6c993363fd8

Our environment is as follows:

- 64 Gb RAM, 16 physical CPUs
- Fedora 15 x86_64, Linux 2.6.40.6-0.fc15.x86_64
- Java HotSpot 64-Bit Server VM build 1.6.0_29-b11
- Tomcat 6.0.32-5
- Gerrit Code Review 2.2.1

We are wondering about a couple of things:

1. Is the intended usage of Gerrit to have it provide all access to
the underlying git repos via Gerrit, or is it only intended to receive
pushes of changes to be reviewed? We have seen some evidence,
particularly for large projects (i.e. Android), that others checkout
their project repos directly from gitd using git or http protocols,
and then push their changes via ssh to Gerrit for review, and if these
are accepted, they are in turn pushed by Gerrit into the git repos.

2. If it is the case that Gerrit should be able to support git access
by itself, what kind of configuration or tuning (i.e. gerrit.config)
is needed to avoid the problems we are facing?

Thanks,
--
Larry Gadallah
larry.g...@gmail.com

[Magnus Bäck]

On Saturday, December 17, 2011 at 22:57 CET,
Larry Gadallah <larry.g...@elektrobit.com> wrote:

[...]

> The user sees messages like this during the repo sync:
>
> fatal: internal server error
> fatal: The remote end hung up unexpectedly
> fatal: protocol error: bad pack header
>
> The Gerrit logs contain messages like this:
>
> com.google.gerrit.server.ssh.BaseCommand - Internal server error
> (user <name> account <number>) during git-upload-pack '/foo.git'
>
> Once these errors appear, the Gerrit web UI seems to become
> unresponsive and both Gerrit and Tomcat have to be kicked to get
> things going again.

Is this the only thing you get in the logs? It seems more like the
symptom than the cause. How big is the Java heap? What kind of user
load are you seeing when this happens?

> 1. Is the intended usage of Gerrit to have it provide all access to
> the underlying git repos via Gerrit, or is it only intended to receive
> pushes of changes to be reviewed?

Both. If you don't need read access control you can boost performance by
serving the gits via the Git protocol (or HTTP), but otherwise Gerrit's
built-in SSH daemon is happy to serve your data. It's what we do, and we
do use Gerrit's read access controls.

[...]

> 2. If it is the case that Gerrit should be able to support git access
> by itself, what kind of configuration or tuning (i.e. gerrit.config)
> is needed to avoid the problems we are facing?

Sorry, I can't really help out here. Hopefully someone else can chime in.

--
Magnus Bäck Opinions are my own and do not necessarily
SW Configuration Manager represent the ones of my employer, etc.
Sony Ericsson

[Larry Gadallah]

Hello Magnus:

Thanks for your response.

On Dec 19 2011, 12:15 pm, Magnus Bäck <magnus.b...@sonyericsson.com>
wrote:

> On Saturday, December 17, 2011 at 22:57 CET,

>      LarryGadallah<larry.gadal...@elektrobit.com> wrote:
>
> [...]
>
> > The user sees messages like this during the repo sync:
>
> >     fatal: internal server error
> >     fatal: The remote end hung up unexpectedly
> >     fatal: protocol error: bad pack header
>
> > The Gerrit logs contain messages like this:
>
> >     com.google.gerrit.server.ssh.BaseCommand - Internal server error
> > (user <name> account <number>) during git-upload-pack '/foo.git'
>
> > Once these errors appear, the Gerrit web UI seems to become
> > unresponsive and both Gerrit and Tomcat have to be kicked to get
> > things going again.
>
> Is this the only thing you get in the logs? It seems more like the
> symptom than the cause. How big is the Java heap? What kind of user
> load are you seeing when this happens?

This happens with only 1 or 2 users on a pretty large server.
Certainly not a "production" context.

>
> > 1. Is the intended usage of Gerrit to have it provide all access to
> > the underlying git repos via Gerrit, or is it only intended to receive
> > pushes of changes to be reviewed?
>
> Both. If you don't need read access control you can boost performance by
> serving the gits via the Git protocol (or HTTP), but otherwise Gerrit's
> built-in SSH daemon is happy to serve your data. It's what we do, and we
> do use Gerrit's read access controls.

I understand from what you are saying that you (and others,
presumably) are able to use Gerrit to control all access to repos.
This is what we would like to do, but we are running into some kind of
tuning or configuration problem with Gerrit.

>
> [...]
>
> > 2. If it is the case that Gerrit should be able to support git access
> > by itself, what kind of configuration or tuning (i.e. gerrit.config)
> > is needed to avoid the problems we are facing?
>
> Sorry, I can't really help out here. Hopefully someone else can chime in.
>

I wonder if anyone else can suggest what we should look at for
problems with the JVM/Tomcat configuration? We do see that when Gerrit
hangs, the java process is taking 100% of CPU, but we have not yet
seen any messages regarding running out of memory. What kinds of
levers and knobs with the JVM/Tomcat should or could normally be
manipulated to "tune" the operation of Gerrit?

Best regards,
--
Larry Gadallah
larry.g...@gmail.com

[Larry Gadallah]

Hi all:

On Jan 3, 11:25 am, Larry Gadallah <larry.gadal...@elektrobit.com>
wrote:

> - Show quoted text -

To close this thread: Our sysadmin modified the JVM options on the
Gerrit server and removed the -Xms and -Mmx limits for the JVM memory
and now Gerrit behaves in the expected way (despite taking up as much
memory as it can grab). Now we can checkout large projects without any
problems and the GUI functions throughout these operations without
hanging.