Suggestions on NFS tuning for Gerrit repo hosting

[Sebastian Schuberth]

Hi,

since quite a while we're seeing performance issues with the repos our Gerrit serves [1]. We've narrowed the cause down to NFS hosting our repos (same repos and config on local storage is fast).

Can anyone share any performance tuning tips for NFS use with Gerrit / JGit? Also, if there are any reliable alternatives to NFS that in general perform better than NFS I would like to hear about them (I was reading about GlusterFS, but results seem to be very mixed).

[1] https://groups.google.com/d/msg/repo-discuss/pWe2GarzJA4/LdeHFckdt50J

Regards,
Sebastian

[Doug Kelly]

Re-reading your original post, "processing changes" looks like that happens whenever there are preReceive hooks running.  Maybe one of the other gurus can help shine some light on what other things might be happening during that message?

I know a number of sites use NFS internally as their primary storage, and while I've personally seen some issues (for example, gc leaving .nfs* files around because JGit hasn't closed the pack yet), it does run okay.  But, at the same time, at $DAYJOB, we don't run any of our master servers on NFS storage anymore.

[Sebastian Schuberth]

On Wed, Apr 15, 2015 at 6:01 PM, Doug Kelly <doug...@gmail.com> wrote:

> Re-reading your original post, "processing changes" looks like that happens
> whenever there are preReceive hooks running. Maybe one of the other gurus
> can help shine some light on what other things might be happening during
> that message?

Meanwhile I really think this is because of the habit of the current
Git pack protocol to always announce *all* refs when talking to the
client. With a lost of refs/changes this can take a while, esp. when
your repo is stored on NFS.

> the same time, at $DAYJOB, we don't run any of our master servers on NFS
> storage anymore.

What are you using instead? Local storage, or some more other network
storage system?

--
Sebastian Schuberth

[Doug Kelly]

On Wednesday, April 15, 2015 at 1:37:11 PM UTC-5, Sebastian Schuberth wrote:

On Wed, Apr 15, 2015 at 6:01 PM, Doug Kelly <doug...@gmail.com> wrote:

> Re-reading your original post, "processing changes" looks like that happens
> whenever there are preReceive hooks running.  Maybe one of the other gurus
> can help shine some light on what other things might be happening during
> that message?

Meanwhile I really think this is because of the habit of the current
Git pack protocol to always announce *all* refs when talking to the
client. With a lost of refs/changes this can take a while, esp. when
your repo is stored on NFS.

Yeah, at first, I wondered if it might be checking reachability of the commits... but I'll agree. Any large amount of operations are slow on NFS.

 

> the same time, at $DAYJOB, we don't run any of our master servers on NFS
> storage anymore.

What are you using instead? Local storage, or some more other network
storage system? 

We do use local storage, on a RAID with solid-state disks.  This of course has an upper bound on repository size due to the practicality of obtaining large enterprise-grade SSDs, but it was at least enough to keep us stable for many years at our current growth rate.