Performance when adding a new SSH key

[thomas...@enalean.com]

Hello,

I currently hit a performance issue when a user add a new SSH key. It takes ~10mn to be processed and taken into account which is quite long for the end user. The platform running gitolite is quite large both in number of user and repos.

If I believe the log, it seems we spent most of the times updating the configs:
2016-12-07.08:15:48 31684 system,/usr/share/gitolite3/triggers/post-compile/update-git-configs,POST_COMPILE
2016-12-07.08:23:05 31684 system,/usr/share/gitolite3/triggers/post-compile/update-gitweb-access-list,POST_COMPILE
2016-12-07.08:23:05 31684 system,/usr/share/gitolite3/commands/access,%,gitweb,R,any
2016-12-07.08:23:38 31684 system,/usr/share/gitolite3/commands/git-config,-r,%,gitweb\.
2016-12-07.08:23:56 31684 system,/usr/share/gitolite3/triggers/post-compile/update-git-daemon-access-list,POST_COMPILE
2016-12-07.08:23:56 31684 system,/usr/share/gitolite3/commands/access,%,daemon,R,any
2016-12-07.08:24:20 31684 END

I have done some strace to try to locate a bit more the issue, the only thing I have really noticed is ~20000 calls to git config. I have benchmarked this outside of gitolite and it takes only ~15 seconds to run so I'm guessing the rest of time is spent in the call to the git_config method and determining if the inspected config option concerned gitolite or not.

I'm able to reproduce this issue on different platform running on different hardware (and I have see nothing in the straces indicating of a performance issue related to something hardware related) on CentOS 6 with gitolite 3.6.6.

This has left me with multiple questions:
* Is my analysis of the gitolite logfile correct?
* I'm not sure to fully understand why gitolite needs to rebuild all the config file when the only change in the admin repo is a new SSH key.
* Are there ways to improve that?


Thanks.

[Sitaram Chamarty]

On Thu, Dec 22, 2016 at 08:37:13AM -0800, thomas...@enalean.com wrote:
> Hello,
>
> I currently hit a performance issue when a user add a new SSH key. It takes ~10mn to be processed and taken into account which is quite long for the end user. The platform running gitolite is quite large both in number of user and repos.
>
> If I believe the log, it seems we spent most of the times updating the configs:
> 2016-12-07.08:15:48 31684 system,/usr/share/gitolite3/triggers/post-compile/update-git-configs,POST_COMPILE
> 2016-12-07.08:23:05 31684 system,/usr/share/gitolite3/triggers/post-compile/update-gitweb-access-list,POST_COMPILE

As you can see, 'update-git-configs' is taking over 7 minutes,
which is the bulk of what is happening.

> This has left me with multiple questions:
> * Is my analysis of the gitolite logfile correct?
> * I'm not sure to fully understand why gitolite needs to rebuild all the config file when the only change in the admin repo is a new SSH key.

There is nothing "incremental" in gitolite. It does not look at
the diff of the repo and act only on the changed parts. Each
time you push the admin repo, it's a full compile.

> * Are there ways to improve that?

Apply the attached patch to your ~/.gitolite.rc file. They're
just two simple changes: comment out one line, and add 3 other
lines just before the "ENABLE" line, so even if automatically
applying it does not work, just do it manually. If your rc file
does not even have an ENABLE section, please mail me a copy and
I'll tell you what to do.

For a more permanent solution, I'd like to reproduce this on my
setup to play with. Can you send me a sufficiently anonymised
copy of your gitolite.conf (privately, if you prefer)?

thanks
sitaram

[Thomas Gerbet]

Thanks for your quick answer.

I have attached a sample of my gitolite.conf with one of the split conf file. It is an extract of a development environment but the content is similar with a production environment. The only things that really changes is the number of repository.

For the suggested patch of the ~/.gitolite.rc file I'm not sure if it is going to change something in my case. The whole gitolite operation is already done in background so it does not block my users, they can do other things while waiting.

[Sitaram Chamarty]

On Wed, Dec 28, 2016 at 02:09:59PM +0100, Thomas Gerbet wrote:
> Thanks for your quick answer.
>
> I have attached a sample of my gitolite.conf with one of the split conf
> file. It is an extract of a development environment but the content is
> similar with a production environment. The only things that really changes
> is the number of repository.

No idea how many you have, but I just tried it -- on my laptop
-- with 10,000 repos (100 repos in each include file, 100
include files), and that step takes 95 seconds.

(Note that the syntax of your description line is wrong for
gitolite v3).

Anyway, this cannot be sped up further, sorry.

> For the suggested patch of the ~/.gitolite.rc file I'm not sure if it is
> going to change something in my case. The whole gitolite operation is
> already done in background so it does not block my users, they can do other
> things while waiting.

Well I guess that's fine then. Your original email said "It

takes ~10mn to be processed and taken into account which is

quite long for the end user", and I (mis-)understood that to
mean they were being blocked somewhere.

regards
sitaram

[Thomas Gerbet]

Hello,

I just checked, there are 13,000 repos on the server. I find surprising the difference between your benchmark and what I got on the server.

How can I pinpoint what's causing this large difference in my environment?

Thanks.

[Sitaram Chamarty]

On Mon, Jan 02, 2017 at 11:02:50AM +0100, Thomas Gerbet wrote:
> Hello,
>
> I just checked, there are 13,000 repos on the server. I find surprising the
> difference between your benchmark and what I got on the server.
> How can I pinpoint what's causing this large difference in my environment?

Well it may not have been fair comparing a server to a laptop,
since a server is, by definition, doing lots of things, while a
laptop is typically not.

As for debugging, you could certainly try to comment out line 52
in that file, which calls system(git-config ...) and see how
long it takes. (2 seconds, on my laptop).

If that fixes things, you could try applying the following patch
to update-git-configs and see if it helps.

(List members: if some of you who use the "config" command in
gitolite.conf heavily could give this a spin and let me know if
it works fine, I'd appreciate it!)

regards
sitaram

[Sitaram Chamarty]

On Mon, Jan 02, 2017 at 08:21:54PM +0530, Sitaram Chamarty wrote:

> If that fixes things, you could try applying the following patch
> to update-git-configs and see if it helps.
>
> (List members: if some of you who use the "config" command in
> gitolite.conf heavily could give this a spin and let me know if
> it works fine, I'd appreciate it!)

Hi everyone,

I thought I'd write up some sort of explanation for this, at
least for people who may not be aware. (Old hands know this
already I am sure).

Incremental compiles are not possible in gitolite, because
config changes can come from anywhere. For example, you can add
something like this:

repo foss/..*
config c = d

and suddenly all your repos whose name matches that pattern have
a new config line that applies to them. Even if you don't have
any "wild repos", and all your repos are explicitly named in
gitolite.conf.

This is true for gitolite *rules* also of course. See [1] and
[2] for details and examples why you may need to do that for
rules and for config lines, respectively.

[1]: http://gitolite.com/gitolite/conf.html#accum
[2]: http://gitolite.com/gitolite/git-config.html#override_conf

As a result, you can't simply take a "diff" of the gitolite.conf
and somehow run only the parts that changed. (Well in theory
you probably could, but it would be far too complex for my
comfort in long term maintenance!)

Hope that explains why we resort to tehe kind of highly
localised optimisations like in the patch in my previous email.

regards
sitaram