Gerrit LFS - 100% CPU when/after cloning

111 views
Skip to first unread message

Duft Markus

unread,
Jun 12, 2018, 4:19:08 AM6/12/18
to Repo and Gerrit Discussion

Hey,

 

I upgraded a 2.15 test instance to current master – including the LFS plugin (current master as well). Now whenever cloning a repository which requires LFS objects, when the download by the client starts Gerrit jumps to 100% CPU (actually 700-800% CPU – 8 cores J) and does not seem to get down anymore. Even when cancelling the cloning client, Gerrit remains in this state. Only restarting Gerrit seems to resolve the issue. I have taken a few (3 or so) jstack dumps of the VM, and it indeed seems that LFS is the only thing being worked on – or does anyone see something else which might be the problem?

 

Thanks for helping investigate this – I’d like to update our instance to master but this is (of course J) blocking me a little.

 

Cheers,

Markus

 

--

Mit freundlichen Grüßen / Best regards

 

Markus Duft | Software Architect

SSI SCHÄFER | SSI Schäfer IT Solutions GmbH | Friesachstraße 15 | 8114 Friesach bei Graz | Austria

Phone +43 3127 200-575 | Fax +43 3127 200-22

marku...@ssi-schaefer.com

Website | Blog | YouTube | Facebook

 


SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
Commercial Court: Landesgericht für Zivilrechtssachen Graz

Duft Markus

unread,
Jun 12, 2018, 4:19:45 AM6/12/18
to Repo and Gerrit Discussion

Attaching files is hard… ;)

lfs-jstack.txt

David Pursehouse

unread,
Jun 12, 2018, 4:22:16 AM6/12/18
to Duft Markus, Repo and Gerrit Discussion
On Tue, Jun 12, 2018 at 5:19 PM Duft Markus <Marku...@ssi-schaefer.com> wrote:

Hey,

 

I upgraded a 2.15 test instance to current master – including the LFS plugin (current master as well). Now whenever cloning a repository which requires LFS objects, when the download by the client starts Gerrit jumps to 100% CPU (actually 700-800% CPU – 8 cores J) and does not seem to get down anymore. Even when cancelling the cloning client, Gerrit remains in this state. Only restarting Gerrit seems to resolve the issue. I have taken a few (3 or so) jstack dumps of the VM, and it indeed seems that LFS is the only thing being worked on – or does anyone see something else which might be the problem?


I don't think anyone else has tried LFS on 2.15 yet.  We (CollabNet) are still based on 2.14 and that's where we've tested it.

I've only done very basic smoke tests of the merges up from stable-2.15.

 

 

Thanks for helping investigate this – I’d like to update our instance to master but this is (of course J) blocking me a little.

 

Cheers,

Markus

 

--

Mit freundlichen Grüßen / Best regards

 

Markus Duft | Software Architect

SSI SCHÄFER | SSI Schäfer IT Solutions GmbH | Friesachstraße 15 | 8114 Friesach bei Graz | Austria

Phone +43 3127 200-575 | Fax +43 3127 200-22

marku...@ssi-schaefer.com

Website | Blog | YouTube | Facebook

 


SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
Commercial Court: Landesgericht für Zivilrechtssachen Graz

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Duft Markus

unread,
Jun 12, 2018, 4:24:07 AM6/12/18
to David Pursehouse, Repo and Gerrit Discussion

Well, 2.15 works very well, we have it in production. Master seems to cause problems J

 

Cheers,

Markus

David Pursehouse

unread,
Jun 12, 2018, 4:27:00 AM6/12/18
to Duft Markus, Repo and Gerrit Discussion
On Tue, Jun 12, 2018 at 5:24 PM Duft Markus <Marku...@ssi-schaefer.com> wrote:

Well, 2.15 works very well, we have it in production. Master seems to cause problems J


Ah, sorry, I've misread your message.

In that case the problem is quite likely that the plugin's master branch is not aligned with gerrit's master.  The first thing I would check is if everything has been merged up from the stable branches, and then if the JGit version has been updated to the same as core.

Duft Markus

unread,
Jun 12, 2018, 4:29:53 AM6/12/18
to David Pursehouse, Repo and Gerrit Discussion

OK, thanks, I will start investigating :D

Duft Markus

unread,
Jun 12, 2018, 4:34:11 AM6/12/18
to David Pursehouse, Repo and Gerrit Discussion

Uh oh – there is a merge missing from stable-2.15 to master – I am not powerful enough to get this done though, the conflicts are too tough for me :D

 

Would you have time this week to help me through the merge – or even do it yourself? For you it’s probably a quick one - ~9 conflicting files.

 

Cheers,

Markus

Duft Markus

unread,
Jun 12, 2018, 4:35:37 AM6/12/18
to David Pursehouse, Repo and Gerrit Discussion

Hm. But on master there is only a single smallish commit since the last merge from stable-2.15. Why is this then conflicting to much?

David Pursehouse

unread,
Jun 12, 2018, 5:12:27 AM6/12/18
to Duft Markus, Repo and Gerrit Discussion
I'll look into it later.

Duft Markus

unread,
Jun 12, 2018, 6:10:45 AM6/12/18
to David Pursehouse, Repo and Gerrit Discussion

Thanks!

Duft Markus

unread,
Jun 12, 2018, 7:02:32 AM6/12/18
to David Pursehouse, Repo and Gerrit Discussion

Just as some additional info: updating JGit to the same 4.11 version as gerrit has, had no effect. Still 700+% CPU usage on download of LFS files on master… I’ve tried to let the download run through now, although my Laptop is nearly dying. Everything is localhost (Gerrit, git clone) so it should be fast, but it’s extremely slow (git clone reports ~100kb/s - ~300kb/s) after the first few megabytes (~60MB) where actually ‘OK’ speed wise (~10mb/s)…  Now (after more than an hour) it is still at “Filtering content:   7% (130/1847), 127.58 MiB | 57.00 KiB/s” – I’m cancelling it now K

David Pursehouse

unread,
Jun 12, 2018, 7:48:19 AM6/12/18
to Duft Markus, Repo and Gerrit Discussion
On Tue, Jun 12, 2018 at 8:02 PM Duft Markus <Marku...@ssi-schaefer.com> wrote:

Just as some additional info: updating JGit to the same 4.11 version as gerrit has, had no effect. Still 700+% CPU usage on download of LFS files on master…


I'm seeing similar after upgrading JGit and merging from stable-2.15.  I only have a small test project with a few LFS objects in it, but just cloning that causes my laptop's fan to kick in.

The merge from stable-2.15 shouldn't have had any effect; the changes there were mostly already in master. The conflicts were due to the Java8 date/time migration being done first on master, and then differently on stable-2.14.

I haven't investigated any further.

David Pursehouse

unread,
Jun 12, 2018, 9:01:08 AM6/12/18
to Duft Markus, Repo and Gerrit Discussion
On Tue, Jun 12, 2018 at 8:48 PM David Pursehouse <david.pu...@gmail.com> wrote:
On Tue, Jun 12, 2018 at 8:02 PM Duft Markus <Marku...@ssi-schaefer.com> wrote:

Just as some additional info: updating JGit to the same 4.11 version as gerrit has, had no effect. Still 700+% CPU usage on download of LFS files on master…


I'm seeing similar after upgrading JGit and merging from stable-2.15.  I only have a small test project with a few LFS objects in it, but just cloning that causes my laptop's fan to kick in.

The merge from stable-2.15 shouldn't have had any effect; the changes there were mostly already in master. The conflicts were due to the Java8 date/time migration being done first on master, and then differently on stable-2.14.

I haven't investigated any further.

It seems to be caused by the recent Jetty upgrade.  I'm not sure why or how, but after reverting it, clone with LFS works as expected.

Duft Markus

unread,
Jun 12, 2018, 9:06:06 AM6/12/18
to David Pursehouse, Repo and Gerrit Discussion

Thanks David for finding this! It definitely helps a lot J

 

With the new jetty (cancelled after more than an hour):

mduft@fril0041 /tmp/TEST002 $ git checkout origin/5.8/master

Checking out files: 100% (58579/58579), done.

^Cltering content:   7% (130/1847), 127.58 MiB | 57.00 KiB/s   

Exiting because of "interrupt" signal.

 

With the reverted-to old jetty:

mduft@fril0041 /tmp $ git clone ssh://marku...@ssi-schaefer.com@fril0041.wamas.com:2501/TEST002

Cloning into 'TEST002'...

remote: Total 70891 (delta 0), reused 70891 (delta 0)

Receiving objects: 100% (70891/70891), 58.79 MiB | 26.40 MiB/s, done.

Resolving deltas: 100% (30164/30164), done.

Checking out files: 100% (58274/58274), done.

Filtering content: 100% (1606/1606), 253.01 MiB | 220.87 MiB/s, done.

 

Too bad we’re using the new jetty version in our own software K will have to check that too :D

thomasmu...@yahoo.com

unread,
Jun 12, 2018, 9:18:54 AM6/12/18
to Repo and Gerrit Discussion

David Ostrovsky

unread,
Sep 15, 2018, 3:32:24 PM9/15/18
to Repo and Gerrit Discussion

On Tuesday, June 12, 2018 at 3:01:08 PM UTC+2, David Pursehouse wrote:
On Tue, Jun 12, 2018 at 8:48 PM David Pursehouse <david.pu...@gmail.com> wrote:
On Tue, Jun 12, 2018 at 8:02 PM Duft Markus <Marku...@ssi-schaefer.com> wrote:

Just as some additional info: updating JGit to the same 4.11 version as gerrit has, had no effect. Still 700+% CPU usage on download of LFS files on master…


I'm seeing similar after upgrading JGit and merging from stable-2.15.  I only have a small test project with a few LFS objects in it, but just cloning that causes my laptop's fan to kick in.

The merge from stable-2.15 shouldn't have had any effect; the changes there were mostly already in master. The conflicts were due to the Java8 date/time migration being done first on master, and then differently on stable-2.14.

I haven't investigated any further.

It seems to be caused by the recent Jetty upgrade.  I'm not sure why or how, but after reverting it, clone with LFS works as expected.


The root cause for performance degradation after bumping of Jetty version
was hopefully fixed in lfs-server project: [1], and here is another attempt to
bump Jetty to 9.4.12: [2].


Reply all
Reply to author
Forward
0 new messages