Pushing files with large delta is stuck and increases CPU usage

[Bassem Rabil]

Hi 

We are facing an issue with Gerrit 2.6.1 with users trying to push machine generated text source files with large delta compared to previous version, the number of changed lines is over 200k lines. The total size of the text file is 6 MB, however pushing this file is stuck and the CPU usage for Gerrit increases causing high load on Gerrit servers. The push is stuck when trying to push this for review "refs/for/master" and it is stuck as well when trying to push this file to "master" branch directly without review. There is no errors in the error log for this push, here below the output when trying to push such file.

Will tuning  the configuration pack.bigFileThreshold down to lower threshold e.g. 2-3m help to overcome this issue. Currently we set this to 20m as per the recommendation by Shawn at EclipseCon 2013:

http://www.eclipsecon.org/2013/sites/eclipsecon.org.2013/files/Scaling%20Up%20JGit%20-%20EclipseCon%202013.pdf

Any suggestion or hint is appreciated.

Thanks and Regards

Bassem Rabil Guendy

$> git push origin HEAD:master

Counting objects: 15, done.

Delta compression using up to 4 threads.

Compressing objects: 100% (6/6), done.

Writing objects: 100% (8/8), 22.53 KiB | 0 bytes/s, done.

Total 8 (delta 5), reused 0 (delta 0)

remote: Resolving deltas:  20% (1/5)   --- Hanging here

[Saša Živkov]

What is the value of core.streamFileThreshold in your Gerrit server?

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
 
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Bassem Rabil Guendy]

Currently core.streamFileThreshold is set to 2048m.

 

Thanks

Bassem

[Saša Živkov]

If your largest file is 6MB then the core.streamFileTreshold should be larger than 6MB.

In general it should be larger than the largest expected object in all Git repositories.

[Doug Kelly]

On Monday, September 30, 2013 2:10:48 PM UTC-5, zivkov wrote:

If your largest file is 6MB then the core.streamFileTreshold should be larger than 6MB.

In general it should be larger than the largest expected object in all Git repositories.

I've seen similar problems, so I'm also curious about this.  Since streamFileThreshold is 2048, it seems unlikely that it is to blame.  But, I wonder, is anything exposed when looking at either jstack or jvisualvm?  I remember there was talk about the zip inflater getting stuck in a nearly-infinite loop (it will actually terminate, it's just excruciatingly slow), and there was talk of a reproduction case as well, but that topic seems to have died off.

--Doug

[Bassem Rabil]

Here below jstack for this issue, however I think lowering pack.bigFileThreshold to some size lower than 6 mega bytes would workaround this hanging push. The  core.streamFileThreshold is fairly away from such file of only 6m to push.

"SSH git-receive-pack '<project-name>' (<user-id>)" prio=10 tid=0x0000000007c01000 nid=0x22c6 runnable [0x0000000043d98000]

   java.lang.Thread.State: RUNNABLE

        at java.util.zip.Inflater.inflateBytes(Native Method)

        at java.util.zip.Inflater.inflate(Inflater.java:256)

        - locked <0x000000078febea10> (a java.util.zip.ZStreamRef)

        at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)

        at java.util.zip.InflaterInputStream.skip(InflaterInputStream.java:208)

        at java.io.BufferedInputStream.skip(BufferedInputStream.java:366)

        - locked <0x00000007a2ac7ab8> (a java.io.BufferedInputStream)

        at org.eclipse.jgit.lib.ObjectStream$Filter.skip(ObjectStream.java:199)

        at org.eclipse.jgit.util.IO.skipFully(IO.java:330)

        at org.eclipse.jgit.internal.storage.pack.DeltaStream.seekBase(DeltaStream.java:333)

        at org.eclipse.jgit.internal.storage.pack.DeltaStream.read(DeltaStream.java:213)

        at org.eclipse.jgit.internal.storage.pack.DeltaStream.read(DeltaStream.java:214)

        at org.eclipse.jgit.internal.storage.pack.DeltaStream.read(DeltaStream.java:214)

        at org.eclipse.jgit.internal.storage.pack.DeltaStream.read(DeltaStream.java:214)

        at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)

        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)

        - locked <0x000000073f205d58> (a java.io.BufferedInputStream)

        at org.eclipse.jgit.util.io.TeeInputStream.read(TeeInputStream.java:112)

        at org.eclipse.jgit.lib.ObjectStream$Filter.read(ObjectStream.java:209)

        at org.eclipse.jgit.util.IO.readFully(IO.java:246)

        at org.eclipse.jgit.lib.ObjectLoader.getCachedBytes(ObjectLoader.java:203)

        at org.eclipse.jgit.transport.PackParser.resolveDeltasWithExternalBases(PackParser.java:782)

        at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:501)

        at org.eclipse.jgit.internal.storage.file.ObjectDirectoryPackParser.parse(ObjectDirectoryPackParser.java:179)

        at org.eclipse.jgit.transport.BaseReceivePack.receivePack(BaseReceivePack.java:938)

        at org.eclipse.jgit.transport.BaseReceivePack.receivePackAndCheckConnectivity(BaseReceivePack.java:766)

        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:191)

        at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:160)

        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:100)

        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)

        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)

        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)

        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:429)

        - locked <0x000000073d331ce8> (a com.google.gerrit.sshd.BaseCommand$TaskThunk)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

        at java.util.concurrent.FutureTask.run(FutureTask.java:166)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:337)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:724)

[Shawn Pearce]

On Mon, Sep 30, 2013 at 12:47 PM, Bassem Rabil
<bassem.ra...@ericsson.com> wrote:
> Here below jstack for this issue, however I think lowering
> pack.bigFileThreshold to some size lower than 6 mega bytes would workaround
> this hanging push. The core.streamFileThreshold is fairly away from such
> file of only 6m to push.

pack.bigFileThreshold is only used when running `gerrit gc`.

> "SSH git-receive-pack '<project-name>' (<user-id>)" prio=10
> tid=0x0000000007c01000 nid=0x22c6 runnable [0x0000000043d98000]
> java.lang.Thread.State: RUNNABLE
> at java.util.zip.Inflater.inflateBytes(Native Method)
> at java.util.zip.Inflater.inflate(Inflater.java:256)
> - locked <0x000000078febea10> (a java.util.zip.ZStreamRef)
> at
> java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
> at
> java.util.zip.InflaterInputStream.skip(InflaterInputStream.java:208)
> at java.io.BufferedInputStream.skip(BufferedInputStream.java:366)
> - locked <0x00000007a2ac7ab8> (a java.io.BufferedInputStream)
> at
> org.eclipse.jgit.lib.ObjectStream$Filter.skip(ObjectStream.java:199)
> at org.eclipse.jgit.util.IO.skipFully(IO.java:330)
> at
> org.eclipse.jgit.internal.storage.pack.DeltaStream.seekBase(DeltaStream.java:333)
> at

core.streamFileThreshold is set too small for this input, whatever it
is. The code is falling into the slow delta resolution path.

[Bassem Rabil]

I tried to increase core.streamFileThreshold to 4098m and 8196m, and still the push is stuck on Resolving Deltas 20%. The user was able to push the file as a new file, and not an updated version of existing file. That is the workaround we suggested for the user to use until we resolve this issue.

Regards

Bassem

[Bassem Rabil]

It seems this was resulting from a jgit bug for repositories with large pack files described at:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=394078

We were able to fix the affected repository by setting bigFileThreshold = 20M 

 

Then repack the repository using git gc --aggressive to repack using the newly set file limit. After that the repository is not hanging during delta compression of pushes.

Regards

Bassem