Error allocating memory when performing delta compression

Mark Bekhet

unread,

Aug 15, 2022, 8:41:24 AM8/15/22

to Repo and Gerrit Discussion

Hi,

We encounter a problem when we try to execute the command in [1] on a large repository and we have pack.deltacompression = on. This happens on Gerrit 3.4.5. We have the following configuration on the server:

[core]
packedGitOpenFiles = 8192
packedGitLimit = 48g
packedGitWindowSize = 64k
streamFileThreshold = 2048m
deltaBaseCacheLimit = 100m
repositoryCacheExpireAfter = 10min
[pack]
bigFileThreshold = 20m
indexVersion = 2
deltacompression = on
threads = 0

When I set deltacompression to off, we don't have a problem and we are able to clone the single branch of the repository.

The server has 18 CPUs, 197902240 kB Total memory, 976688 kB free memory and 85742796 kB available memory.

I reproduced the behaviour locally and it seems that threads are trying to write in the memory but they fail to find free memory to complete the packedGitLimit , I get the following error which I found also on the server:

[2022-08-10T14:41:22.450Z] [pool-30-thread-1] ERROR org.eclipse.jgit.internal.storage.file.PackDirectory : Exception caught while accessing pack file /home/XXXX/gerrit_testsite_3.4/git/XXXX.git/objects/pack/pack-0eb563704e6b771d5a9136643e1acc677d4495df.pack, the pack file might be corrupt. Caught 1 consecutive errors while trying to read this pack.
java.io.IOException: In-memory buffer limit exceeded
at org.eclipse.jgit.internal.storage.file.Pack.load(Pack.java:874)
at org.eclipse.jgit.internal.storage.file.Pack.get(Pack.java:274)
at org.eclipse.jgit.internal.storage.file.PackDirectory.open(PackDirectory.java:212)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:390)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:354)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObjectWithoutRestoring(ObjectDirectory.java:345)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:330)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132)
at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:212)
at org.eclipse.jgit.internal.storage.pack.PackWriter.buffer(PackWriter.java:1885)
at org.eclipse.jgit.internal.storage.pack.DeltaWindow.buffer(DeltaWindow.java:377)
at org.eclipse.jgit.internal.storage.pack.DeltaWindow.delta(DeltaWindow.java:264)
at org.eclipse.jgit.internal.storage.pack.DeltaWindow.searchInWindow(DeltaWindow.java:188)
at org.eclipse.jgit.internal.storage.pack.DeltaWindow.search(DeltaWindow.java:144)
at org.eclipse.jgit.internal.storage.pack.DeltaTask.runWindow(DeltaTask.java:295)
at org.eclipse.jgit.internal.storage.pack.DeltaTask.call(DeltaTask.java:270)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[2022-08-10T14:43:45.722Z] [HTTP POST /a/XXXXX/git-upload-pack (admin from 192.168.122.1)] ERROR com.google.gerrit.httpd.GitOverHttpServlet.GerritUploadPackErrorHandler : Internal error during upload-pack from Repository[/home/XXXX/gerrit_testsite_3.4/git/XXXX.git] [CONTEXT project="XXXXX" ]
org.eclipse.jgit.errors.MissingObjectException: Missing unknown f4132132c28b9b76224f52569e31a1d495ba38fe
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:136)
at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:212)
at org.eclipse.jgit.internal.storage.pack.PackWriter.buffer(PackWriter.java:1885)
at org.eclipse.jgit.internal.storage.pack.DeltaWindow.buffer(DeltaWindow.java:377)
at org.eclipse.jgit.internal.storage.pack.DeltaWindow.delta(DeltaWindow.java:264)
at org.eclipse.jgit.internal.storage.pack.DeltaWindow.searchInWindow(DeltaWindow.java:188)
at org.eclipse.jgit.internal.storage.pack.DeltaWindow.search(DeltaWindow.java:144)
at org.eclipse.jgit.internal.storage.pack.DeltaTask.runWindow(DeltaTask.java:295)
at org.eclipse.jgit.internal.storage.pack.DeltaTask.call(DeltaTask.java:270)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

result: clone fails

Also, casually when I run the clone locally I found this gc-alloc warnings:

[305.466s][warning][gc,alloc] pool-30-thread-1: Retried waiting for GCLocker too often allocating 2417775 words.

Is there a way to prevent this error from happening?

[1] git clone --quiet https://<USER>@<GERRIT_URL>/a/<PROJECT> --single-branch --branch master

Thank you

Mark Bekhet

Matthias Sohn

unread,

Aug 15, 2022, 9:14:37 AM8/15/22

to Mark Bekhet, Repo and Gerrit Discussion

On Mon, Aug 15, 2022 at 2:41 PM Mark Bekhet <bekhe...@gmail.com> wrote:

Hi,
We encounter a problem when we try to execute the command in [1] on a large repository and we have pack.deltacompression = on. This happens on Gerrit 3.4.5. We have the following configuration on the server:

[core]
packedGitOpenFiles = 8192
packedGitLimit = 48g
packedGitWindowSize = 64k
streamFileThreshold = 2048m
deltaBaseCacheLimit = 100m
repositoryCacheExpireAfter = 10min
[pack]
bigFileThreshold = 20m
indexVersion = 2
deltacompression = on
threads = 0

When I set deltacompression to off, we don't have a problem and we are able to clone the single branch of the repository.
The server has 18 CPUs, 197902240 kB Total memory, 976688 kB free memory and 85742796 kB available memory.

I reproduced the behaviour locally and it seems that threads are trying to write in the memory but they fail to find free memory to complete the packedGitLimit , I get the following error which I found also on the server:

What's the maximum heap size of the JVM running Gerrit ?

Looks like you need a larger heap size.

Which Java gc implementation are you using on which Java version ?

-Matthias

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/8566257c-64ab-4a69-b41e-7a21e0a7a020n%40googlegroups.com.

Mark Bekhet

unread,

Aug 15, 2022, 9:41:46 AM8/15/22

to Repo and Gerrit Discussion

Hi Matthias,

Thanks for the quick reply. Please find my answers inline.

On Monday, August 15, 2022 at 9:14:37 AM UTC-4 Matthias Sohn wrote:

On Mon, Aug 15, 2022 at 2:41 PM Mark Bekhet <bekhe...@gmail.com> wrote:
Hi,
We encounter a problem when we try to execute the command in [1] on a large repository and we have pack.deltacompression = on. This happens on Gerrit 3.4.5. We have the following configuration on the server:

[core]
packedGitOpenFiles = 8192
packedGitLimit = 48g
packedGitWindowSize = 64k
streamFileThreshold = 2048m
deltaBaseCacheLimit = 100m
repositoryCacheExpireAfter = 10min
[pack]
bigFileThreshold = 20m
indexVersion = 2
deltacompression = on
threads = 0

When I set deltacompression to off, we don't have a problem and we are able to clone the single branch of the repository.
The server has 18 CPUs, 197902240 kB Total memory, 976688 kB free memory and 85742796 kB available memory.

I reproduced the behaviour locally and it seems that threads are trying to write in the memory but they fail to find free memory to complete the packedGitLimit , I get the following error which I found also on the server:

What's the maximum heap size of the JVM running Gerrit ?
Looks like you need a larger heap size.

Which Java gc implementation are you using on which Java version ?

In container section in gerrit config, we specify the following:

javaOptions = -Xmx128g

javaOptions = -XX:+UseG1GC

javaOptions = -XX:ParallelGCThreads=24j

For the java version, it is:

openjdk version "11.0.13" 2021-10-19 LTS
OpenJDK Runtime Environment Zulu11.52+13-CA (build 11.0.13+8-LTS)
OpenJDK 64-Bit Server VM Zulu11.52+13-CA (build 11.0.13+8-LTS, mixed mode)

Please let me know if that answer your questions.

Matthias Sohn

unread,

Aug 15, 2022, 9:57:46 AM8/15/22

to Mark Bekhet, Repo and Gerrit Discussion

On Mon, Aug 15, 2022 at 3:41 PM Mark Bekhet <bekhe...@gmail.com> wrote:

Hi Matthias,
Thanks for the quick reply. Please find my answers inline.

On Monday, August 15, 2022 at 9:14:37 AM UTC-4 Matthias Sohn wrote:
On Mon, Aug 15, 2022 at 2:41 PM Mark Bekhet <bekhe...@gmail.com> wrote:
Hi,
We encounter a problem when we try to execute the command in [1] on a large repository and we have pack.deltacompression = on. This happens on Gerrit 3.4.5. We have the following configuration on the server:

[core]
packedGitOpenFiles = 8192
packedGitLimit = 48g

JGit caps this to 1/4 of maximum heap size, for your configuration that's 32 GB.

You may consider to set core.packedGitUseStrongRefs=true to improve efficiency of this cache.

See https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#core.packedGitUseStrongRefs

packedGitWindowSize = 64k
streamFileThreshold = 2048m
deltaBaseCacheLimit = 100m
repositoryCacheExpireAfter = 10min
[pack]
bigFileThreshold = 20m
indexVersion = 2
deltacompression = on
threads = 0

When I set deltacompression to off, we don't have a problem and we are able to clone the single branch of the repository.
The server has 18 CPUs, 197902240 kB Total memory, 976688 kB free memory and 85742796 kB available memory.

I reproduced the behaviour locally and it seems that threads are trying to write in the memory but they fail to find free memory to complete the packedGitLimit , I get the following error which I found also on the server:

What's the maximum heap size of the JVM running Gerrit ?
Looks like you need a larger heap size.
Which Java gc implementation are you using on which Java version ?

In container section in gerrit config, we specify the following:
javaOptions = -Xmx128g
javaOptions = -XX:+UseG1GC

Try parallelGC, it shows longer stop-the-world-pauses but has higher throughput.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/f5b17535-80e5-4c4b-a08f-8e2645e6e15bn%40googlegroups.com.

Mark Bekhet

unread,

Aug 15, 2022, 10:05:00 AM8/15/22

to Repo and Gerrit Discussion

Hi Matthias,

Thanks for the suggestion, I will try it today and update the thread with the results.

Mark Bekhet

unread,

Aug 16, 2022, 8:13:59 AM8/16/22

to Repo and Gerrit Discussion

Hi Matthias,

I tried the suggested changes, but still the clone is failing.

Nguyen Tuan Khang Phan

unread,

Sep 20, 2022, 10:53:35 AM9/20/22

to Repo and Gerrit Discussion

Hi Matthias,

Can you advice on what should we try next? The clone command still fails with tuning you suggested.

Matthias Sohn

unread,

Sep 20, 2022, 3:16:21 PM9/20/22

to Nguyen Tuan Khang Phan, Repo and Gerrit Discussion

On Tue, Sep 20, 2022 at 4:53 PM Nguyen Tuan Khang Phan <phan....@gmail.com> wrote:

Hi Matthias,

Can you advice on what should we try next? The clone command still fails with tuning you suggested.

I looked at the first stack trace more carefully. It looks like this repo hits an internal buffer size limit

when pack.deltacompression = on so that it cannot be processed. Not sure why it doesn't hit this

exception when deltacompression is off.

The exception you observed was introduced in this change

https://git.eclipse.org/r/c/jgit/jgit/+/25449

when removing the streaming code which was used in such cases previously. This streaming code was

removed since it was too slow to be useful. In that jgit version this exception was thrown in line 800 of

https://git.eclipse.org/r/c/jgit/jgit/+/25449/4/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/PackFile.java

Could you run git-sizer on this repo to check if it contains huge objects ?

https://github.com/github/git-sizer

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/388f5be8-ffee-4856-997a-b17bf9561cdbn%40googlegroups.com.

Nguyen Tuan Khang Phan

unread,

Oct 17, 2022, 4:50:36 PM10/17/22

to Repo and Gerrit Discussion

Could you run git-sizer on this repo to check if it contains huge objects ?
https://github.com/github/git-sizer

Processing blobs: 4747221
Processing trees: 13598036
Processing commits: 7156097
Matching commits to trees: 7156097
Processing annotated tags: 381
Processing references: 1553188
| Name | Value | Level of concern |
| ---------------------------- | --------- | ------------------------------ |
| Overall repository size | | |
| * Commits | | |
| * Count | 7.16 M | ************** |
| * Total size | 2.33 GiB | ********** |
| * Trees | | |
| * Count | 13.6 M | ********* |
| * Total size | 14.6 GiB | ******* |
| * Total tree entries | 410 M | ******** |
| * Blobs | | |
| * Count | 4.75 M | *** |
| * Total size | 212 GiB | ********************** |
| * Annotated tags | | |
| * Count | 381 | |
| * References | | |
| * Count | 1.55 M | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
| * Branches | 28.2 k | * |
| * Tags | 381 | |
| * Changeset refs | 1.41 M | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
| * Git notes | 112 k | **** |
| * Other | 1.07 k | |
| | | |
| Biggest objects | | |
| * Commits | | |
| * Maximum size [1] | 547 KiB | *********** |
| * Maximum parents [2] | 4 | |
| * Trees | | |
| * Maximum entries [3] | 2.92 k | ** |
| * Blobs | | |
| * Maximum size [4] | 226 MiB | *********************** |
| | | |
| History structure | | |
| * Maximum history depth | 474 k | |
| * Maximum tag depth [5] | 1 | |
| | | |
| Biggest checkouts | | |
| * Number of directories [6] | 60.4 k | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
| * Maximum path depth [7] | 17 | * |
| * Maximum path length [8] | 289 B | ** |
| * Number of files [6] | 164 k | *** |
| * Total size of files [9] | 2.18 GiB | ** |
| * Number of symlinks [10] | 1.08 k | |
| * Number of submodules [11] | 46 | |

So, another issue popped up when we disabled delta compression. The replication fails with read timeout.

Reply all

Reply to author

Forward