java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space

108 views
Skip to first unread message

Guy Levkowitz

unread,
Apr 29, 2025, 8:14:08 AM4/29/25
to Repo and Gerrit Discussion
Hey 
We have Gerrit 3.10.1 server on Red Hat Enterprise Linux release 8.10 (Ootpa)

with memory defined:
 free -h
              total        used        free      shared  buff/cache   available
Mem:          125Gi        93Gi       884Mi       225Mi        31Gi        30Gi
Swap:          23Gi       3.9Gi        20Gi

and the 
cat ../etc/gerrit.config  | grep heap
    heapLimit = 84g

Still we get lots of :
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
Caused by: java.lang.OutOfMemoryError: Java heap space


which cause our server to flip 
and we need to restart the service/server from time to time 

what can be done more ? 

Guy



Guy Levkowitz

unread,
May 8, 2025, 7:11:25 AM5/8/25
to Repo and Gerrit Discussion
looks like that metrics plugin cause our server to get  Java heap space - and stuck our server, after disabling this plugin server works fine 
  https://gerrit.googlesource.com/plugins/metrics-reporter-prometheu

ב-יום שלישי, 29 באפריל 2025 בשעה 15:14:08 UTC+3, Guy Levkowitz כתב/ה:

Luca Milanesio

unread,
May 8, 2025, 7:13:59 AM5/8/25
to Repo and Gerrit Discussion, Luca Milanesio

On 8 May 2025, at 12:11, Guy Levkowitz <sil...@gmail.com> wrote:

looks like that metrics plugin cause our server to get  Java heap space - and stuck our server, after disabling this plugin server works fine 
  https://gerrit.googlesource.com/plugins/metrics-reporter-prometheu

How many projects do you have?

Luca.


ב-יום שלישי, 29 באפריל 2025 בשעה 15:14:08 UTC+3, Guy Levkowitz כתב/ה:
Hey 
We have Gerrit 3.10.1 server on Red Hat Enterprise Linux release 8.10 (Ootpa)

with memory defined:
 free -h
              total        used        free      shared  buff/cache   available
Mem:          125Gi        93Gi       884Mi       225Mi        31Gi        30Gi
Swap:          23Gi       3.9Gi        20Gi

and the 
cat ../etc/gerrit.config  | grep heap
    heapLimit = 84g

Still we get lots of :
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
Caused by: java.lang.OutOfMemoryError: Java heap space


which cause our server to flip 
and we need to restart the service/server from time to time 

what can be done more ? 

Guy




-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/40da4e02-4f07-4c99-8ab6-fedb5c1291een%40googlegroups.com.

Guy Levkowitz

unread,
May 8, 2025, 7:38:23 AM5/8/25
to Repo and Gerrit Discussion

We about : 3500 projects

ב-יום חמישי, 8 במאי 2025 בשעה 14:13:59 UTC+3, Luca Milanesio כתב/ה:

Guy Levkowitz

unread,
May 8, 2025, 7:39:50 AM5/8/25
to Repo and Gerrit Discussion
we have just this definitions in gerrit.conf file --> 
[plugin "metrics-reporter-prometheus"]
    enabled = true

ב-יום חמישי, 8 במאי 2025 בשעה 14:38:23 UTC+3, Guy Levkowitz כתב/ה:

Luca Milanesio

unread,
May 8, 2025, 7:41:53 AM5/8/25
to Repo and Gerrit Discussion, Luca Milanesio


> On 8 May 2025, at 12:38, Guy Levkowitz <sil...@gmail.com> wrote:
>
>
> We about : 3500 projects


Have you disabled the project-specific metrics?
What does your [plugin "metrics-reporter-prometheus”] section look like in gerrit.config?
Did you exclude any metric?

Luca.
> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/932cbb64-0227-4539-9527-6dd1ec18eaebn%40googlegroups.com.

Luca Milanesio

unread,
May 8, 2025, 7:43:12 AM5/8/25
to Repo and Gerrit Discussion, Luca Milanesio


> On 8 May 2025, at 12:39, Guy Levkowitz <sil...@gmail.com> wrote:
>
> we have just this definitions in gerrit.conf file -->
> [plugin "metrics-reporter-prometheus"]
> enabled = true

That explains the problem.
I’d suggest to enable the plugin again in staging, look at the project-specific metrics generated and exclude them.

There’s an excludeMetrics option to exclude metrics and metrics prefixes.

HTH

Luca.
> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/a7b1ebba-a2be-4ac1-b013-38ce67006b63n%40googlegroups.com.


Guy Levkowitz

unread,
May 8, 2025, 8:54:42 AM5/8/25
to Repo and Gerrit Discussion

So what is suggested is:
[plugin "metrics-reporter-prometheus"]
  enable = true
  prometheusPort = 9091
  prometheusPrefix = gerrit
  prometheusIncludeJvm = true
  excludeMetrics = gerrit_project_


✅ What stays available (examples)

  • Gerrit system metrics
    gerrit_http_server_request_count
    gerrit_git_receive_pack_latency

  • JVM stats (if enabled)
    gerrit_jvm_memory_used_bytes

  • Git-level metrics
    gerrit_git_upload_pack_latency

❌ What is removed

  • All project-specific metrics like:
    gerrit_project_os_web_merge_latency_seconds
    gerrit_project_core_push_count

ב-יום חמישי, 8 במאי 2025 בשעה 14:43:12 UTC+3, Luca Milanesio כתב/ה:
Reply all
Reply to author
Forward
0 new messages