Gerrit 3.6.1 connection timeout - 504 issues

[Ps]

We recently migrated gerrit from 2.16.16 to 3.6.1 (all looks good including metadata and review history).

Initially we migrated staging env and UAT done with minimal load - all went well.

Based on Staging env results - we migrated production env.

We are using standalone linux server with 32 vCPU and 128GB memory.  We are using below gerrit.conf file. 

Our environment looks like - 

Jenkins CI uses all - http connections 

all developers (500) uses both - ssh & https (from both windows and linux) to clone/fetch/pull 

[auth]

type = LDAP

userNameToLowerCase = True

gitBasicAuth = True

gitBasicAuthPolicy = HTTP_LDAP

[cache]

directory = cache

[container]

user = gerrit

javaHome = /etc/alternatives/jre_openjdk

startupTimeout = 200

heapLimit = 85g

javaOptions = "-Dflogger.backend_factory=com.google.common.flogger.backend.log4j.Log4jBackendFactory#getInstance"

javaOptions = "-Dflogger.logging_context=com.google.gerrit.server.logging.LoggingContext#getInstance"

[gerrit]

basePath = git

canonicalWebUrl = https://gerrit.prod.com

serverId = a5647ebf-39c1-493f-64df-832452ea

canLoadInIFrame = true

xframeOption = Allow

[httpd]

listenUrl = proxy-https://*:8080/

maxThreads = 99

maxQueued = 999

[index]

type = lucene

[change]

mergeabilityComputationBehavior = API_REF_UPDATED_AND_CHANGE_REINDEX

cumulativeCommentSizeLimit = 10m

maxComments = 40000

maxUpdates = 10000

[hooks]

executorThreads = 12

[sshd]

listenAddress = *:29418

commandStartThreads = 3

maxConnectionsPerUser = 0

batchThreads = 0

[cache "sshkeys"]

maxAge = 300

[core]

packedGitLimit = 4g

packedGitWindowSize = 16k

packedGitOpenFiles = 2048

[database]

poolLimit = 150

connectionPool = True

poolMinIdle = 16

[user]

email = ger...@gerrit.prod.in

[receive]

enableSignedPush = false

[sendemail]

smtpServer = localhost

enable = false

[plugins]

allowRemoteAdmin = true

[gc]

startTime = 11:00

interval = 1 days

[commitmessage]

maxSubjectLength = 72

maxLineLength = 72

[receive]

enableSignedPush = false

[cache "web_sessions"]

maxAge = 7d 

Thanks in Advance - Srinivasa

[Matthias Sohn]

What's your question ?

-Matthias 

[Ps]

Hello Matthias,

Thanks for replying - 

Endusers are frequently getting with below 504 errors !! 

my question is - Should I do any changes in gerrit.config to overcome below issue ?

unable to access  svc-jen...@prod.com The requested URL returned error: 504

Thanks, Srinivasa

[Luca Milanesio]

On 27 Sep 2022, at 11:25, Ps <psre...@gmail.com> wrote:

Hello Matthias,

Thanks for replying - 

Endusers are frequently getting with below 504 errors !! 

my question is - Should I do any changes in gerrit.config to overcome below issue ?

unable to access  svc-jen...@prod.com The requested URL returned error: 504

HTTP 504 is possibly returned by your reverse proxy and is a gateway timeout. It could be either the connections taking too long or you have exhausted the maximum number of connections or both.

Are you collecting the Gerrit metrics? How does the HTTP connections trend look like?

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/7535806c-7b38-4567-83a4-328be296dec9n%40googlegroups.com.

[Ps]

Hi Luca,

I'm collecting http connections by a simple command -  HTTP Connections: $(netstat -latuen | egrep '8080'  | wc -l

for every 5 seconds - on average http connection count is 600 and sometime it is reaching 1400+ as well.

I'm using HAproxy as reverse proxy solution. below is my backend configuration.

backend gerrit

    balance     roundrobin

    mode        http

    cookie SERVERID insert indirect nocache

    option httpchk HEAD /config/server/version

    server      localhost localhost:8080 maxconn 200 check cookie localhost

    timeout check 30s

    timeout client 30s

    timeout connect 20s

[Luca Milanesio]

On 27 Sep 2022, at 12:14, Ps <psre...@gmail.com> wrote:

Hi Luca,

I'm collecting http connections by a simple command -  HTTP Connections: $(netstat -latuen | egrep '8080'  | wc -l

for every 5 seconds - on average http connection count is 600 and sometime it is reaching 1400+ as well.

From your gerrit.config you have just 99 threads:

[httpd]

listenUrl = proxy-https://*:8080/

maxThreads = 99

How was typically the figure in v2.16.x?

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/16798b94-501b-4082-81ba-f2fc78b8acc4n%40googlegroups.com.

[Ps]

Luca,

maxThreads = 64 in 2.16.16 version - it looks good at that time - same team / same usage - but now we are getting this 504 timeout issues. I'm not seeing CPU utilisation not more than 40% and memory not crossing heap 85G. Shall I increase the value to higher values ??

Thanks, Srinivasa

[Luca Milanesio]

On 27 Sep 2022, at 13:21, Ps <psre...@gmail.com> wrote:

Luca,

maxThreads = 64 in 2.16.16 version - it looks good at that time - same team / same usage - but now we are getting this 504 timeout issues. I'm not seeing CPU utilisation not more than 40% and memory not crossing heap 85G.

I’d suggest to get JVM thread dumps and see why the incoming HTTP calls seem to be stuck.

Did you have this problem in staging when testing the migration? v2.16 to v3.6.1 is quite a big migration jump.

See my talk about Gerrit migrations at [1].

Also, you should start collection more comprehensive metrics, using tools such as JavaMelody [2] and the Prometheus reporter [3]. Just counting the number of HTTP connections at TCP level may not be enough.

HTH

Luca.

[1] https://youtu.be/uAUw1hHiVC4

[2] https://gerrit.googlesource.com/plugins/javamelody/

[3] https://gerrit.googlesource.com/plugins/metrics-reporter-prometheus/

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/356278a2-aad3-42da-a135-4cd99c54b5b5n%40googlegroups.com.

[Sven Selberg]

On Tuesday, September 27, 2022 at 2:32:41 PM UTC+2 lucamilanesio wrote:

On 27 Sep 2022, at 13:21, Ps <psre...@gmail.com> wrote:

Luca,

maxThreads = 64 in 2.16.16 version - it looks good at that time - same team / same usage - but now we are getting this 504 timeout issues. I'm not seeing CPU utilisation not more than 40% and memory not crossing heap 85G.

I’d suggest to get JVM thread dumps and see why the incoming HTTP calls seem to be stuck.

Just a thought.
We have at times experienced running out of HTTP threads due to LDAP issues (long respons-times, shaky connections, etc.).
It might be worth looking into, if for nothing else to exclude it as a potential culprit.

A JVM thread dump should reveal this.

[Luca Milanesio]

On 27 Sep 2022, at 13:37, Sven Selberg <sven.s...@axis.com> wrote:

On Tuesday, September 27, 2022 at 2:32:41 PM UTC+2 lucamilanesio wrote:

On 27 Sep 2022, at 13:21, Ps <psre...@gmail.com> wrote:

Luca,

maxThreads = 64 in 2.16.16 version - it looks good at that time - same team / same usage - but now we are getting this 504 timeout issues. I'm not seeing CPU utilisation not more than 40% and memory not crossing heap 85G.

I’d suggest to get JVM thread dumps and see why the incoming HTTP calls seem to be stuck.

Just a thought.
We have at times experienced running out of HTTP threads due to LDAP issues (long respons-times, shaky connections, etc.).
It might be worth looking into, if for nothing else to exclude it as a potential culprit.

It could be, but that won’t explain why it was not happening with Gerrit v2.16 though :-(

My suggestion is always to get the data first and analyse it afterwards.

Without proper data is hard to guess what it would be.

A JVM thread dump should reveal this.

Yep, that would be the bare minimum to start digging.

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/715ab512-dcdd-4e3f-9015-70d8ea65dcedn%40googlegroups.com.

[Ps]

Hello Luca & Sven Selberg,

Thanks for your inputs - 

let me work on to get JVM thread dump (not sure how to), but I will dig in the threads to get the details.

[Luca Milanesio]

> On 27 Sep 2022, at 13:44, Ps <psre...@gmail.com> wrote:
>
> Hello Luca & Sven Selberg,
>
> Thanks for your inputs -
>
> let me work on to get JVM thread dump (not sure how to),

The following command would do the job:
jstack <Gerrit pid>

Example:
jstack 1234

HTH

Luca.

> To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/45a6697d-3143-425c-a334-a831dff797afn%40googlegroups.com.

[Ps]

I'm  maybe wrong - but I didn't find  jstack open-jdk-11 rp, - I'm trying to get command on rhel.

 rpm -ql java-11-openjdk-headless-11.0.8.10-1.el7.x86_64 | grep -i jstack

Thanks, Srinivasa

[Luca Milanesio]

On 27 Sep 2022, at 14:04, Ps <psre...@gmail.com> wrote:

I'm  maybe wrong - but I didn't find  jstack open-jdk-11 rp, - I'm trying to get command on rhel.

 rpm -ql java-11-openjdk-headless-11.0.8.10-1.el7.x86_64 | grep -i jstack

I believe you should find it in the standard or develop package of the JDK.

A simple search on stack-overflow or Google should give you all directions to get it installed on your box.

HTH

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/667868b7-daa1-4ea3-acc4-35a6347e8496n%40googlegroups.com.

[Ps]

Thanks Luca,

I think I have taken dump correctly !! Please find the attached JVM thread dump details.

[Luca Milanesio]

On 27 Sep 2022, at 14:28, Ps <psre...@gmail.com> wrote:

Thanks Luca,

I think I have taken dump correctly !! Please find the attached JVM thread dump details.

It looks like all threads a stuck checking for the visible changes: 

at com.google.gerrit.server.permissions.GitVisibleChangeFilter.getVisibleChanges(GitVisibleChangeFilter.java:77)

Do you allow visibility of all refs to all users authorised to access the project?

If not, Gerrit checks for the status of each individual change and hide the refs associated to the private ones or the ones with pending edits.

The computation of the status of each change could be significant and is most likely the root cause of the slowdown of the Gerrit node.

Were you on NoteDb on Gerrit v2.16 or ReviewDb?

On Gerrit v2.16 the fetching of the status of a change was resolved with a DB query, whilst requires a Git access to the change notes with NoteDb.

HTH

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/74b36d20-662f-4dc2-a693-e8d7ca3f4b8bn%40googlegroups.com.
<thread_dump>

[Ps]

Hello Luca,

We will not allow visibility of all refs to all users. 

We were on reviewdb - and before migration we migrated mandatory to notedb. 

Thanks, Srinivasa

[Luca Milanesio]

On 27 Sep 2022, at 14:45, Ps <psre...@gmail.com> wrote:

Hello Luca,

We will not allow visibility of all refs to all users. 

We were on reviewdb - and before migration we migrated mandatory to notedb. 

That explains what happened: before the check of the change status was quick because of a simple DBMS lookup, now it is way more expensive and therefore you’re hit with the slowdowns.

You’ve got two options:

1. Increase visibility (if possible) and allow to avoid the expensive change status check, that would bring the performance back to what it was before

2. Enable cache-ability of the change notes status using a high-performance cache like cache-chronicle map (see [1]) and then rely on caching for the change status

The 1. would be a lot easier and no footprint on the system. The 2. requires the setup of the libModule at [1] and tune the change_notes cache. It may also use a significant amount of disk-space.

HTH

Luca.

[1] https://gerrit.googlesource.com/modules/cache-chroniclemap/  

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/421307e1-d703-4a54-98f2-1cff8c83b786n%40googlegroups.com.

[Ps]

Thank yo very much - Luca,

I will check with team and try to increase the visibility changes. This looks to be a feasible option here.

Thanks once again for your time.

Thanks, Srinivasa

[mail...@gmail.com]

Today we increased visibility - no luck. I installed javamelody as well. Let me monitor - it may give us more insights related to http (bottleneck)

We are getting 504 timeout errors from jenkins ci jobs - that too https connections only.

hudson.plugins.git.GitException: Command "git fetch --no-tags --progress https://svc-fwtoolsjenkinsci%40pro...@gerrit.prod.com/a/BCC +refs/heads/*:refs/remotes/origin/*" returned status code 128: 08:49:34 stdout: 

08:49:34 stderr: error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504 Gateway Time-out 

08:49:34 fatal: The remote end hung up unexpectedly

Thanks, Srinivasa

[mail...@gmail.com]

some more error logs related  git-upload-pack internal errors !!

[2022-09-28T02:28:59.325Z] [HTTP POST /a/CCB/git-upload-pack (svc-fwtool...@prod.com from 10.248.202.79)] ERROR com.google.gerrit.httpd.GitOverHttpServlet.GerritUploadPackErrorHandler : Internal error during upload-pack from Repository[/var/gerrit/git/CCB.git] [CONTEXT project="CCB" request="GIT_UPLOAD" ]

org.eclipse.jetty.io.EofException: Closed

   at org.eclipse.jetty.server.HttpOutput.checkWritable(HttpOutput.java:733)

   at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:757)

   at net.bull.javamelody.internal.web.FilterServletOutputStream.write(FilterServletOutputStream.java:88)

   at net.bull.javamelody.internal.web.CounterResponseStream.write(CounterResponseStream.java:81)

   at org.eclipse.jgit.util.TemporaryBuffer.writeTo(TemporaryBuffer.java:287)

   at org.eclipse.jgit.http.server.SmartOutputStream.writeResponse(SmartOutputStream.java:102)

   at org.eclipse.jgit.http.server.SmartOutputStream.close(SmartOutputStream.java:92)

   at org.eclipse.jgit.http.server.UploadPackServlet.lambda$doPost$0(UploadPackServlet.java:187)

   at com.google.gerrit.httpd.GitOverHttpServlet$GerritUploadPackErrorHandler.upload(GitOverHttpServlet.java:494)

   at org.eclipse.jgit.http.server.UploadPackServlet.doPost(UploadPackServlet.java:202)

   at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

   at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

   at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:212)

   at com.google.gerrit.httpd.GitOverHttpServlet$UploadFilter.doFilter(GitOverHttpServlet.java:462)

   at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:210)

   at org.eclipse.jgit.http.server.UploadPackServlet$Factory.doFilter(UploadPackServlet.java:137)

   at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:210)

   at org.eclipse.jgit.http.server.RepositoryFilter.doFilter(RepositoryFilter.java:112)

   at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:210)

   at org.eclipse.jgit.http.server.NoCacheFilter.doFilter(NoCacheFilter.java:53)

   at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:210)

   at org.eclipse.jgit.http.server.glue.UrlPipeline.service(UrlPipeline.java:189)

   at org.eclipse.jgit.http.server.glue.SuffixPipeline.service(SuffixPipeline.java:70)

   at org.eclipse.jgit.http.server.glue.MetaFilter.doFilter(MetaFilter.java:151)

   at org.eclipse.jgit.http.server.glue.MetaServlet.service(MetaServlet.java:109)

   at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

   at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)

   at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)

   at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)

   at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)

   at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)

   at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:393)

   at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

   at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92)

   at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

   at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:72)

   at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

   at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:120)

   at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

   at com.google.gerrit.httpd.RequireIdentifiedUserFilter.doFilter(RequireIdentifiedUserFilter.java:50)

   at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

   at com.google.gerrit.httpd.SetThreadNameFilter.doFilter(SetThreadNameFilter.java:62)

   at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

   at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:139)

   at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)

   at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)

   at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:66)

   at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)

   at com.google.gerrit.httpd.AllowRenderInFrameFilter.doFilter(AllowRenderInFrameFilter.java:52)

   at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)

   at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:141)

   at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

   at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:60)

   at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

   at com.google.gerrit.pgm.http.jetty.ProjectQoSFilter.doFilter(ProjectQoSFilter.java:183)

   at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

   at com.google.gerrit.httpd.ProjectBasicAuthFilter.doFilter(ProjectBasicAuthFilter.java:111)

   at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

   at com.google.gerrit.httpd.ProjectBasicAuthFilter.doFilter(ProjectBasicAuthFilter.java:111)

   at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

   at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:92)

   at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

   at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:64)

   at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

   at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)

   at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)

   at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

   at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

   at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)

   at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

   at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)

   at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

   at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)

   at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)

   at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)

   at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)

   at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)

   at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)

   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

   at org.eclipse.jetty.server.Server.handleAsync(Server.java:559)

   at org.eclipse.jetty.server.HttpChannel.lambda$handle$2(HttpChannel.java:396)

   at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)

   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:396)

   at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:340)

   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)

   at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905

Thanks, Srinivasa

[luca.mi...@gmail.com]

Sent from my iPhone

On 28 Sep 2022, at 02:33, mail...@gmail.com <mail...@gmail.com> wrote:



Today we increased visibility - no luck. I installed javamelody as well. Let me monitor - it may give us more insights related to http (bottleneck)

Have you taken fresh thread dumps?

Luca

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/52d208be-779d-4b7c-a0ec-ff257c64a86en%40googlegroups.com.

[Luca Milanesio]

On 28 Sep 2022, at 07:34, mail...@gmail.com <mail...@gmail.com> wrote:

Thanks Luca,
 

Yes - latest thread dump is attached. 

-- Srini

On Wednesday, 28 September, 2022 at 11:48:34 am UTC+5:30 lucamilanesio wrote:

Sent from my iPhone

On 28 Sep 2022, at 02:33, mail...@gmail.com <mail...@gmail.com> wrote:



Today we increased visibility - no luck. I installed javamelody as well. Let me monitor - it may give us more insights related to http (bottleneck)

Have you taken fresh thread dumps?

I don’t believe the ACL change you did was enough: I still see the full filtering going on.

"SSH git-upload-pack /CCB (svc-fwtool...@wdc.com)" #120 prio=1 os_prio=0 cpu=563972.44ms elapsed=30544.27s tid=0x00007f2a5000a000 nid=0xe651 waiting for monitor entry  [0x00007f2aa2ae9000]

   java.lang.Thread.State: BLOCKED (on object monitor)

at java.util.concurrent.ConcurrentHashMap.compute(java...@11.0.8/ConcurrentHashMap.java:1923)

- waiting to lock <0x00007f3dfd2a18b8> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)

at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2411)

at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2394)

at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)

at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:54)

at com.github.benmanes.caffeine.guava.CaffeinatedGuavaLoadingCache.get(CaffeinatedGuavaLoadingCache.java:59)

at com.google.gerrit.server.git.SearchingChangeCacheImpl.getChangeData(SearchingChangeCacheImpl.java:125)

at com.google.gerrit.server.permissions.GitVisibleChangeFilter.getVisibleChanges(GitVisibleChangeFilter.java:77)

at com.google.gerrit.server.permissions.DefaultRefFilter.filter(DefaultRefFilter.java:137)

at com.google.gerrit.server.permissions.ProjectControl$ForProjectImpl.filter(ProjectControl.java:417)

Have you granted READ permissions to ‘refs/*’ for the svc-fwtool...@wdc.com user?

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/b6a66cca-6af5-4112-a14f-59156a494cabn%40googlegroups.com.
<thread_dump_2809>

[Luca Milanesio]

Hold on, I found the culprit of the problem in this change [1] and I recall that it was also raised before: even if you give the ‘refs/*’ permission Gerrit would still go and check the visibility of *ALL changes*.

I guess you have a lot of changes on that repository, isn’t it?

So far, nobody has uploaded any fix to this problem: I’d recommend to downgrade to Gerrit v3.5.2 for the time being.

Luca.

[1] https://gerrit-review.googlesource.com/c/gerrit/+/336914

[mail...@gmail.com]

Yes  Luca,

We have lot of changes  on that repository. 

Seems - I have only option to revert 3.5.2 for time being. Hope we will come out of this 504 issues. 

Thanks for your time - I will consider this is final solution for this issue now and downgrade the version to 3.5.2

Thanks, Srinivasa

[Luca Milanesio]

On 28 Sep 2022, at 09:56, mail...@gmail.com <mail...@gmail.com> wrote:

Yes  Luca,

We have lot of changes  on that repository. 

Seems - I have only option to revert 3.5.2 for time being. Hope we will come out of this 504 issues. 

Thanks for your time - I will consider this is final solution for this issue now and downgrade the version to 3.5.2

We will fix it for sure on stable-3.6 (next forthcoming v3.6.2), I am in touch with the other maintainers to find a solution.

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/02e6c079-1460-4aa1-8f59-b0c1d92d52fen%40googlegroups.com.

[Srinivasa Reddy]

Thanks Luca,

I didn't find any rollback plan documentation for 3.6 - I will use the same manual steps mentioned for 3.5/3.4.

-- Srinivasa

You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/V7909VlGiEU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/C3D0C076-64FB-412D-A0D4-01901D02BF95%40gmail.com.

--

regards..,
SRenuVasu..

[lucamilanesio]

On Wednesday, September 28, 2022 at 11:37:43 AM UTC+1 Ps wrote:

Thanks Luca,

I didn't find any rollback plan documentation for 3.6 - I will use the same manual steps mentioned for 3.5/3.4.

Bear in mind that Submit Requirements are not supported in 3.5/3.4: if you were using them already, you should go back to Prolog rules.

I've created the issue [16284] for this problem in Gerrit v3.6.

I believe the fix is quite easy, it won't take long :-)

Luca.

[16284] https://bugs.chromium.org/p/gerrit/issues/detail?id=16284 

[Luca Milanesio]

On 28 Sep 2022, at 12:01, lucamilanesio <luca.mi...@gmail.com> wrote:

On Wednesday, September 28, 2022 at 11:37:43 AM UTC+1 Ps wrote:

Thanks Luca,

I didn't find any rollback plan documentation for 3.6 - I will use the same manual steps mentioned for 3.5/3.4.

Bear in mind that Submit Requirements are not supported in 3.5/3.4: if you were using them already, you should go back to Prolog rules.

I've created the issue [16284] for this problem in Gerrit v3.6.

I believe the fix is quite easy, it won't take long :-)

The fix is quite simple, see [346974] below. Hopefully it will be reviewed and merged quite quickly as it is quite small and self-contained.

Do you have the ability to build and test it on your environment?

Luca.

[346974] https://gerrit-review.googlesource.com/c/gerrit/+/346974

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/984194f1-fd79-46ac-8ba3-6218d86bedb8n%40googlegroups.com.

[pian...@gmail.com]

Hi, Luca -

We have Gerrit 3.6.1 source code on our server.  Can we just copy this fix (DefaultRefFilter.java) to there and rebulid the war?  Have there been no other changes to DefaultRefFilter.java since 3.6.1?

[Luca Milanesio]

On 28 Sep 2022, at 21:14, pian...@gmail.com <pian...@gmail.com> wrote:

Hi, Luca -

We have Gerrit 3.6.1 source code on our server.  Can we just copy this fix (DefaultRefFilter.java) to there and rebulid the war?  Have there been no other changes to DefaultRefFilter.java since 3.6.1?

You should checkout the change 346974’s SHA1 (cd1ebc6f824a0b8227e8fd5fbeba17e3de3b6f1f)

The build is green which means it has passed all tests: it should be safe for you to try.

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/d70636fc-fe4f-4eda-bcd3-6e542a0acc70n%40googlegroups.com.

[mail...@gmail.com]

Hello Luca,

Thank you very much on quick responses. Our colleagues Brent&Jon made local changes and able to deploy successfully in test environment.

Is there anyway to do quick check in test env ? We have done preliminary tests like (process check, log check, simple fetch, pull & push) 

 We can't do all traffic/load transfer to test env now.  

-- Srini

[mail...@gmail.com]

Thanks Luca,

We are able to deploy locally built war on production environment. All the 50X related are resolved now and gerrit services are working as expected.

Once again thank you very much for fixing the issue and sharing the fix with us.

Thanks, Srinivasa

[Luca Milanesio]

On 1 Oct 2022, at 03:31, mail...@gmail.com <mail...@gmail.com> wrote:

Thanks Luca,

We are able to deploy locally built war on production environment. All the 50X related are resolved now and gerrit services are working as expected.

Once again thank you very much for fixing the issue and sharing the fix with us.

This is great news, thanks for sharing it.

The change has been merged and it will be included in the next Gerrit v3.6.2 early next week.

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/89a685ad-40b5-42fc-9267-07dbe8a3f6cbn%40googlegroups.com.