HighAvailability flush-caches issue

56 views

Skip to first unread message

elzoc...@gmail.com

unread,

Sep 14, 2021, 8:14:51 AM9/14/21

to Repo and Gerrit Discussion

Hi all,

Some errors started to pop up this morning on our Gerrit servers, context:

2 Gerrit 3.3.6 servers, shared EFS, HA plugin.

Run Flush caches (projects) on Main (r/w) gerrit. HA plugin forwards the event to Secondary (ro) instance.

curl -n -X POST https://mygerrit.example.com/r/a/config/server/caches --header "Content-Type: application/json; charset=UTF-8" --data '{"operation": "FLUSH_ALL"}'

On the secondary I can see lots of the expected following flushes

10.1.0.120 [HTTP-78] - - [2021-09-14T12:46:38.018+01:00] "POST /r/plugins/high-availability/cache/projects HTTP/1.1" 204 - 1 - "Apache-HttpClient/4.5.2 (Java/11.0.11)"

10.1.0.120 [HTTP-135] - - [2021-09-14T12:46:38.018+01:00] "POST /r/plugins/high-availability/cache/projects HTTP/1.1" 204 - - - "Apache-HttpClient/4.5.2 (Java/11.0.11)"

until it fails:

10.1.0.120 [HTTP-163] - - [2021-09-14T12:46:38.051+01:00] "POST /r/plugins/high-availability/cache/projects HTTP/1.1" 500 12 16 - "Apache-HttpClient/4.5.2 (Java/11.0.11)"

after that first error (500) only have 500s, 1 every 30sec (retries).

Gerrit 3.3.6 (was upgraded from 3.3.5 last Sunday, but same issue was not seen yesterday).

HA plugin https://gerrit-ci.gerritforge.com/job/plugin-high-availability-bazel-stable-3.3/23/

On Main Gerrit (r/w):

[2021-09-14T13:03:38.266+01:00] [RestForwarderScheduler-4[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@322d307a[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@40934652[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@1ebf7556]]]] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : Executing invalidate cache projects projects towards http://gerritdev2-secondary:8081/r

[2021-09-14T13:03:38.271+01:00] [RestForwarderScheduler-4[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@322d307a[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@40934652[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@1ebf7556]]]] DEBUG com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder : Failed to invalidate cache projects projects on http://gerritdev2-secondary:8081/r [35/360]

com.ericsson.gerrit.plugins.highavailability.forwarder.rest.ForwardingException: Unable to invalidate cache projects projects : Server Error

at com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder$Request.tryOnce(RestForwarder.java:244)

at com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarder$Request.execute(RestForwarder.java:216)

at com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarderScheduler.lambda$execute$0(RestForwarderScheduler.java:92)

at com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarderScheduler$BooleanAsyncSupplier.run(RestForwarderScheduler.java:135)

at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:103)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)

at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

[2021-09-14T13:03:38.271+01:00] [RestForwarderScheduler-4[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@322d307a[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@40934652[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@1ebf7556]]]] WARN com.ericsson.gerrit.plugins.highavailability.forwarder.rest.RestForwarderScheduler : Rescheduling invalidate cache projects:projects => http://gerritdev2-secondary:8081/r (try #35) for retry after 30000 msec

On the Secondary (ro):

[2021-09-14T13:03:38.271+01:00] [HTTP-181] WARN org.eclipse.jetty.server.HttpChannel : /r/plugins/high-availability/cache/projects

java.lang.UnsupportedOperationException: JsonObject

at com.google.gson.JsonElement.getAsString(JsonElement.java:192)

at com.ericsson.gerrit.plugins.highavailability.forwarder.rest.CacheKeyJsonParser.fromJson(CacheKeyJsonParser.java:56)

at com.ericsson.gerrit.plugins.highavailability.forwarder.rest.CacheRestApiServlet.doPost(CacheRestApiServlet.java:55)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)

at com.google.gerrit.httpd.plugins.HttpPluginServlet.service(HttpPluginServlet.java:230)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)

at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:390)

at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92)

at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:72)

at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:120)

at com.google.gerrit.httpd.SetThreadNameFilter.doFilter(SetThreadNameFilter.java:62)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:139)

at com.googlesource.gerrit.plugins.readonly.ReadOnly.doFilter(ReadOnly.java:79)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)

at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:66)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)

at com.google.gerrit.httpd.AllowRenderInFrameFilter.doFilter(AllowRenderInFrameFilter.java:56)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:141)

at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:60)

at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:57)

at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:64)

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)

at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at org.eclipse.jetty.server.Server.handle(Server.java:516)

at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)

at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)

at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)

at java.base/java.lang.Thread.run(Thread.java:829)

[2021-09-14T13:03:38.272+01:00] [HTTP-181] ERROR com.google.gerrit.pgm.http.jetty.HiddenErrorHandler : Error in POST /r/plugins/high-availability/cache/projects