High Availability setup, cloning issues on replica.

elzoc...@gmail.com

unread,

May 24, 2021, 10:14:45 AM5/24/21

to Repo and Gerrit Discussion

Hi all,

Status: Gerrit 3.3.4, all data on a shared AWS EFS.

1x Master

1x Replica

On the EFS, both master and replica have their own home folder, mounted at /gerrit in both VMs. Then we have shared which is a common folder to both, it will have the websessions and is mounted in /gerrit/shared. And last the git folder, also common to both VMs, mounted in /gerrit/shared/git, but mounted as read-only on the replica, and rw on master.

Replica is kept up-to-date via the HighAvailability plugin.

Only gerritmaster HA is configured to talk with ha creds.

gerritmaster:~$ cat etc/high-availability.config

[main]

sharedDirectory = /gerrit/shared

[autoReindex]

enabled = false

[peerInfo]

strategy = static

[peerInfo "static"]

url = http://gerritreplica:8081/r

[http]

user = hauser

password = password

gerritreplica:~$ cat etc/high-availability.config

[main]

sharedDirectory = /gerrit/shared

[autoReindex]

enabled = false

[peerInfo]

strategy = static

A month or so ago, on one of the threads I saw a comment saying that since Gerrit 3.3 with HA plugin, both nodes could be serving clones, so last week I decided to test it for our local traffic.

On the ELB, I started redirecting traffic: if host is my IP and git-upload-pack in path or query then send to replica.

I had to add a retry to some of our gerrit-triggered jenkins jobs because I saw some errors like the following:

git fetch origin refs/changes/72/56872/6

fatal: couldn't find remote ref refs/changes/72/56872/6

an error '128' has occurred in the scanning script

Changes had not been re-indexed yet on the replica I think, but after that retry no problem.

During the WE though, some jobs started failing with various errors:

repo sync -d -c -q --no-tags

component/test:

fatal: early EOF

fatal: unpack-objects failed

component/test:

fatal: early EOF

fatal: unpack-objects failed

error: Cannot checkout component/test: ManifestInvalidRevisionError: revision branch1 in component/test not found

error: in `sync -d -c -q --no-tags`: revision branch1 in component/test not found

This morning, one of the component started failing 100% of the time with:

git clone "https://example.com/r/component/common"

Cloning into 'common'...

remote: Counting objects: 3, done

fatal: The remote end hung up unexpectedly

fatal: early EOFs: 75% (790/1053)

fatal: index-pack failed

So I had to remove the ELB redirection to get everything back to gerritmaster.

That last error, everytime invoked the clone, I had the following trace on the replica server error_log:

[2021-05-24T13:10:49.968+01:00] [HTTP POST /r/component/common/git-upload-pack (Cedric from 10.1.0.78)] WARN org.eclipse.jetty.server.handler.ContextHandler.r : Internal error during upload-pack from /gerrit/shared/git/component/common.git

java.io.IOException: Stale file handle

at java.base/java.io.RandomAccessFile.readBytes(Native Method)

at java.base/java.io.RandomAccessFile.read(RandomAccessFile.java:406)

at java.base/java.io.RandomAccessFile.readFully(RandomAccessFile.java:470)

at org.eclipse.jgit.internal.storage.file.PackFile.read(PackFile.java:725)

at org.eclipse.jgit.internal.storage.file.WindowCache.load(WindowCache.java:516)

at org.eclipse.jgit.internal.storage.file.WindowCache.getOrLoad(WindowCache.java:603)

at org.eclipse.jgit.internal.storage.file.WindowCache.get(WindowCache.java:386)

at org.eclipse.jgit.internal.storage.file.WindowCursor.pin(WindowCursor.java:327)

at org.eclipse.jgit.internal.storage.file.WindowCursor.copyPackAsIs(WindowCursor.java:247)

at org.eclipse.jgit.internal.storage.file.PackFile.copyPackAsIs(PackFile.java:391)

at org.eclipse.jgit.internal.storage.file.LocalCachedPack.copyAsIs(LocalCachedPack.java:53)

at org.eclipse.jgit.internal.storage.file.WindowCursor.copyPackAsIs(WindowCursor.java:239)

at org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1242)

at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2312)

at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2143)

at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:1077)

at org.eclipse.jgit.transport.UploadPack.uploadWithExceptionPropagation(UploadPack.java:834)

at org.eclipse.jgit.http.server.UploadPackServlet.lambda$doPost$0(UploadPackServlet.java:184)

at org.eclipse.jgit.http.server.UploadPackServlet.defaultUploadPackHandler(UploadPackServlet.java:207)

at org.eclipse.jgit.http.server.UploadPackServlet.doPost(UploadPackServlet.java:201)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)

at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:211)

at com.google.gerrit.httpd.GitOverHttpServlet$UploadFilter.doFilter(GitOverHttpServlet.java:435)

at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:209)

at org.eclipse.jgit.http.server.UploadPackServlet$Factory.doFilter(UploadPackServlet.java:137)

at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:209)

at org.eclipse.jgit.http.server.RepositoryFilter.doFilter(RepositoryFilter.java:112)

at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:209)

at org.eclipse.jgit.http.server.NoCacheFilter.doFilter(NoCacheFilter.java:53)

at org.eclipse.jgit.http.server.glue.UrlPipeline$Chain.doFilter(UrlPipeline.java:209)

at org.eclipse.jgit.http.server.glue.UrlPipeline.service(UrlPipeline.java:188)

at org.eclipse.jgit.http.server.glue.SuffixPipeline.service(SuffixPipeline.java:70)

at org.eclipse.jgit.http.server.glue.MetaFilter.doFilter(MetaFilter.java:150)

at org.eclipse.jgit.http.server.glue.MetaServlet.service(MetaServlet.java:109)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)

at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:390)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:72)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:120)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.SetThreadNameFilter.doFilter(SetThreadNameFilter.java:62)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:139)

at com.googlesource.gerrit.plugins.readonly.ReadOnly.doFilter(ReadOnly.java:79)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)

at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:66)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)

at com.google.gerrit.httpd.AllowRenderInFrameFilter.doFilter(AllowRenderInFrameFilter.java:56)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:141)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:60)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.pgm.http.jetty.ProjectQoSFilter.doFilter(ProjectQoSFilter.java:182)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.ProjectBasicAuthFilter.doFilter(ProjectBasicAuthFilter.java:105)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:57)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:64)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)

at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at org.eclipse.jetty.server.Server.handleAsync(Server.java:559)

at org.eclipse.jetty.server.HttpChannel.lambda$handle$2(HttpChannel.java:396)

at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:396)

at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:340)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)

at java.base/java.lang.Thread.run(Thread.java:829)

Note: A similar error appears in the error_log, but I don't think it caused an error on the git client side (I can't be sure because there were lots of them at that time, for only a few failures)

[2021-05-23T03:26:30.148+01:00] [HTTP POST /r/docs/git-upload-pack (jenkins from xx.xx.xx.xxx)] WARN org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /gerrit/shared/git/docs.git/objects/pack/pack-226d13 165970d2f9c33aaeeb8d14cef05494ab62.pack handle is stale, removing it from pack list

9110 java.io.IOException: Stale file handle