Upgrading from 3.6.8 to 3.7.0 fails and logs fill with sshd related errors

45 views
Skip to first unread message

Erik Thorsell

unread,
Jul 10, 2024, 7:06:12 PM (12 days ago) Jul 10
to Repo and Gerrit Discussion
Hi,

I'm on an upgrade train with a very old system.

System info
Debian 10
Java 11.0.7
Apache/2.4.38 (Debian) handling authentication via HTTP_LDAP

Background
When I started upgrading, Gerrit was at 3.5.0.1. I have slowly made my way through:

* 3.5.2
* 3.5.6
* 3.6.0 (before running init, I ran the copy-approvals command on 3.5.6)
* 3.6.2
* 3.6.6
* 3.6.8

I have run init and reindex for every upgrade. Only 3.6.0 gave me issues due to some lucene truncation change, but that was solved by 3.6.2. After every upgrade I have verified that Gerrit boots up properly, that I can browse the UI and that things seem to be in order.

Problem
After initialising and reindexing with 3.7.0 (no errors printed for the commands themselves) Gerrit does not boot up. 

I see the following in the logs:

[sshd-SshDaemon[6321e29b](port=22)-nio2-thread-1] WARN  com.google.gerrit.sshd.CachingPublicKeyAuthenticator : authenticate(jenkins@ServerSessionImpl[null@/127.0.0.1:58042]) failed (NoSuchMethodError) to consult delegate for ssh-rsa key=SHA256:tIvC+la9FRYdwh59U1qH5XSVYdujQU0s4i83MXKBL4Y: 'long com.sun.management.ThreadMXBean.getCurrentThreadAllocatedBytes()'
[sshd-SshDaemon[6321e29b](port=22)-nio2-thread-1] WARN  org.apache.sshd.server.session.ServerUserAuthService : handleUserAuthRequestMessage(ServerSessionImpl[null@/127.0.0.1:58042]) Failed (RuntimeSshException) to authenticate using factory method=publickey: null
[main] INFO  org.eclipse.jetty.server.AbstractConnector : Started ServerConnector@6b9b9cb9{HTTP/1.1, (http/1.1)}{0.0.0.0:8081}
[main] INFO  org.eclipse.jetty.server.Server : Started @12153ms
[main] INFO  com.google.gerrit.pgm.Daemon : Gerrit Code Review 3.7.0 ready

but after the "Gerrit Code Review 3.7.0 ready" the following messages are repeated forever (and the server never becomes available):

[sshd-SshDaemon[6321e29b](port=22)-nio2-thread-4] WARN  org.apache.sshd.server.session.ServerUserAuthService : handleUserAuthRequestMessage(ServerSessionImpl[null@/127.0.0.1:58058]) Failed (RuntimeSshException) to authenticate using factory method=publickey: null
[sshd-SshDaemon[6321e29b](port=22)-nio2-thread-5] WARN  com.google.gerrit.sshd.CachingPublicKeyAuthenticator : authenticate(jenkins@ServerSessionImpl[null@/127.0.0.1:58060]) failed (NoSuchMethodError) to consult delegate for ssh-rsa key=SHA256:tIvC+la9FRYdwh59U1qH5XSVYdujQU0s4i83MXKBL4Y: 'long com.sun.management.ThreadMXBean.getCurrentThreadAllocatedBytes()'

First of all, yes, we do use Jenkins, but to the best of my knowledge Gerrit does not attempt to authenticate towards Jenkins; so I don't quite understand why Jenkins is mentioned in the logs... Furthermore, I read somewhere -- which I have miss placed -- that someone had an issue upgrading to 3.7.0 and solved it by upgrading their Java version. However, according to the release notes the first mention of needing a specific Java 11 version is for 3.7.3:

  • Change 370835: Enforce minimum Java version to Java 11.0.10 or higher

    Since release 3.5 the minimum supported java version was raised to Java 11. Since Change 335625 the minimum Java version 11.0.10 is required.

To the best of my knowledge, Debian 10 does not package a higher Java version than 11.0.7 but before I start down that path I would love to hear if someone has any other ideas as to what could be the problem here.

Thank you in advance,
Erik

Clark Boylan

unread,
Jul 10, 2024, 7:25:38 PM (12 days ago) Jul 10
to Erik Thorsell, Repo and Gerrit Discussion
Do you mean Gerrit as a whole is never responsive or just that the
sshd never works?

>
> [sshd-SshDaemon[6321e29b](port=22)-nio2-thread-4] WARN org.apache.sshd.server.session.ServerUserAuthService : handleUserAuthRequestMessage(ServerSessionImpl[null@/127.0.0.1:58058]) Failed (RuntimeSshException) to authenticate using factory method=publickey: null
> [sshd-SshDaemon[6321e29b](port=22)-nio2-thread-5] WARN com.google.gerrit.sshd.CachingPublicKeyAuthenticator : authenticate(jenkins@ServerSessionImpl[null@/127.0.0.1:58060]) failed (NoSuchMethodError) to consult delegate for ssh-rsa key=SHA256:tIvC+la9FRYdwh59U1qH5XSVYdujQU0s4i83MXKBL4Y: 'long com.sun.management.ThreadMXBean.getCurrentThreadAllocatedBytes()'
>
> First of all, yes, we do use Jenkins, but to the best of my knowledge Gerrit does not attempt to authenticate towards Jenkins; so I don't quite understand why Jenkins is mentioned in the logs... Furthermore, I read somewhere -- which I have miss placed -- that someone had an issue upgrading to 3.7.0 and solved it by upgrading their Java version. However, according to the release notes the first mention of needing a specific Java 11 version is for 3.7.3:

I think these logs may indicate that your Jenkins is trying to
authenticate with Gerrit and there is a failure to handle the ssh-rsa
key. The failure to handle the ssh-rsa key reminds me of the problems
with openssh 8.8 trying to use sha2 hashed keys instead of sha1 hashed
keys and Gerrit's MINA SSHD didn't do kex extension negotiation to
address that. But Gerrit 3.7 should support that based on the release
notes, and it was working prior to the upgrade as well (which would've
been broken too if that was the issue). Might be worth trying a
different key type anyway to see if that helps resolve it.

>
> Change 370835: Enforce minimum Java version to Java 11.0.10 or higher
>
> Since release 3.5 the minimum supported java version was raised to Java 11. Since Change 335625 the minimum Java version 11.0.10 is required.
>
> To the best of my knowledge, Debian 10 does not package a higher Java version than 11.0.7 but before I start down that path I would love to hear if someone has any other ideas as to what could be the problem here.

You might also want to check your sshd_log contents since the problem
seems related to sshd. There may be more useful information in that
log file.

Erik Thorsell

unread,
Jul 11, 2024, 12:59:02 AM (11 days ago) Jul 11
to Repo and Gerrit Discussion
> Do you mean Gerrit as a whole is never responsive or just that the
> sshd never works? 

Gerrit as a whole never boots up. The logs I see by running: journalctl -fu gerrit simply prints out those two messages over, and over, and over. If I attempt to browse to the Gerrit UI the page loads but there's just a small text with "Server Error" at the top corner.

Concerning the sshd-issue, I turned off the Gerrit service and the Jenkins service, then I started Gerrit again. According to the log output Gerrit now starts, but when I attempt to browse to the GUI I still only see "Server Error" and I see this in the logs:

[2024-07-11T06:29:58.528+02:00] [HTTP-72] WARN  org.eclipse.jetty.server.HttpChannel : /gerrit/
java.lang.NoSuchMethodError: 'long com.sun.management.ThreadMXBean.getCurrentThreadAllocatedBytes()'
        at com.google.gerrit.metrics.proc.ThreadMXBeanSun.getCurrentThreadAllocatedBytes(ThreadMXBeanSun.java:38)
        at com.google.gerrit.httpd.RequestMetricsFilter$Context.<init>(RequestMetricsFilter.java:55)
        at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:89)
        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:64)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at java.base/java.lang.Thread.run(Thread.java:834)
[2024-07-11T06:29:58.538+02:00] [HTTP-72] ERROR com.google.gerrit.pgm.http.jetty.HiddenErrorHandler : Error in GET /gerrit/
java.lang.NoSuchMethodError: 'long com.sun.management.ThreadMXBean.getCurrentThreadAllocatedBytes()'
        at com.google.gerrit.metrics.proc.ThreadMXBeanSun.getCurrentThreadAllocatedBytes(ThreadMXBeanSun.java:38)
        at com.google.gerrit.httpd.RequestMetricsFilter$Context.<init>(RequestMetricsFilter.java:55)
        at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:89)
        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:64)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
        at java.base/java.lang.Thread.run(Thread.java:834)

I'm quite confident this did not appear before but was suppressed by the sshd error messages, but this was probably the underlying issue.

To me, the java.lang.NoSuchMethodError seems to be the culprit. Now when I have a new thing to search for, I was able to find this thread -> https://groups.google.com/g/repo-discuss/c/ANIrtqI5YVc and it does indeed seem to be a Java version issue.

I have now downgraded to 3.6.8 following the Downgrade instructions in the 3.7 release notes. The server is back up and running.

Two questions:

1. Who should one get in contact with concerning updating the release notes to say that the Java version-issue is present already in 3.7.0, not only in 3.7.2?
2. Is it possible to circumvent this or is upgrading Java the only way forward?

Best regards,
Erik

Matthias Sohn

unread,
Jul 11, 2024, 4:09:38 PM (11 days ago) Jul 11
to Erik Thorsell, Repo and Gerrit Discussion
On Thu, Jul 11, 2024 at 6:59 AM Erik Thorsell <thorse...@gmail.com> wrote:
> Do you mean Gerrit as a whole is never responsive or just that the
> sshd never works? 

Please avoid top posting on this list, use interleaved posting instead to simplify following the conversation across a mail thread.
Monitoring allocated memory was added in 3.4.0 (https://gerrit-review.googlesource.com/c/gerrit/+/301213).
It used com.sun.management.ThreadMXBean#getThreadAllocatedBytes(long) as a workaround since back
then the Java version used by the bazel version we used back then didn't support the method
com.sun.management.ThreadMXBean#getCurrentThreadAllocatedBytes.

This workaround was removed in https://gerrit-review.googlesource.com/c/gerrit/+/335625 when the bazel
version we used at this time used a newer Java version supporting this method.
Since then (3.7.0) at least Java 11.0.10 is required. Though it's recommended to use the latest bugfix release
of the Java version required by the respective Gerrit version. For Java 11 that's currently 11.0.23.

I'm quite confident this did not appear before but was suppressed by the sshd error messages, but this was probably the underlying issue.

To me, the java.lang.NoSuchMethodError seems to be the culprit. Now when I have a new thing to search for, I was able to find this thread -> https://groups.google.com/g/repo-discuss/c/ANIrtqI5YVc and it does indeed seem to be a Java version issue.

I have now downgraded to 3.6.8 following the Downgrade instructions in the 3.7 release notes. The server is back up and running.

Two questions:

1. Who should one get in contact with concerning updating the release notes to say that the Java version-issue is present already in 3.7.0, not only in 3.7.2?

Release notes are maintained in the homepage repository 
everybody can contribute improvements if you find errors or feel that
something is missing or not understandable.
 
2. Is it possible to circumvent this or is upgrading Java the only way forward?

Since gerrit 3.7.0 uses a method which is only available in Java 11.0.10 or higher you need to upgrade Java to at least this version.
 

Best regards,
Erik

On Thursday, July 11, 2024 at 1:25:38 AM UTC+2 Clark Boylan wrote:
On Wed, Jul 10, 2024 at 4:06 PM Erik Thorsell <thorse...@gmail.com> wrote:
>
> Hi,
>
> I'm on an upgrade train with a very old system.
>
> System info
> Debian 10
> Java 11.0.7
> Apache/2.4.38 (Debian) handling authentication via HTTP_LDAP
>
> Background
> When I started upgrading, Gerrit was at 3.5.0.1. I have slowly made my way through:
>
> * 3.5.2
> * 3.5.6
> * 3.6.0 (before running init, I ran the copy-approvals command on 3.5.6)
> * 3.6.2
> * 3.6.6
> * 3.6.8

There is no need to update through all the service releases, it's sufficient to upgrade from minor version to next minor version,
preferably using the highest available service release to ensure you benefit from all the bug fixes in the service releases (aka bugfix releases).
Read the release notes, at the minimum the section "Important notes".

Upgrade sequence you could follow when starting from 3.5.0.1:
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/6c8df352-d735-4e6b-9ce1-8f9e009819b6n%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages