Since upgrading to Gerrit 3.0.3 and java 11.0.5 from Gerrit 2.16.7 and 8.0.0_172 we've noticed a lot of threads hanging, which is eventually causing gerrit to blow out its open file limit due to the open file descriptors the thread hangs accumulate. The thread dumps aren't exactly clear what the threads are doing aside from they're clearly trying to read *something*. Does anyone have any ideas?
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/17f8a32e-411e-4029-abb3-196a5b10879a%40googlegroups.com.
On 16 Dec 2019, at 21:58, Matthias Sohn <matthi...@gmail.com> wrote:On Mon, Dec 16, 2019 at 10:06 PM Luke Engle <luke....@hpe.com> wrote:Since upgrading to Gerrit 3.0.3 and java 11.0.5 from Gerrit 2.16.7 and 8.0.0_172 we've noticed a lot of threads hanging, which is eventually causing gerrit to blow out its open file limit due to the open file descriptors the thread hangs accumulate. The thread dumps aren't exactly clear what the threads are doing aside from they're clearly trying to read *something*. Does anyone have any ideas?prerequisite is Java 8 [1] running on java 11 is not yet supported
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAKSZd3Rc7_A6g-Xq7DD3mwVAwyRvWeh7%2BbtxDPjz2rNjYKfUgw%40mail.gmail.com.
On 16 Dec 2019, at 21:58, Matthias Sohn <matthi...@gmail.com> wrote:On Mon, Dec 16, 2019 at 10:06 PM Luke Engle <luke....@hpe.com> wrote:Since upgrading to Gerrit 3.0.3 and java 11.0.5 from Gerrit 2.16.7 and 8.0.0_172 we've noticed a lot of threads hanging, which is eventually causing gerrit to blow out its open file limit due to the open file descriptors the thread hangs accumulate. The thread dumps aren't exactly clear what the threads are doing aside from they're clearly trying to read *something*. Does anyone have any ideas?prerequisite is Java 8 [1] running on java 11 is not yet supportedOne of the reasons why it is officially support, is the known issues of Java11 with LDAPS (see [2]).I believe the problem you are seeing is *exactly* on the SSL handshake hanging on LDAP, isn’t it? You nailed it :-)Luca.
On Mon, Dec 16, 2019 at 11:02 PM Luca Milanesio <luca.mi...@gmail.com> wrote:On 16 Dec 2019, at 21:58, Matthias Sohn <matthi...@gmail.com> wrote:On Mon, Dec 16, 2019 at 10:06 PM Luke Engle <luke....@hpe.com> wrote:Since upgrading to Gerrit 3.0.3 and java 11.0.5 from Gerrit 2.16.7 and 8.0.0_172 we've noticed a lot of threads hanging, which is eventually causing gerrit to blow out its open file limit due to the open file descriptors the thread hangs accumulate. The thread dumps aren't exactly clear what the threads are doing aside from they're clearly trying to read *something*. Does anyone have any ideas?prerequisite is Java 8 [1] running on java 11 is not yet supportedOne of the reasons why it is officially support, is the known issues of Java11 with LDAPS (see [2]).I believe the problem you are seeing is *exactly* on the SSL handshake hanging on LDAP, isn’t it? You nailed it :-)Luca.you can try the workaround given in [2] and add these Java system properties disabling TLSv1.3:Disabling TLS 1.3 completely with -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2
On 16 Dec 2019, at 22:02, Luca Milanesio <luca.mi...@gmail.com> wrote:On 16 Dec 2019, at 21:58, Matthias Sohn <matthi...@gmail.com> wrote:On Mon, Dec 16, 2019 at 10:06 PM Luke Engle <luke....@hpe.com> wrote:Since upgrading to Gerrit 3.0.3 and java 11.0.5 from Gerrit 2.16.7 and 8.0.0_172 we've noticed a lot of threads hanging, which is eventually causing gerrit to blow out its open file limit due to the open file descriptors the thread hangs accumulate. The thread dumps aren't exactly clear what the threads are doing aside from they're clearly trying to read *something*. Does anyone have any ideas?prerequisite is Java 8 [1] running on java 11 is not yet supportedOne of the reasons why it is officially support, is the known issues of Java11 with LDAPS (see [2]).
On 16 Dec 2019, at 21:58, Matthias Sohn <matthi...@gmail.com> wrote:
at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(java.ba...@11.0.5/SSLSocketInputRecord.java:68)at sun.security.ssl.SSLSocketImpl.readApplicationRecord(java.ba...@11.0.5/SSLSocketImpl.java:1104)
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(java...@11.0.5/SSLSocketImpl.java:823)- locked <0x00007efa8a0580c0> (a sun.security.ssl.SSLSocketImpl$AppInputStream)at java.io.BufferedInputStream.fill(java...@11.0.5/BufferedInputStream.java:252)at java.io.BufferedInputStream.read1(java...@11.0.5/BufferedInputStream.java:292)at java.io.BufferedInputStream.read(java...@11.0.5/BufferedInputStream.java:351)- locked <0x00007efa8a05a1c0> (a java.io.BufferedInputStream)at com.sun.jndi.ldap.Connection.run(java....@11.0.5/Connection.java:793)
at java.lang.Thread.run(java.ba...@11.0.5/Thread.java:834)Thanks,Luke--
--
To unsubscribe, email repo-discuss+unsub...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/17f8a32e-411e-4029-abb3-196a5b10879a%40googlegroups.com.
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.
On 16 Dec 2019, at 22:28, Luke Engle <luke....@hpe.com> wrote:Ah darn :( I saw the thread on [1] and wrongly assumed 'So, to clarify, Gerrit can be built with Java 11, and produce byte code major number 55, and also all tests are passing on Java 11' meant it was 'unofficially' supported with java 11. In fact, it *does* seem to work perfectly fine for everything except that ssl handshake hang. I even added ldap connect/read timeouts to the gerrit.config with no luck there as well.The interesting thing is that it doesn't cause any noticeable problems, even though since Friday we've had >1500 ldap ssl handshake threads that have hung. It seems like we could bump the ulimit to a very high number and bypass any open file limit issues for many weeks/months.So, reverting to java 8 is our only viable option?
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/d1ffed48-ca03-45ca-842c-419aa45debb5%40googlegroups.com.
On Mon, Dec 16, 2019 at 11:24 PM Matthias Sohn <matthi...@gmail.com> wrote:
at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(java.ba...@11.0.5/SSLSocketInputRecord.java:68)at sun.security.ssl.SSLSocketImpl.readApplicationRecord(java.ba...@11.0.5/SSLSocketImpl.java:1104)
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(java...@11.0.5/SSLSocketImpl.java:823)- locked <0x00007efa8a0580c0> (a sun.security.ssl.SSLSocketImpl$AppInputStream)at java.io.BufferedInputStream.fill(java...@11.0.5/BufferedInputStream.java:252)at java.io.BufferedInputStream.read1(java...@11.0.5/BufferedInputStream.java:292)at java.io.BufferedInputStream.read(java...@11.0.5/BufferedInputStream.java:351)- locked <0x00007efa8a05a1c0> (a java.io.BufferedInputStream)at com.sun.jndi.ldap.Connection.run(java....@11.0.5/Connection.java:793)
at java.lang.Thread.run(java.ba...@11.0.5/Thread.java:834)Thanks,Luke--
--
To unsubscribe, email repo-discuss+unsub...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/17f8a32e-411e-4029-abb3-196a5b10879a%40googlegroups.com.
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.