Java client error: javax.net.ssl.SSLException: Unsupported record version Unknown-0.0

[Dmitry Andrianov]

Hello.

after switching to NIO version of RabbitMQ Java client library, we started seeing these exceptions from time to time:

2017-09-14T13:37:04+00:00  WARN   [rabbitmq-nio] c.r.c.i.n.NioLoop Error during reading frames

javax.net.ssl.SSLException: Unsupported record version Unknown-0.0

at sun.security.ssl.InputRecord.checkRecordVersion(InputRecord.java:552)

at sun.security.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:113)

at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:868)

at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)

at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)

at com.rabbitmq.client.impl.nio.SslEngineByteBufferInputStream.read(SslEngineByteBufferInputStream.java:78)

at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:288)

at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:91)

at com.rabbitmq.client.impl.nio.NioLoop.run(NioLoop.java:152)

at java.lang.Thread.run(Thread.java:748)

2017-09-14T13:37:04+00:00  ERROR  [rabbitmq-connection-shutdown-amqp://user@HOSTNAME:5671/] c.r.c.i.ForgivingExceptionHandler An unexpected connection driver error occured

javax.net.ssl.SSLException: Unsupported record version Unknown-0.0

at sun.security.ssl.InputRecord.checkRecordVersion(InputRecord.java:552)

at sun.security.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:113)

at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:868)

at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)

at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)

at com.rabbitmq.client.impl.nio.SslEngineByteBufferInputStream.read(SslEngineByteBufferInputStream.java:78)

at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:288)

at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:91)

at com.rabbitmq.client.impl.nio.NioLoop.run(NioLoop.java:152)

at java.lang.Thread.run(Thread.java:748)

ERROR  [rabbitmq-connection-shutdown-amqp://CN=202972831...@54.229.117.162:443/] c.r.c.i.ForgivingExceptionHandler An unexpected connection driver error occured

javax.net.ssl.SSLException: Buffer closed

at com.rabbitmq.client.impl.nio.SslEngineHelper.write(SslEngineHelper.java:158)

at com.rabbitmq.client.impl.nio.SslEngineByteBufferOutputStream.doFlush(SslEngineByteBufferOutputStream.java:59)

at com.rabbitmq.client.impl.nio.SslEngineByteBufferOutputStream.flush(SslEngineByteBufferOutputStream.java:53)

at java.io.DataOutputStream.flush(DataOutputStream.java:123)

at com.rabbitmq.client.impl.nio.NioLoop.run(NioLoop.java:238)

at java.lang.Thread.run(Thread.java:748)

I am not sure what triggers it but it happens on an already established AMQP connection where we already exchanged many messages with the broker. (As in not during initial SSL handshake)

Do you know what these could be? We did not see it before with non-NIO client.

Also, just out of curiosity, the "ForgivingExceptionHandler" above - it does not mean that Rabbit considers this error recoverable, right? Because I doubt anything SSL related can be recoverable... (If it is SSL related of course...)

Cheers

[Michael Klishin]

ForgivingExceptionHandler logs an unhandled exception from a consumer-executed operation and otherwise ignores it.

There was (and still is, just not used by default) a strict version that immediately closes the channel.

TLS issues are connection-level, not channel. I'd start by looking at server logs and making sure you run the most

recent Erlang/OTP version supported (e.g. 19.3.6.2 or 20.0.4), since RabbitMQ itself does not implement any parts of TLS.

ERROR  [rabbitmq-connection-shutdown-amqp://CN=202972831479653...@54.229.117.162:443/] c.r.c.i.ForgivingExceptionHandler An unexpected connection driver error occured

javax.net.ssl.SSLException: Buffer closed

at com.rabbitmq.client.impl.nio.SslEngineHelper.write(SslEngineHelper.java:158)

at com.rabbitmq.client.impl.nio.SslEngineByteBufferOutputStream.doFlush(SslEngineByteBufferOutputStream.java:59)

at com.rabbitmq.client.impl.nio.SslEngineByteBufferOutputStream.flush(SslEngineByteBufferOutputStream.java:53)

at java.io.DataOutputStream.flush(DataOutputStream.java:123)

at com.rabbitmq.client.impl.nio.NioLoop.run(NioLoop.java:238)

at java.lang.Thread.run(Thread.java:748)

I am not sure what triggers it but it happens on an already established AMQP connection where we already exchanged many messages with the broker. (As in not during initial SSL handshake)

Do you know what these could be? We did not see it before with non-NIO client.

Also, just out of curiosity, the "ForgivingExceptionHandler" above - it does not mean that Rabbit considers this error recoverable, right? Because I doubt anything SSL related can be recoverable... (If it is SSL related of course...)

Cheers

hivehome.com

Hive | London | Cambridge | Houston | Toronto

The information contained in or attached to this email is confidential and intended only for the use of the individual(s) to which it is addressed. It may contain information which is confidential and/or covered by legal professional or other privilege. The views expressed in this email are not necessarily the views of Centrica plc, and the company, its directors, officers or employees make no representation or accept any liability for their accuracy or completeness unless expressly stated to the contrary. 

Centrica Connected Home Limited (company no: 5782908), registered in England and Wales with its registered office at Millstream, Maidenhead Road, Windsor, Berkshire SL4 5GD.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Dmitry Andrianov]

Hi, Michael.

No, we are definitely not running most recent RabbitMQ/Erlang. We are evaluating the upgrade path right now but migration is going to take time.

Thing is that it worked ok with the very same brokers before and it is the transition to NIO client seem to have triggered that problem.

I am still investigating (and looking into server logs as you suggested) just hoped someone else has already seen that kind of issue before.

Cheers

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

I don't think we've seen this and list archives only reveal this thread:

https://groups.google.com/forum/#!searchin/rabbitmq-users/Unsupported$20record$20version%7Csort:relevance.

ERROR  [rabbitmq-connection-shutdown-amqp://CN=2029728314796539918@54.229.117.162:443/] c.r.c.i.ForgivingExceptionHandler An unexpected connection driver error occured

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Arnaud Cogoluègnes]

You can also upgrade to the latest Java 8 version. Right now, it's difficult to help without more information.

Try to reproduce with a simple Java program and provide us with the exact versions you're using (Java, Java Client, RabbitMQ, Erlang) and the server logs as well.

ERROR  [rabbitmq-connection-shutdown-amqp://CN=2029728314796539918@54.229.117.162:443/] c.r.c.i.ForgivingExceptionHandler An unexpected connection driver error occured

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

[Dmitry Andrianov]

Hi.

At the moment situation looks like the exceptions actually started exactly because we upgraded to Java 8.

So I think NIO was a red herring.

Issue seems to be easily reproducible - each time we send a large message from broker to client it crashes.

We are still investigating but looks like it is not RabbitMQ Java client to blame but Java itself, the specific version we are using.

Thank you and sorry for the false alarm.

Dmitry

ERROR  [rabbitmq-connection-shutdown-amqp://CN=202972831479653...@54.229.117.162:443/] c.r.c.i.ForgivingExceptionHandler An unexpected connection driver error occured

[Michael Klishin]

Apparently there are threads as old as 3-4 years that mention something similar:

https://stackoverflow.com/questions/23324807/randomly-sslexception-unsupported-record-version-unknown-0-0

Are you configuring a TLS version to use or relying on the JDK default?

Do you limit TLS versions allowed on the RabbitMQ end?

ERROR  [rabbitmq-connection-shutdown-amqp://CN=2029728314796539918@54.229.117.162:443/] c.r.c.i.ForgivingExceptionHandler An unexpected connection driver error occured

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dmitry Andrianov]

Yeah, I saw all those and I think they are different - the connection fails to establish for all these guys.

In our case connection is established and passes messages back and forth until broker sends a large message when everything fails.

On the client we do SSLContext.getInstance("TLS") (not "TLSv1" as your link says but I bet it is the same).

We are not configuring SSL explicitly on RabbitMQ side and to be fair we are using pretty old rabbit there as well as Erlang so God knows what is available on the broker side.

While connection is still established, the broker reports:

tlsv1.2_{ecdhe_rsa aes_256_cbc sha384}

I still blame JVM - something must have changed in SSL configuration in version 8. The only thing that puzzles me is why it happens in the middle of connection.

Maybe something is trying to re-negotiate ciphers, I dunno. Still investigating.

Thanks

Dmitry 

ERROR  [rabbitmq-connection-shutdown-amqp://CN=202972831479653...@54.229.117.162:443/] c.r.c.i.ForgivingExceptionHandler An unexpected connection driver error occured

[Michael Klishin]

TLSv1.2 and ecdhe_rsa aes_256_cbc sha384 don't seem to be too outdated ;)

You can control how strict the negotiation is in the Erlang TLS implementation: is it something you've tried?

http://www.rabbitmq.com/ssl.html

ERROR  [rabbitmq-connection-shutdown-amqp://CN=2029728314796539918@54.229.117.162:443/] c.r.c.i.ForgivingExceptionHandler An unexpected connection driver error occured

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dmitry Andrianov]

Hello again. Apparently, RabbitMQ client does implement some parts of SSL :)  - it is the SslEngineByteBufferInputStream that fails to reassemble the stream properly.

I raised you https://github.com/rabbitmq/rabbitmq-java-client/issues/307

My analysis there can be wrong, I have zero experience with NIO and ByteBuffers but how the buffer treated in "then" and "else" branches is visible.

Cheers

On Thursday, 14 September 2017 15:59:20 UTC+1, Michael Klishin wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Arnaud Cogoluègnes]

Thanks for the detailed report, we'll work on this.

ERROR  [rabbitmq-connection-shutdown-amqp://CN=2029728314796539918@54.229.117.162:443/] c.r.c.i.ForgivingExceptionHandler An unexpected connection driver error occured

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

[Arnaud Cogoluègnes]

I pushed a snapshot [1] with a fix, can you please try it?

Thanks.

[1] https://oss.sonatype.org/content/repositories/snapshots/com/rabbitmq/amqp-client/4.2.2-SNAPSHOT/