NullPointerException getChannelNumber

232 views
Skip to first unread message

Youtube Solace

unread,
Feb 1, 2016, 12:07:29 PM2/1/16
to rabbitmq-users
Hi All,


Below is the stacktrace and error message:

Caught an exception during connection recovery!
java.lang.NullPointerException
        at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.getChannelNumber(AutorecoveringChannel.java:49)
        at com.rabbitmq.client.impl.DefaultExceptionHandler.handleChannelRecoveryException(DefaultExceptionHandler.java:89)
        at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverChannels(AutorecoveringConnection.java:450)
        at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.beginAutomaticRecovery(AutorecoveringConnection.java:410)
        at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.access$000(AutorecoveringConnection.java:53)
        at com.rabbitmq.client.impl.recovery.AutorecoveringConnection$1.shutdownCompleted(AutorecoveringConnection.java:352)
        at com.rabbitmq.client.impl.ShutdownNotifierComponent.notifyListeners(ShutdownNotifierComponent.java:75)
        at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:576)
        at java.lang.Thread.run(Thread.java:745)

These started appearing over a matter of days and growing in number, 1st day 3 or 4, 2nd day  7, 3rd day 20+
From the looks of things the connection did recover as we were still able to consume messages off the queue.
It may help to know that:
- Connection Reset exceptions started appearing shortly after the 2nd day
- Finally the process started running out of Heap Space.



Also I don't believe this is the same issue as : https://github.com/rabbitmq/rabbitmq-java-client/issues/105

FYI the requested heartbeat for ConnectionFactory has been set to 10 seconds, I'm not sure if this will make a difference to the Exception being raised, but it was worth pointing out.

Thanks
PP

Michael Klishin

unread,
Feb 1, 2016, 12:09:53 PM2/1/16
to rabbitm...@googlegroups.com, Youtube Solace
On 1 February 2016 at 20:07:32, Youtube Solace (p300...@gmail.com) wrote:
> FYI the requested heartbeat for ConnectionFactory has been
> set to 10 seconds, I'm not sure if this will make a difference to
> the Exception being raised, but it was worth pointing out.

Please post a heap dump of a JVM process that observes these exceptions
and specify what client version you use .
--
MK

Staff Software Engineer, Pivotal/RabbitMQ


Michael Klishin

unread,
Feb 1, 2016, 12:11:49 PM2/1/16
to rabbitm...@googlegroups.com, Youtube Solace
On 1 February 2016 at 20:07:32, Youtube Solace (p300...@gmail.com) wrote:
> Connection Reset exceptions started appearing shortly after
> the 2nd day

Do you mean ECONNRESET, which translates into a SocketException in Java? That means
TCP connection failed during recovery and can be perfectly normal in case of  network connectivity issues.

Youtube Solace

unread,
Feb 1, 2016, 12:22:42 PM2/1/16
to rabbitmq-users, p300...@gmail.com
Apologies Michael,

The RabbitMQ Java Client version is 3.5.3 and so is the RabbitMQ Server.

I'm afraid we cannot get a Heap Dump of the JVM Process.

The connection reset error:

com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; cause: java.net.SocketException: Connection reset 

        at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:195)

        at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:309)

        at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:657)

        at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:640)

        at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:631)

        at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicPublish(AutorecoveringChannel.java:168)


What was interesting is that there were 3 other processes using RabbitMQ java client, all of which connecting to the same node, but had no issues regarding connectivity (also note everything lives on the same box on localhost).


I will raise the NullPointerException stack trace on github.


Thanks

PP

Michael Klishin

unread,
Feb 1, 2016, 12:36:36 PM2/1/16
to rabbitm...@googlegroups.com, Youtube Solace, p300...@gmail.com
 On 1 February 2016 at 20:22:45, Youtube Solace (p300...@gmail.com) wrote:
> The RabbitMQ Java Client version is 3.5.3 and so is the RabbitMQ
> Server

Please upgrade to 3.6.0 (the client, server doesn't have to be upgraded in lock step).

Robert Oschwald

unread,
Oct 24, 2016, 10:04:37 AM10/24/16
to rabbitmq-users
Got the same problem, but already using amqp-client 3.6.1 (Grails 3.x using rabbitmq-native plugin), while doing integration tests. This happens when both RMQ servers in the cluster are stopped and restarted. I expect the clients to reconnect all the time, but randomly I get:

Caught an exception during connection recovery!
java.lang.NullPointerException
    at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.getChannelNumber(AutorecoveringChannel.java:49)
    at com.rabbitmq.client.impl.ForgivingExceptionHandler.handleChannelRecoveryException(ForgivingExceptionHandler.java:95)
    at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverChannels(AutorecoveringConnection.java:504)
    at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.beginAutomaticRecovery(AutorecoveringConnection.java:449)
    at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.access$000(AutorecoveringConnection.java:53)
    at com.rabbitmq.client.impl.recovery.AutorecoveringConnection$1.shutdownCompleted(AutorecoveringConnection.java:383)
    at com.rabbitmq.client.impl.ShutdownNotifierComponent.notifyListeners(ShutdownNotifierComponent.java:75)
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:578)
    at java.lang.Thread.run(Thread.java:745)

Michael Klishin

unread,
Oct 24, 2016, 10:17:39 AM10/24/16
to rabbitm...@googlegroups.com
Channel ID set to `null` usually means that there were no more channel numbers to allocate
(there are 65535 slots by default). I don't recall any other specific reasons when a channel's number
could be nullified.

There were 4 point releases after 3.6.1 and two 4.0 milestones published to https://bintray.com/rabbitmq/maven-milestones.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

vikinghawk

unread,
Oct 24, 2016, 10:44:44 AM10/24/16
to rabbitmq-users
My guess is you are hitting the same scenario i did. Are you creating brand new channels while your connection is disconnected or being recovered? If so you can hit a NPE during recovery, it then tries to pass that NPE to the exception handler and then gets another NPE there when it tries to get the channel number for logging.

See bullet 2 here:

Fix was done here and will be released in 3.6.6 and 4.0.0:

Robert Oschwald

unread,
Oct 24, 2016, 11:58:52 AM10/24/16
to rabbitmq-users
Tried with 2.6.6-SNAPSHOT (milestone5), problem solved.  Thanks vikinghawk! 
If either the ha_proxy cluster or both RMQ nodes are down and up again, consumers reliable reconnect now.
Reply all
Reply to author
Forward
0 new messages