Runnable thread hangs in java.net.SocketOutputStream.socketWrite0

[Jörgen Risholt]

HI, 

This problem is related to the one we saw earlier this summer: https://groups.google.com/forum/#!topic/rabbitmq-users/GZx3vAilfK8. However then we did not really understand what we did see. We had a similar situation the other day in one of our system that has around 1400 consumers connected to a messagebus. Suddenly (?) all consumer threads hanged when they did a call down to rabbit (in this case a basicAck()). Now it seems that all threads are dependent on one of these (single-point-of-failure) that seems to locked on the native SocketOutputStream.socketWrite0(). We had to do a complete system restart to overcome the problem.   

This is the hanging thread and it was hanging for, at least one hour before restart. We did not notice any other problems: 

"EiffelQB_eiffel003.seki.fem001.fem001-eiffel003.a38ad4e5-dab4-4430-96c8-14d0993ad1c1-RuleTrigger.durable" daemon prio=10 tid=0x00007f2c59f55800 nid=0x3b1a runnable [0x00007f2c22ed4000]

   java.lang.Thread.State: RUNNABLE

at java.net.SocketOutputStream.socketWrite0(Native Method)

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)

at java.net.SocketOutputStream.write(SocketOutputStream.java:159)

at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:377)

at sun.security.ssl.OutputRecord.write(OutputRecord.java:363)

at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:830)

at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:801)

at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)

- locked <0x0000000685bdbdd8> (a sun.security.ssl.AppOutputStream)

at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)

at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)

- locked <0x0000000685be18a8> (a java.io.BufferedOutputStream)

at java.io.DataOutputStream.flush(DataOutputStream.java:123)

at com.rabbitmq.client.impl.SocketFrameHandler.flush(SocketFrameHandler.java:150)

at com.rabbitmq.client.impl.AMQConnection.flush(AMQConnection.java:514)

at com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:125)

at com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:321)

- locked <0x0000000681bb6fb8> (a java.lang.Object)

at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:297)

- locked <0x0000000681bb6fb8> (a java.lang.Object)

at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:290)

- locked <0x0000000681bb6fb8> (a java.lang.Object)

at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:1012)

       .......

[Michael Klishin]

 On 25 Sep 2015 at 09:34:56, Jörgen Risholt (jorgen....@gmail.com) wrote:
> We had a similar situation the other day in one of our system that
> has around 1400 consumers connected to a messagebus. Suddenly
> (?) all consumer threads hanged when they did a call down to rabbit
> (in this case a basicAck()). Now it seems that all threads are
> dependent on one of these (single-point-of-failure) that seems
> to locked on the native SocketOutputStream.socketWrite0().

Your connections probably were also publishing and were blocked by a resource-driven alarm
(e.g. node has surpassed its RAM watermark).

See server log files.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ