ForgivingExceptionHandler issue coming in one of the rabbitmq queue

1,087 views
Skip to first unread message

Ankush Garg

unread,
Oct 11, 2020, 3:33:12 AM10/11/20
to rabbitmq-users
Hi all,

I am using rabbitmq Amqp queue to receive packets from publisher. Below are some details for this program.
1) Spring boot version: 2.0.6
2) Rabbitmq Cli version: org.springframework.boot:spring-boot-starter-amqp
3) rabbitmq.port: 5678
4) Exchange Type: Fanout
5) queueType: non durable and non exclusive queue

some code snippet:

channel.queueDeclare(queueName, false, false, false, null);

channel.queueBind(queueName, topicExchangeName, "");


The issue is queue gets connection reset as mention below:

ERROR c.r.c.impl.ForgivingExceptionHandler - An unexpected connection driver error occured

javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset

        at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1551)

        at sun.security.ssl.AppInputStream.read(AppInputStream.java:95)

        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)

        at java.io.BufferedInputStream.read(BufferedInputStream.java:265)

        at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:288)

        at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:91)

        at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:164)

        at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:571)

        at java.lang.Thread.run(Thread.java:748)

Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection reset

        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)

        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959)

        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1916)

        at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1880)

        at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1825)

        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:128)

        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)

        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)

        at java.io.DataOutputStream.flush(DataOutputStream.java:123)

        at com.rabbitmq.client.impl.SocketFrameHandler.flush(SocketFrameHandler.java:177)

        at com.rabbitmq.client.impl.HeartbeatSender$HeartbeatRunnable.run(HeartbeatSender.java:140)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        ... 1 common frames omitted

Caused by: java.net.SocketException: Connection reset

        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:115)

        at java.net.SocketOutputStream.write(SocketOutputStream.java:155)

        at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)

        at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)

        at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:886)

        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:857)

        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)

        ... 12 common frames omitted


Can any one help me to understand the issue as my message are deliver on my java code but the message-count on rabbitmq UI is increasing. This happens again and again sometime it works. We are able to see both incoming and deliver rate.

Arnaud Cogoluegnes

unread,
Oct 12, 2020, 3:13:34 AM10/12/20
to rabbitm...@googlegroups.com
Could you provide the version and logs of the server? With the server logs we should be able to match the connection closing with some log message and find out more about what's going on.

________________________________________
From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of Ankush Garg <ankus...@gslab.com>
Sent: Sunday, October 11, 2020 9:33 AM
To: rabbitmq-users
Subject: [rabbitmq-users] ForgivingExceptionHandler issue coming in one of the rabbitmq queue
Confidentiality Notice and Disclaimer: This email (including any attachments) contains information that may be confidential, privileged and/or copyrighted. If you are not the intended recipient, please notify the sender immediately and destroy this email. Any unauthorized use of the contents of this email in any manner whatsoever, is strictly prohibited. If improper activity is suspected, all available information may be used by the sender for possible disciplinary action, prosecution, civil claim or any remedy or lawful purpose. Email transmission cannot be guaranteed to be secure or error-free, as information could be intercepted, lost, arrive late, or contain viruses. The sender is not liable whatsoever for damage resulting from the opening of this message and/or the use of the information contained in this message and/or attachments. Expressions in this email cannot be treated as opined by the sender company management – they are solely expressed by the sender unless authorized.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com<mailto:rabbitmq-user...@googlegroups.com>.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/c7056011-6ec0-480d-acb9-666da47a6197n%40googlegroups.com<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2Fc7056011-6ec0-480d-acb9-666da47a6197n%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cacogoluegnes%40vmware.com%7Cde88ccf64bf544c9295508d86db7ed50%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637379984020470177%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=CWlCtHPA6fz7rY3RorL9t5N2h2maOHYBAvW2W1Ouc2Y%3D&reserved=0>.

Ankush Garg

unread,
Oct 14, 2020, 12:04:49 PM10/14/20
to rabbitmq-users

Rabbitmq server version 3.5.7

Logs from file:

=SUPERVISOR REPORT==== 13-Oct-2020::10:01:25 ===

     Supervisor: {<0.5100.6420>,rabbit_channel_sup_sup}

     Context:    shutdown_error

     Reason:     shutdown

     Offender:   [{nb_children,1},

                  {name,channel_sup},

                  {mfargs,{rabbit_channel_sup,start_link,[]}},

                  {restart_type,temporary},

                  {shutdown,infinity},

                  {child_type,supervisor}]

Ankush Garg

unread,
Oct 14, 2020, 12:11:14 PM10/14/20
to rabbitmq-users

This are below messages which are repeating again and again in log file.

=ERROR REPORT==== 14-Oct-2020::15:54:57 ===

closing AMQP connection <0.9128.68> (subscriber_server_private_IP:45710 -> rabbitmq_server_private_IP:5678):

Missed heartbeats from client, timeout: 60s


=INFO REPORT==== 14-Oct-2020::15:55:34 ===

accepting AMQP connection <0.22162.75> (subscriber_server_private_IP:45940 -> rabbitmq_server_private_IP:5678)

Arnaud Cogoluegnes

unread,
Oct 15, 2020, 3:06:47 AM10/15/20
to rabbitm...@googlegroups.com
OK, this is explicit, the server closes the connection because it does not receive any heartbeat frames for a while. The connection between the 2 must be dead.

You can take a traffic capture [1] if you can reproduce.

Disconnections can happen, network is not 100% reliable. Nevertheless, they should not happen too often, so it can be some proxy or network device that chooses to kill the connection at some point. You can inspect this as well.

I see that you're mentioning Spring Boot and Spring AMQP, but your code sample is using the Java client API. Note that Spring AMQP disables the Java client connection recovery mechanism to use its own, so you may not benefit from automatic connection recovery when using the Java client API in this case, this could also explain your problem.

[1] https://www.rabbitmq.com/amqp-wireshark.html

________________________________________
From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of Ankush Garg <ankus...@gslab.com>
Sent: Wednesday, October 14, 2020 6:11 PM
To: rabbitmq-users
Subject: Re: [rabbitmq-users] ForgivingExceptionHandler issue coming in one of the rabbitmq queue
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/c7056011-6ec0-480d-acb9-666da47a6197n%40googlegroups.com<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2Fc7056011-6ec0-480d-acb9-666da47a6197n%2540googlegroups.com&data=04%7C01%7Cacogoluegnes%40vmware.com%7C5f5d1e2822b740593bd108d8705bcb34%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637382886860542789%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=TS32vGjUGdpevKr%2FEve3LveZMN7fPFNw2dPcThsXjyM%3D&reserved=0><https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2Fc7056011-6ec0-480d-acb9-666da47a6197n%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cacogoluegnes%40vmware.com%7Cde88ccf64bf544c9295508d86db7ed50%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637379984020470177%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=CWlCtHPA6fz7rY3RorL9t5N2h2maOHYBAvW2W1Ouc2Y%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2Fc7056011-6ec0-480d-acb9-666da47a6197n%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cacogoluegnes%40vmware.com%7C5f5d1e2822b740593bd108d8705bcb34%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637382886860552781%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=c43WqL1fBR8VXxLFzVrGC5HmEfKlxIi%2FVbnpbSbe7gg%3D&reserved=0>>.

Confidentiality Notice and Disclaimer: This email (including any attachments) contains information that may be confidential, privileged and/or copyrighted. If you are not the intended recipient, please notify the sender immediately and destroy this email. Any unauthorized use of the contents of this email in any manner whatsoever, is strictly prohibited. If improper activity is suspected, all available information may be used by the sender for possible disciplinary action, prosecution, civil claim or any remedy or lawful purpose. Email transmission cannot be guaranteed to be secure or error-free, as information could be intercepted, lost, arrive late, or contain viruses. The sender is not liable whatsoever for damage resulting from the opening of this message and/or the use of the information contained in this message and/or attachments. Expressions in this email cannot be treated as opined by the sender company management – they are solely expressed by the sender unless authorized.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com<mailto:rabbitmq-user...@googlegroups.com>.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/97c18f65-5b34-4ec5-9d7f-c4db15d1f60an%40googlegroups.com<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2F97c18f65-5b34-4ec5-9d7f-c4db15d1f60an%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cacogoluegnes%40vmware.com%7C5f5d1e2822b740593bd108d8705bcb34%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637382886860562776%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=kA6iSifQsdZc5ApU%2FaSnMD%2BH4n5lQ7v2C%2FbYHu9yj40%3D&reserved=0>.

Ankush Garg

unread,
Oct 15, 2020, 3:43:14 AM10/15/20
to rabbitmq-users
Hi Arnaud,

I don't think network would be issue in this case as this module run on dedicated amazon EC2 server and same type of module is running on another same with same configuration But we are not facing this issue on the module.

Can you tell me if this issue is due to same security file is used for connection. 

Arnaud Cogoluegnes

unread,
Oct 15, 2020, 4:24:06 AM10/15/20
to rabbitm...@googlegroups.com
> Can you tell me if this issue is due to same security file is used for connection.

Sorry, I don't understand what this means.

Ankush Garg

unread,
Oct 15, 2020, 4:25:59 AM10/15/20
to rabbitmq-users
.p12 file that we use to connect to rabbitmq server.

Arnaud Cogoluegnes

unread,
Oct 15, 2020, 4:58:23 AM10/15/20
to rabbitm...@googlegroups.com
No, there's no reason for this, it seems really like a network problem if the logs you provided are those of the error.

________________________________________
From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of Ankush Garg <ankus...@gslab.com>
Sent: Thursday, October 15, 2020 10:25 AM
To: rabbitmq-users
Subject: Re: [rabbitmq-users] ForgivingExceptionHandler issue coming in one of the rabbitmq queue

Confidentiality Notice and Disclaimer: This email (including any attachments) contains information that may be confidential, privileged and/or copyrighted. If you are not the intended recipient, please notify the sender immediately and destroy this email. Any unauthorized use of the contents of this email in any manner whatsoever, is strictly prohibited. If improper activity is suspected, all available information may be used by the sender for possible disciplinary action, prosecution, civil claim or any remedy or lawful purpose. Email transmission cannot be guaranteed to be secure or error-free, as information could be intercepted, lost, arrive late, or contain viruses. The sender is not liable whatsoever for damage resulting from the opening of this message and/or the use of the information contained in this message and/or attachments. Expressions in this email cannot be treated as opined by the sender company management – they are solely expressed by the sender unless authorized.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com<mailto:rabbitmq-user...@googlegroups.com>.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/5d28328b-7b2e-4a7a-85cb-8ee0c0a6a6cen%40googlegroups.com<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2F5d28328b-7b2e-4a7a-85cb-8ee0c0a6a6cen%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cacogoluegnes%40vmware.com%7Cd40dc172de2a40f8c13c08d870e3f629%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637383471677346894%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=zkf4OVycGfd1aQoletMDcHnKkdXdXyV%2Bzx8L%2By5zqbw%3D&reserved=0>.
Reply all
Reply to author
Forward
0 new messages