a lot of error closing AMQP connection

[cche...@gmail.com]

Hello, all. I run a three nodes mq cluster with RabbitMQ 3.8.7 on Erlang 22.3.4.9.

Maybe some network error, there are a lot of error closing AMQP connection in my mq's log, cat mq-1.log |grep error |head -200  show below:

2021-09-28 16:24:14.209 [error] <0.9625.3363> Supervisor {<0.9625.3363>,rabbit_channel_sup} had child channel started with rabbit_channel:start_link(1, <0.13039.5505>, <0.26949.3018>, <0.13039.5505>, <<"10.42.185.19:56468 -> 10.42.184.14:5672">>, rabbit_framing_amqp_0_9_1, {user,<<"openstack">>,[administrator],[{rabbit_auth_backend_internal,none}]}, <<"/">>, [{<<"connection.blocked">>,bool,true},{<<"authentication_failure_close">>,bool,true},{<<"consumer...">>,...}], <0.29097.1297>, <0.24565.2500>) at <0.4051.1960> exit with reason noproc in context shutdown_error

2021-09-28 16:24:15.162 [error] <0.10359.1846> Supervisor {<0.10359.1846>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error

2021-09-28 16:24:15.202 [error] <0.2190.6533> Supervisor {<0.2190.6533>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason noproc in context shutdown_error

2021-09-28 16:24:50.344 [error] <0.2695.1436> closing AMQP connection <0.2695.1436> (10.42.184.70:42772 -> 10.42.184.14:5672):

2021-09-28 16:24:57.736 [error] <0.24.5285> Supervisor {<0.24.5285>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error

2021-09-28 16:25:09.089 [error] <0.16419.2527> closing AMQP connection <0.16419.2527> (10.42.184.70:45050 -> 10.42.184.14:5672):

2021-09-28 16:27:27.436 [error] <0.30477.645> Supervisor {<0.30477.645>,rabbit_channel_sup} had child channel started with rabbit_channel:start_link(1, <0.31374.1209>, <0.819.4860>, <0.31374.1209>, <<"10.42.184.39:43946 -> 10.42.184.14:5672">>, rabbit_framing_amqp_0_9_1, {user,<<"openstack">>,[administrator],[{rabbit_auth_backend_internal,none}]}, <<"/">>, [{<<"connection.blocked">>,bool,true},{<<"authentication_failure_close">>,bool,true},{<<"consumer...">>,...}], <0.6061.5484>, <0.9103.961>) at <0.24291.564> exit with reason noproc in context shutdown_error

2021-09-28 16:28:12.145 [error] <0.19823.8007> closing AMQP connection <0.19823.8007> (10.42.185.95:48306 -> 10.42.184.14:5672):

2021-09-28 16:28:12.265 [error] <0.20905.6970> closing AMQP connection <0.20905.6970> (10.42.185.95:48294 -> 10.42.184.14:5672):

2021-09-28 16:28:12.265 [error] <0.12370.4265> closing AMQP connection <0.12370.4265> (10.42.185.95:48318 -> 10.42.184.14:5672):

2021-09-28 16:28:12.265 [error] <0.19539.1912> closing AMQP connection <0.19539.1912> (10.42.185.95:48296 -> 10.42.184.14:5672):

2021-09-28 16:29:42.214 [error] <0.8721.5772> Supervisor {<0.8721.5772>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason noproc in context shutdown_error

2021-09-28 16:30:43.343 [error] <0.32450.1979> closing AMQP connection <0.32450.1979> (10.42.184.14:59666 -> 10.42.184.14:5672 - nova-conductor:67084:f0f5e3a7-624c-4e7a-a380-bfbfd7d1b4f4):

2021-09-28 16:30:59.123 [error] <0.4739.4812> closing AMQP connection <0.4739.4812> (10.42.184.51:59830 -> 10.42.184.14:5672 - nova-conductor:68229:500579e0-6265-45e4-8475-a34efff164d3):

2021-09-28 16:31:00.493 [error] <0.27052.3875> closing AMQP connection <0.27052.3875> (10.42.184.1:42140 -> 10.42.184.14:5672 - nova-api:33533:e2826b1a-d894-4543-8d1e-8cfc9b709d92):

2021-09-28 16:31:01.952 [error] <0.18332.562> closing AMQP connection <0.18332.562> (10.42.184.14:39134 -> 10.42.184.14:5672 - nova-conductor:67047:bc34baac-e788-4be8-9d7b-06c60acf2c6e):

2021-09-28 16:31:02.502 [error] <0.18211.5847> closing AMQP connection <0.18211.5847> (10.42.184.14:39100 -> 10.42.184.14:5672 - nova-conductor:67098:f55cffcb-0069-45ff-a3b0-cb72c8966cd9):

2021-09-28 16:31:03.184 [error] <0.30696.1483> closing AMQP connection <0.30696.1483> (10.42.186.8:46076 -> 10.42.184.14:5672 - nova-compute:35726:b37c8bf7-8d98-46cd-abea-05491c2a70cc):

2021-09-28 16:31:04.847 [error] <0.26557.3588> closing AMQP connection <0.26557.3588> (10.42.185.95:48354 -> 10.42.184.14:5672 - nova-compute:29503:7152f95f-930c-48c7-9097-ce2764490b0f):

2021-09-28 16:31:05.379 [error] <0.22372.6496> closing AMQP connection <0.22372.6496> (10.42.185.191:42818 -> 10.42.184.14:5672 - nova-compute:36264:7fe1ee19-31fe-4819-ba0f-ada6564f8b67):

2021-09-28 16:31:06.738 [error] <0.13950.1565> closing AMQP connection <0.13950.1565> (10.42.184.1:43150 -> 10.42.184.14:5672 - nova-conductor:34075:0e031944-d41b-40ba-8ba7-38f3c663567c):

2021-09-28 16:31:07.372 [error] <0.21218.5119> closing AMQP connection <0.21218.5119> (10.42.184.14:58916 -> 10.42.184.14:5672 - nova-conductor:67105:4190f410-d1e3-4552-a626-2b7d6fd87f7c):

2021-09-28 16:31:08.054 [error] <0.4490.2282> closing AMQP connection <0.4490.2282> (10.42.184.230:49578 -> 10.42.184.14:5672 - nova-compute:73476:57a027eb-95f8-4caa-9535-8790c0a8f522):

2021-09-28 16:31:10.223 [error] <0.27999.1917> closing AMQP connection <0.27999.1917> (10.42.185.252:36316 -> 10.42.184.14:5672 - nova-compute:33090:309e364b-680f-4906-acbf-8b34fac7cad1):

And some error occurs immediately after a accept log like:

2021-09-28 16:13:37.127 [info] <0.19149.4215> accepting AMQP connection <0.19149.4215> (10.42.184.56:41178 -> 10.42.184.14:5672)

2021-09-28 16:13:37.127 [error] <0.19149.4215> closing AMQP connection <0.19149.4215> (10.42.184.56:41178 -> 10.42.184.14:5672):

At the same time, there is some other error like:

[error] <0.12988.5775> Supervisor {<0.12988.5775>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason noproc in context shutdown_error

2021-09-28 16:24:57.736 [error] <0.24.5285> Supervisor {<0.24.5285>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason shutdown in context shutdown_error

2021-09-28 16:27:27.436 [error] <0.30477.645> Supervisor {<0.30477.645>,rabbit_channel_sup} had child channel started with rabbit_channel:start_link(1, <0.31374.1209>, <0.819.4860>, <0.31374.1209>, <<"10.42.184.39:43946 -> 10.42.184.14:5672">>, rabbit_framing_amqp_0_9_1, {user,<<"openstack">>,[administrator],[{rabbit_auth_backend_internal,none}]}, <<"/">>, [{<<"connection.blocked">>,bool,true},{<<"authentication_failure_close">>,bool,true},{<<"consumer...">>,...}], <0.6061.5484>, <0.9103.961>) at <0.24291.564> exit with reason noproc in context shutdown_error

2021-09-28 16:29:42.214 [error] <0.8721.5772> Supervisor {<0.8721.5772>,rabbit_channel_sup_sup} had child channel_sup started with rabbit_channel_sup:start_link() at undefined exit with reason noproc in context shutdown_error

2021-09-28 16:42:39.786 [error] <0.27585.3136> CRASH REPORT Process <0.27585.3136> with 0 neighbours exited with reason: {unexpected_message,{'DOWN',{delegate_1,<45199.1521.338>},process,<45199.1521.338>,shutdown}} in rabbit_heartbeat:heartbeater/3 line 138

2021-09-28 16:42:39.793 [error] <0.18999.5185> Supervisor {<0.18999.5185>,rabbit_connection_helper_sup} had child heartbeat_sender started with rabbit_heartbeat:start_heartbeat_sender(#Port<0.12451817>, 60, #Fun<rabbit_reader.43.113297234>, {heartbeat_sender,<<"10.42.185.232:42478 -> 10.42.184.14:5672">>}) at <0.27585.3136> exit with reason {unexpected_message,{'DOWN',{delegate_1,<45199.1521.338>},process,<45199.1521.338>,shutdown}} in context child_terminated

Can anyone tell me what there errors mean and why they happends?

Is the param num_acceptors.tcp = 10  relates to this error?

Thank you!

[cche...@gmail.com]

Anyone here?

[Wes Peng]

Maybe you want to check if there is any system issue. such as:

1. disk/inode full

2. file system read only

3. network filter issue

4. network congestion

5. limited memory

etc..

Regards.

[cche...@gmail.com]

Thank you, I check the log agains and find a lot of CRASH REPORT log：

2021-09-28 15:52:25.705 [error] <0.236.0> ** Generic server aten_detector terminating

2021-09-28 15:52:26.745 [error] <0.236.0> CRASH REPORT Process aten_detector with 0 neighbours exited with reason: {timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}} in gen_server:call/2 line 215

2021-09-28 15:52:26.775 [error] <0.233.0> Supervisor aten_sup had child aten_detector started with aten_detector:start_link() at <0.236.0> exit with reason {timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}} in context child_terminated

...

2021-09-28 16:42:39.786 [error] <0.27585.3136> CRASH REPORT Process <0.27585.3136> with 0 neighbours exited with reason: {unexpected_message,{'DOWN',{delegate_1,<45199.1521.338>},process,<45199.1521.338>,shutdown}} in rabbit_heartbeat:heartbeater/3 line 138

2021-09-28 16:42:39.793 [error] <0.18999.5185> Supervisor {<0.18999.5185>,rabbit_connection_helper_sup} had child heartbeat_sender started with rabbit_heartbeat:start_heartbeat_sender(#Port<0.12451817>, 60, #Fun<rabbit_reader.43.113297234>, {heartbeat_sender,<<"10.42.185.232:42478 -> 10.42.184.14:5672">>}) at <0.27585.3136> exit with reason {unexpected_message,{'DOWN',{delegate_1,<45199.1521.338>},process,<45199.1521.338>,shutdown}} in context child_terminated

2021-09-28 16:42:39.840 [warning] <0.28619.5867> closing AMQP connection <0.28619.5867> (10.42.184.51:56566 -> 10.42.184.14:5672 - nova-conductor:68269:8e6fac05-347e-45a8-9c42-991d24e027e3, vhost: '/', user: 'openstack'):

2021-09-28 16:42:39.894 [warning] <0.2279.0> rabbit_sysmon_handler busy_dist_port <0.3765.0> [{initial_call,{gm,init,1}},{erts_internal,dsend_continue_trap,1},{message_queue_len,0}] {#Port<0.75>,unknown}

maybe this CRASH REPORT  is the root issue? But why CRASH?

[Loïc Hoguin]

Please ignore these shutdown_error errors, they will be fixed once you upgrade to the most recent Erlang version (24.1.1+)

 

-- 

Loïc Hoguin

2021-09-28 16:42:39.786 [error] <0.27585.3136> CRASH REPORT Process <0.27585.3136> with 0 neighbours exited with reason: {unexpected_message,{'DOWN',{delegate_1,<45199.1521.338>},process,<45199.1521.338>,shutdown}} in rabbit_heartbeat:heartbeater/3 line 138

2021-09-28 16:42:39.793 [error] <0.18999.5185> Supervisor {<0.18999.5185>,rabbit_connection_helper_sup} had child heartbeat_sender started with rabbit_heartbeat:start_heartbeat_sender(#Port<0.12451817>, 60, #Fun<rabbit_reader.43.113297234>, {heartbeat_sender,<<"10.42.185.232:42478 -> 10.42.184.14:5672">>}) at <0.27585.3136> exit with reason {unexpected_message,{'DOWN',{delegate_1,<45199.1521.338>},process,<45199.1521.338>,shutdown}} in context child_terminated

 

Can anyone tell me what there errors mean and why they happends?

Is the param num_acceptors.tcp = 10  relates to this error?

Thank you!

 

 

 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/3d967d04-6511-487d-b2e8-43250f233bbfn%40googlegroups.com.