CRASH REPORT Process aten_detector with 0 neighbours exited with reason: {timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}} in gen_server:call/2 line 215'

177 views
Skip to first unread message

Rahul R

unread,
May 8, 2020, 4:36:53 AM5/8/20
to rabbitmq-users
Hello,

The rabbitmq server shows the error CRASH REPORT Process aten_detector with 0 neighbours exited with reason: {timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}} in gen_server:call/2 line 215''" triggering connection restarts for all the dependent services.

I see the warning in the logs 

** WARNING ** Mnesia is overloaded: {dump_log,time_threshold}'


But there is nothing which points to the root cause of the problem. Can someone please take a look?


Around same time, pika shows the following error for one of the clients:

MainThread pika.heartbeat: INFO - Connection is idle, 1 stale byte intervals'


b'flexsnap-rabbitmq': b'client unexpectedly closed TCP connection'

b'flexsnap-rabbitmq': b"2020-05-05 17:30:02.262 [warning] <0.730.0> closing AMQP connection <0.730.0> (172.18.0.8:33770 -> 172.18.0.5:5672 - connectionFactory#2361365c:0, vhost: '/', user: 'idm'):"

b'flexsnap-rabbitmq': b'client unexpectedly closed TCP connection'

b'flexsnap-rabbitmq': b"2020-05-05 17:30:02.512 [warning] <0.1288.0> closing AMQP connection <0.1288.0> (172.18.0.14:38058 -> 172.18.0.5:5671, vhost: '/', user: 'policy.0'):"

b'flexsnap-rabbitmq': b'client unexpectedly closed TCP connection'

b'flexsnap-rabbitmq': b"2020-05-05 17:30:02.512 [warning] <0.1248.0> closing AMQP connection <0.1248.0> (172.18.0.13:54734 -> 172.18.0.5:5671, vhost: '/', user: 'scheduler.0'):"

b'flexsnap-rabbitmq': b'client unexpectedly closed TCP connection'

b'flexsnap-rabbitmq': b"2020-05-05 17:30:02.512 [warning] <0.1828.0> closing AMQP connection <0.1828.0> (172.18.0.3:56108 -> 172.18.0.5:5671, vhost: '/', user: 'agent.89118754ed2043d395ee38da900141c6'):"

b'flexsnap-rabbitmq': b'client unexpectedly closed TCP connection'

b'flexsnap-rabbitmq': b"2020-05-05 17:30:02.512 [warning] <0.1333.0> closing AMQP connection <0.1333.0> (172.18.0.15:49856 -> 172.18.0.5:5671, vhost: '/', user: 'notification.0'):"

b'flexsnap-rabbitmq': b'client unexpectedly closed TCP connection'

b'flexsnap-rabbitmq': b"2020-05-05 17:30:02.512 [warning] <0.649.0> closing AMQP connection <0.649.0> (172.18.0.6:58810 -> 172.18.0.5:5671, vhost: '/', user: 'certmaster.0'):"

b'flexsnap-rabbitmq': b'client unexpectedly closed TCP connection'

b'flexsnap-rabbitmq': b'2020-05-05 17:30:01.647 [error] <0.246.0> ** Generic server aten_detector terminating'

b'flexsnap-rabbitmq': b'** Last message in was poll'

b'flexsnap-rabbitmq': b'** When Server state == {state,#Ref<0.1449668655.766246914.65048>,1000,0.99,#{},#{}}'

b'flexsnap-rabbitmq': b'** Reason for termination =='

b'flexsnap-rabbitmq': b'** {{timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}},[{gen_server,call,2,[{file,"gen_server.erl"},{line,215}]},{aten_detector,handle_info,2,[{file,"src/aten_detector.erl"},{line,97}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}'

b'flexsnap-rabbitmq': b"2020-05-05 17:30:02.514 [warning] <0.1200.0> closing AMQP connection <0.1200.0> (172.18.0.10:53286 -> 172.18.0.5:5671, vhost: '/', user: 'coordinator.0'):"

b'flexsnap-rabbitmq': b'client unexpectedly closed TCP connection'

b'flexsnap-rabbitmq': b'2020-05-05 17:30:02.514 [error] <0.246.0> CRASH REPORT Process aten_detector with 0 neighbours exited with reason: {timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}} in gen_server:call/2 line 215'

b'flexsnap-rabbitmq': b'2020-05-05 17:30:02.695 [error] <0.243.0> Supervisor aten_sup had child aten_detector started with aten_detector:start_link() at <0.246.0> exit with reason {timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}} in context child_terminated'

b'flexsnap-rabbitmq': b'2020-05-05 17:30:02.994 [warning] <0.304.0> Mnesia(rabbit@6e5b87496c59): ** WARNING ** Mnesia is overloaded: {dump_log,time_threshold}'

b'flexsnap-rabbitmq': b'2020-05-05 17:30:03.001 [warning] <0.304.0> Mnesia(rabbit@6e5b87496c59): ** WARNING ** Mnesia is overloaded: {dump_log,time_threshold}'

b'flexsnap-rabbitmq': b'2020-05-05 17:30:06.492 [info] <0.8209.21> accepting AMQP connection <0.8209.21> (172.18.0.8:34794 -> 172.18.0.5:5672)'

b'flexsnap-rabbitmq': b'2020-05-05 17:30:07.650 [info] <0.8209.21> Connection <0.8209.21> (172.18.0.8:34794 -> 172.18.0.5:5672) has a client-provided name: connectionFactory#2361365c:1'

b'flexsnap-rabbitmq': b"2020-05-05 17:30:08.054 [info] <0.8209.21> connection <0.8209.21> (172.18.0.8:34794 -> 172.18.0.5:5672 - connectionFactory#2361365c:1): user 'idm' authenticated and granted access to vhost '/'"

b'flexsnap-rabbitmq': b'2020-05-05 17:30:12.393 [info] <0.8309.21> accepting AMQP connection <0.8309.21> (172.18.0.13:55600 -> 172.18.0.5:5671)'

b'flexsnap-rabbitmq': b'2020-05-05 17:30:12.394 [info] <0.8288.21> accepting AMQP connection <0.8288.21> (172.18.0.6:54990 -> 172.18.0.5:5671)'

b'flexsnap-rabbitmq': b'2020-05-05 17:30:12.420 [info] <0.8312.21> accepting AMQP connection <0.8312.21> (172.18.0.12:55242 -> 172.18.0.5:5671)'

b'flexsnap-rabbitmq': b'2020-05-05 17:30:12.422 [info] <0.8293.21> accepting AMQP connection <0.8293.21> (172.18.0.14:38896 -> 172.18.0.5:5671)'

b'flexsnap-rabbitmq': b'2020-05-05 17:30:12.434 [info] <0.8306.21> accepting AMQP connection <0.8306.21> (172.18.0.3:56240 -> 172.18.0.5:5671)'

b'flexsnap-rabbitmq': b'2020-05-05 17:30:12.436 [info] <0.8318.21> accepting AMQP connection <0.8318.21> (172.18.0.15:50696 -> 172.18.0.5:5671)'

b'flexsnap-rabbitmq': b'2020-05-05 17:30:12.453 [info] <0.8315.21> accepting AMQP connection <0.8315.21> (172.18.0.11:45748 -> 172.18.0.5:5671)'

Johan Rhodin

unread,
May 8, 2020, 4:13:52 PM5/8/20
to rabbitmq-users
Hi,

In order to help you, people on this list would need: a) full logs (or as complete as possible), b) your RabbitMQ and Erlang versions and preferably c) monitoring data.

If you search the archives you can find this thread [1] where it is pointed out that this is likely just a symptom of another issue such as high memory usage, and/or related to high connection churn.

Reply all
Reply to author
Forward
0 new messages