RabbitMQ 3.6.2 RC1 -- suspicious processes

[John F]

Based on the command from this post: https://groups.google.com/d/msg/rabbitmq-users/IsE5axbzkZ8/n9pJkFZkBQAJ I tried it on one of our RabbitMQ nodes. The output is here...

$ sudo rabbitmqctl eval 'rabbit_diagnostics:maybe_stuck().'
2016-05-03 07:13:55 There are 1128 processes.
2016-05-03 07:13:55 Investigated 7 processes this round, 5000ms to go.
2016-05-03 07:13:55 Investigated 7 processes this round, 4500ms to go.
2016-05-03 07:13:56 Investigated 7 processes this round, 4000ms to go.
2016-05-03 07:13:56 Investigated 7 processes this round, 3500ms to go.
2016-05-03 07:13:57 Investigated 7 processes this round, 3000ms to go.
2016-05-03 07:13:57 Investigated 7 processes this round, 2500ms to go.
2016-05-03 07:13:58 Investigated 7 processes this round, 2000ms to go.
2016-05-03 07:13:58 Investigated 6 processes this round, 1500ms to go.
2016-05-03 07:13:59 Investigated 6 processes this round, 1000ms to go.
2016-05-03 07:13:59 Investigated 6 processes this round, 500ms to go.
2016-05-03 07:14:00 Found 6 suspicious processes.
2016-05-03 07:14:00 [{pid,<10283.17060.123>},
                     {registered_name,[]},
                     {current_stacktrace,
                         [{gen,do_call,4,[{file,"gen.erl"},{line,168}]},
                          {gen_fsm,sync_send_all_state_event,3,
                              [{file,"gen_fsm.erl"},{line,253}]},
                          {ssl_connection,sync_send_all_state_event,2,
                              [{file,"ssl_connection.erl"},{line,1669}]},
                          {rabbit_net,'-async_recv/3-fun-0-',5,
                              [{file,"src/rabbit_net.erl"},{line,149}]}]},
                     {initial_call,{erlang,apply,2}},
                     {message_queue_len,0},
                     {links,[]},
                     {monitors,[{process,<10283.20341.94>}]},
                     {monitored_by,[]},
                     {heap_size,233}]
2016-05-03 07:14:00 [{pid,<10283.17354.123>},
                     {registered_name,[]},
                     {current_stacktrace,
                         [{gen,do_call,4,[{file,"gen.erl"},{line,168}]},
                          {gen_fsm,sync_send_all_state_event,3,
                              [{file,"gen_fsm.erl"},{line,253}]},
                          {ssl_connection,sync_send_all_state_event,2,
                              [{file,"ssl_connection.erl"},{line,1669}]},
                          {rabbit_net,'-async_recv/3-fun-0-',5,
                              [{file,"src/rabbit_net.erl"},{line,149}]}]},
                     {initial_call,{erlang,apply,2}},
                     {message_queue_len,0},
                     {links,[]},
                     {monitors,[{process,<10283.18446.95>}]},
                     {monitored_by,[]},
                     {heap_size,233}]
2016-05-03 07:14:00 [{pid,<10283.17381.123>},
                     {registered_name,[]},
                     {current_stacktrace,
                         [{gen,do_call,4,[{file,"gen.erl"},{line,168}]},
                          {gen_fsm,sync_send_all_state_event,3,
                              [{file,"gen_fsm.erl"},{line,253}]},
                          {ssl_connection,sync_send_all_state_event,2,
                              [{file,"ssl_connection.erl"},{line,1669}]},
                          {rabbit_net,'-async_recv/3-fun-0-',5,
                              [{file,"src/rabbit_net.erl"},{line,149}]}]},
                     {initial_call,{erlang,apply,2}},
                     {message_queue_len,0},
                     {links,[]},
                     {monitors,[{process,<10283.9893.87>}]},
                     {monitored_by,[]},
                     {heap_size,233}]
2016-05-03 07:14:00 [{pid,<10283.17442.123>},
                     {registered_name,[]},
                     {current_stacktrace,
                         [{gen,do_call,4,[{file,"gen.erl"},{line,168}]},
                          {gen_fsm,sync_send_all_state_event,3,
                              [{file,"gen_fsm.erl"},{line,253}]},
                          {ssl_connection,sync_send_all_state_event,2,
                              [{file,"ssl_connection.erl"},{line,1669}]},
                          {rabbit_net,'-async_recv/3-fun-0-',5,
                              [{file,"src/rabbit_net.erl"},{line,149}]}]},
                     {initial_call,{erlang,apply,2}},
                     {message_queue_len,0},
                     {links,[]},
                     {monitors,[{process,<10283.27762.122>}]},
                     {monitored_by,[]},
                     {heap_size,233}]
2016-05-03 07:14:00 [{pid,<10283.17489.123>},
                     {registered_name,[]},
                     {current_stacktrace,
                         [{gen,do_call,4,[{file,"gen.erl"},{line,168}]},
                          {gen_fsm,sync_send_all_state_event,3,
                              [{file,"gen_fsm.erl"},{line,253}]},
                          {ssl_connection,sync_send_all_state_event,2,
                              [{file,"ssl_connection.erl"},{line,1669}]},
                          {rabbit_net,'-async_recv/3-fun-0-',5,
                              [{file,"src/rabbit_net.erl"},{line,149}]}]},
                     {initial_call,{erlang,apply,2}},
                     {message_queue_len,0},
                     {links,[]},
                     {monitors,[{process,<10283.18306.95>}]},
                     {monitored_by,[]},
                     {heap_size,233}]
2016-05-03 07:14:00 [{pid,<10283.17580.123>},
                     {registered_name,[]},
                     {current_stacktrace,
                         [{gen,do_call,4,[{file,"gen.erl"},{line,168}]},
                          {gen_fsm,sync_send_all_state_event,3,
                              [{file,"gen_fsm.erl"},{line,253}]},
                          {ssl_connection,sync_send_all_state_event,2,
                              [{file,"ssl_connection.erl"},{line,1669}]},
                          {rabbit_net,'-async_recv/3-fun-0-',5,
                              [{file,"src/rabbit_net.erl"},{line,149}]}]},
                     {initial_call,{erlang,apply,2}},
                     {message_queue_len,0},
                     {links,[]},
                     {monitors,[{process,<10283.14939.100>}]},
                     {monitored_by,[]},
                     {heap_size,233}]

Is this normal?

[Michael Klishin]

Please don't assume that maybe_stuck/1 is particularly smart: all it does is listing processes

that don't have their stack trace changed in N seconds or after N attempts. This is not necessarily

an indication of an issue.

There are processes in any Erlang system that do nothing but wait for and monitor other processes. They

can be supervisor processes or something else.

Specifically for gen:do_call/4, those calls can be different. Even if they are not, they are part of

[TLS] connection termination, which will be cleaned up by the heartbeat mechanism even if one of

the calls gets no response (e.g. because target process is dead or on an unavailable node).

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[John F]

Okay, thanks for the clarification.