Identifying a "client IP" process
82 views
Skip to first unread message
Johan Rhodin
May 22, 2019, 4:03:56 PM5/22/19
to rabbitm...@googlegroups.com
Hi,
[E*] rabbitmq_delayed_message_exchange 20171201-3.7.x
[E*] rabbitmq_management 3.7.8
[E*] rabbitmq_shovel 3.7.8
[E*] rabbitmq_shovel_management 3.7.8
[E*] rabbitmq_stomp 3.7.8
[E*] rabbitmq_top 3.7.8
[E*] rabbitmq_web_stomp 3.7.8
- RabbitMQ 3.7.8, Erlang 20.1, 2 node-cluster.
In the attached image we had a process that I'd appreciate some help in identifying what it is. Shortly after the image was taken the node restarted (due to OOM) and the problem went away, so I don't have any possibility of getting more live metrics.
It looks like the process is a connection process, but it has the IP-address of the local node (-01) and shows as "client IP:port (1)" instead of the usual "IP:port" that is frequently seen with connections. The PID is not showing in the logs.
/Johan
Enabled plugins:
[E*] rabbitmq_amqp1_0 3.7.8
[E*] rabbitmq_consistent_hash_exchange 3.7.8[E*] rabbitmq_delayed_message_exchange 20171201-3.7.x
[E*] rabbitmq_management 3.7.8
[E*] rabbitmq_shovel 3.7.8
[E*] rabbitmq_shovel_management 3.7.8
[E*] rabbitmq_stomp 3.7.8
[E*] rabbitmq_top 3.7.8
[E*] rabbitmq_web_stomp 3.7.8
Karl Nilsson
May 23, 2019, 3:04:08 AM5/23/19
to rabbitm...@googlegroups.com
Do they have any shovels configured?
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/D7A7A246-DABA-4081-9F20-675DB05437C1%40cloudamqp.com.
For more options, visit https://groups.google.com/d/optout.
Karl Nilsson
Johan Rhodin
May 23, 2019, 12:07:44 PM5/23/19
to rabbitm...@googlegroups.com
Thanks Karl!
Yes at least one shovel. No errors are reported about it, but could very well be that it accumulated messages that for one reason or another couldn't be delivered.
(Looking into another node I see that this "client IP" pattern is common for the shovel)
/Johan
> On May 23, 2019, at 2:03 AM, Karl Nilsson <kjni...@gmail.com> wrote:
>
> Do they have any shovels configured?
>
> On Wed, 22 May 2019 at 21:03, Johan Rhodin <jo...@84codes.com> wrote:
> Hi,
>
> - RabbitMQ 3.7.8, Erlang 20.1, 2 node-cluster.
>
> In the attached image we had a process that I'd appreciate some help in identifying what it is. Shortly after the image was taken the node restarted (due to OOM) and the problem went away, so I don't have any possibility of getting more live metrics.
>
> It looks like the process is a connection process, but it has the IP-address of the local node (-01) and shows as "client IP:port (1)" instead of the usual "IP:port" that is frequently seen with connections. The PID is not showing in the logs.
>
> /Johan
>
> <image.png>
Yes at least one shovel. No errors are reported about it, but could very well be that it accumulated messages that for one reason or another couldn't be delivered.
(Looking into another node I see that this "client IP" pattern is common for the shovel)
/Johan
> On May 23, 2019, at 2:03 AM, Karl Nilsson <kjni...@gmail.com> wrote:
>
> Do they have any shovels configured?
>
> On Wed, 22 May 2019 at 21:03, Johan Rhodin <jo...@84codes.com> wrote:
> Hi,
>
> - RabbitMQ 3.7.8, Erlang 20.1, 2 node-cluster.
>
> In the attached image we had a process that I'd appreciate some help in identifying what it is. Shortly after the image was taken the node restarted (due to OOM) and the problem went away, so I don't have any possibility of getting more live metrics.
>
> It looks like the process is a connection process, but it has the IP-address of the local node (-01) and shows as "client IP:port (1)" instead of the usual "IP:port" that is frequently seen with connections. The PID is not showing in the logs.
>
> /Johan
>
>
>
> Enabled plugins:
> [E*] rabbitmq_amqp1_0 3.7.8
> [E*] rabbitmq_consistent_hash_exchange 3.7.8
> [E*] rabbitmq_delayed_message_exchange 20171201-3.7.x
> [E*] rabbitmq_management 3.7.8
> [E*] rabbitmq_shovel 3.7.8
> [E*] rabbitmq_shovel_management 3.7.8
> [E*] rabbitmq_stomp 3.7.8
> [E*] rabbitmq_top 3.7.8
> [E*] rabbitmq_web_stomp 3.7.8
>
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send email to rabbitm...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/D7A7A246-DABA-4081-9F20-675DB05437C1%40cloudamqp.com.
> For more options, visit https://groups.google.com/d/optout.
> --
> Karl Nilsson
>
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send email to rabbitm...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAHC35TArrLk7ooQigMuh6AFQ6T46-n0-mmq6sZEqdecS4SxKYQ%40mail.gmail.com.
>
> Enabled plugins:
> [E*] rabbitmq_amqp1_0 3.7.8
> [E*] rabbitmq_consistent_hash_exchange 3.7.8
> [E*] rabbitmq_delayed_message_exchange 20171201-3.7.x
> [E*] rabbitmq_management 3.7.8
> [E*] rabbitmq_shovel 3.7.8
> [E*] rabbitmq_shovel_management 3.7.8
> [E*] rabbitmq_stomp 3.7.8
> [E*] rabbitmq_top 3.7.8
> [E*] rabbitmq_web_stomp 3.7.8
>
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send email to rabbitm...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/D7A7A246-DABA-4081-9F20-675DB05437C1%40cloudamqp.com.
> For more options, visit https://groups.google.com/d/optout.
> --
> Karl Nilsson
>
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send email to rabbitm...@googlegroups.com.
Michael Klishin
Jun 30, 2019, 8:01:06 PM6/30/19
to rabbitmq-users
It's a connection writer, the entity that takes protocol frames and writes them to the socket.
While it could be a Shovel it could also be any other connection.
Erlang observer can be used to inspect connection process tree and help identify the connection from its writer.
There is no way to do that in the management UI.
> To view this discussion on the web, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_msgid_rabbitmq-2Dusers_D7A7A246-2DDABA-2D4081-2D9F20-2D675DB05437C1-2540cloudamqp.com&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=wK8593osN75HgBaGPuOJ8rdBO5dnseij4hzid56kb34&m=UWBEC5E2FEwfvNgIvPfBm6j04C5re-EoD3_NrSPDO4Y&s=Zcd7muiryjZtko9oDYHGsASAXPQOnw1noc58RlxHFnQ&e= .
> For more options, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_optout&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=wK8593osN75HgBaGPuOJ8rdBO5dnseij4hzid56kb34&m=UWBEC5E2FEwfvNgIvPfBm6j04C5re-EoD3_NrSPDO4Y&s=8qTtSD7d3pr1BUKAm61dx5gXeAnrmeUHNRrMfLma4Ck&e= .
> --
> Karl Nilsson
>
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send email to rabbitm...@googlegroups.com.
> To view this discussion on the web, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_msgid_rabbitmq-2Dusers_CAHC35TArrLk7ooQigMuh6AFQ6T46-2Dn0-2Dmmq6sZEqdecS4SxKYQ-2540mail.gmail.com&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=wK8593osN75HgBaGPuOJ8rdBO5dnseij4hzid56kb34&m=UWBEC5E2FEwfvNgIvPfBm6j04C5re-EoD3_NrSPDO4Y&s=H4vp8SiLPhip0po0WpSdhk5ISGJ2m8ovflSmd4afAnE&e= .
> For more options, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_optout&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=wK8593osN75HgBaGPuOJ8rdBO5dnseij4hzid56kb34&m=UWBEC5E2FEwfvNgIvPfBm6j04C5re-EoD3_NrSPDO4Y&s=8qTtSD7d3pr1BUKAm61dx5gXeAnrmeUHNRrMfLma4Ck&e= .
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send an email to rabbitm...@googlegroups.com.
To view this discussion on the web, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_msgid_rabbitmq-2Dusers_D6CAE796-2D5277-2D4A3E-2DB447-2D151C76462578-2540cloudamqp.com&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=wK8593osN75HgBaGPuOJ8rdBO5dnseij4hzid56kb34&m=UWBEC5E2FEwfvNgIvPfBm6j04C5re-EoD3_NrSPDO4Y&s=Bd5T_nLBuYFJres87oCfiWy2OYWwQts5j25q6_097lE&e= .
For more options, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_optout&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=wK8593osN75HgBaGPuOJ8rdBO5dnseij4hzid56kb34&m=UWBEC5E2FEwfvNgIvPfBm6j04C5re-EoD3_NrSPDO4Y&s=8qTtSD7d3pr1BUKAm61dx5gXeAnrmeUHNRrMfLma4Ck&e= .
--
MK
Staff Software Engineer, Pivotal/RabbitMQ
Johan Rhodin
Feb 17, 2020, 7:15:21 PM2/17/20
to rabbitmq-users
Stumbled on this issue again (this time in RabbitMQ 3.7.10 and Erlang 21.2.3) where most of the memory was in binaries:
Looking in RabbitMQ top we can identify the client writer:
Trying to look in observer on this PID unfortunately crashes observer.
But we can get the process_info for this PID:
rabbitmqctl eval 'process_info(c:pid(0,16100,1)).'[{current_function,{erts_internal,port_command,3}},
{initial_call,{proc_lib,init_p,5}},
{status,suspended},
{message_queue_len,1465720},
{links,[<9854.16097.1>]},
{dictionary,
[{process_name,
{rabbit_writer,
{<<"client SERVER_IP:XXXX9 -> REMOTE_IP:5672">>,1}}},
{'$initial_call',{rabbit_writer,enter_mainloop,2}},
{msg_size_for_gc,539730},
{'$ancestors',
[<9854.16097.1>,<9854.16085.1>,<9854.16083.1>,<9854.16082.1>,
amqp_sup,<9854.290.0>]}]},
{trap_exit,false},
{error_handler,error_handler},
{priority,normal},
{group_leader,<9854.289.0>},
{total_heap_size,45988046},
{heap_size,45988046},
{stack_size,17},
{reductions,18744280649},
{garbage_collection,
[{max_heap_size,#{error_logger => true,kill => true,size => 0}},
{min_bin_vheap_size,46422},
{min_heap_size,233},
{fullsweep_after,65535},
{minor_gcs,0}]},
{suspending,[]}]
In the process_name we have the remote IP. By using inet:i() we can find which Erlang PID are handling connections to that host
rabbitmqctl eval 'inet:i().' | grep REMOTE_IP
48000 inet_tcp 35161 9290357 <0.16067.1> SERVER_IP:YYYYYY REMOTE_IP:amqp ???? STREAM
48008 inet_tcp 35158 9317634326 <0.16084.1> SERVER_IP:XXXX9 REMOTE_IP:amqp ???? STREAM
The second one is the one we are looking for (matching port).
By using process_info() for that PID and looking at the links and their ancestors (in observer), we can find the shovel name it corresponds to.
/Johan
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.
> To post to this group, send email to rabbitm...@googlegroups.com.
> To view this discussion on the web, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_msgid_rabbitmq-2Dusers_D7A7A246-2DDABA-2D4081-2D9F20-2D675DB05437C1-2540cloudamqp.com&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=wK8593osN75HgBaGPuOJ8rdBO5dnseij4hzid56kb34&m=UWBEC5E2FEwfvNgIvPfBm6j04C5re-EoD3_NrSPDO4Y&s=Zcd7muiryjZtko9oDYHGsASAXPQOnw1noc58RlxHFnQ&e= .
> For more options, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_optout&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=wK8593osN75HgBaGPuOJ8rdBO5dnseij4hzid56kb34&m=UWBEC5E2FEwfvNgIvPfBm6j04C5re-EoD3_NrSPDO4Y&s=8qTtSD7d3pr1BUKAm61dx5gXeAnrmeUHNRrMfLma4Ck&e= .
> --
> Karl Nilsson
>
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.
> To post to this group, send email to rabbitm...@googlegroups.com.
> To view this discussion on the web, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_msgid_rabbitmq-2Dusers_CAHC35TArrLk7ooQigMuh6AFQ6T46-2Dn0-2Dmmq6sZEqdecS4SxKYQ-2540mail.gmail.com&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=wK8593osN75HgBaGPuOJ8rdBO5dnseij4hzid56kb34&m=UWBEC5E2FEwfvNgIvPfBm6j04C5re-EoD3_NrSPDO4Y&s=H4vp8SiLPhip0po0WpSdhk5ISGJ2m8ovflSmd4afAnE&e= .
> For more options, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_optout&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=wK8593osN75HgBaGPuOJ8rdBO5dnseij4hzid56kb34&m=UWBEC5E2FEwfvNgIvPfBm6j04C5re-EoD3_NrSPDO4Y&s=8qTtSD7d3pr1BUKAm61dx5gXeAnrmeUHNRrMfLma4Ck&e= .
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.
To post to this group, send an email to rabbitm...@googlegroups.com.
To view this discussion on the web, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_msgid_rabbitmq-2Dusers_D6CAE796-2D5277-2D4A3E-2DB447-2D151C76462578-2540cloudamqp.com&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=wK8593osN75HgBaGPuOJ8rdBO5dnseij4hzid56kb34&m=UWBEC5E2FEwfvNgIvPfBm6j04C5re-EoD3_NrSPDO4Y&s=Bd5T_nLBuYFJres87oCfiWy2OYWwQts5j25q6_097lE&e= .
For more options, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_optout&d=DwIBaQ&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=wK8593osN75HgBaGPuOJ8rdBO5dnseij4hzid56kb34&m=UWBEC5E2FEwfvNgIvPfBm6j04C5re-EoD3_NrSPDO4Y&s=8qTtSD7d3pr1BUKAm61dx5gXeAnrmeUHNRrMfLma4Ck&e= .
--MKStaff Software Engineer, Pivotal/RabbitMQ
Johan Rhodin
Feb 19, 2020, 3:22:13 PM2/19/20
to rabbitm...@googlegroups.com
For completeness I want to complement my reply that the "in observer" part here: "By using process_info() for that PID and looking at the links and their ancestors (in observer), we can find the shovel name it corresponds to" is not strictly needed to get the shovel name, one can use recon:get_state to get that information:
rabbitmqctl eval 'recon:get_state(c:pid(0,815,0)).'
or
rabbitmqctl eval 'sys:get_state(c:pid(0,815,0)).'
/Johan
> On Feb 17, 2020, at 6:15 PM, Johan Rhodin <jo...@cloudamqp.com> wrote:
>
> Stumbled on this issue again (this time in RabbitMQ 3.7.10 and Erlang 21.2.3) where most of the memory was in binaries:
>
> <rabbitmq-top-client-writer.png>
> To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/4fbab3c9-18da-42f6-84f0-f5241f82eb20%40googlegroups.com.
> <RabbitMQ-Management-BINARIES.png><rabbitmq-top-client-writer.png><rabbitmq-top-client-writer.png>
rabbitmqctl eval 'recon:get_state(c:pid(0,815,0)).'
or
rabbitmqctl eval 'sys:get_state(c:pid(0,815,0)).'
/Johan
> On Feb 17, 2020, at 6:15 PM, Johan Rhodin <jo...@cloudamqp.com> wrote:
>
> Stumbled on this issue again (this time in RabbitMQ 3.7.10 and Erlang 21.2.3) where most of the memory was in binaries:
>
> <RabbitMQ-Management-BINARIES.png>
>
>
> Looking in RabbitMQ top we can identify the client writer:
>
>
>
> Looking in RabbitMQ top we can identify the client writer:
>
>
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/4fbab3c9-18da-42f6-84f0-f5241f82eb20%40googlegroups.com.
> <RabbitMQ-Management-BINARIES.png><rabbitmq-top-client-writer.png><rabbitmq-top-client-writer.png>
Reply all
Reply to author
Forward
0 new messages