CPU usage - 10K connections / queues
554 views
Skip to first unread message
Laurent Luce
Mar 15, 2018, 10:54:32 AM3/15/18
to rabbitmq-users
Hello,
I am running RMQ 3.6.12. I have 10K active connections (web-stomp), 10K queues. Each queue is replicated once. 4 RMQ nodes. No messages are sent during this test. I am trying to find out where CPU slices are spent and I got some information using the rabbitmq_top plugin. Results are sorted using "reductions". The top five processes are:
- queue_metrics_metrics_collector
- queue_coarse_metrics_metrics_collector
- rabbit_mgmt_db_cache_connections
- connection_created_metrics_collector
- consumer_created_metrics_collector
I would like to know what those processes do exactly and why they are using the most CPU. collecting_stats is set to None. Below is the rabbitmq_top output and the output of the rabbitmqctl report command.
{
"node":"rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld",
"row_count":20,
"processes":[
{
"pid":"<0.843.0>",
"name":{
"type":"registered",
"name":"queue_metrics_metrics_collector"
},
"reduction_delta":1813795,
"memory":56180792,
"message_queue_len":0,
"reductions":11599671234,
"status":"runnable"
},
{
"pid":"<0.837.0>",
"name":{
"type":"registered",
"name":"channel_metrics_metrics_collector"
},
"reduction_delta":0,
"memory":11170704,
"message_queue_len":0,
"reductions":4210612754,
"status":"waiting"
},
{
"pid":"<0.844.0>",
"name":{
"type":"registered",
"name":"queue_coarse_metrics_metrics_collector"
},
"reduction_delta":0,
"memory":13089992,
"message_queue_len":0,
"reductions":3001707729,
"status":"waiting"
},
{
"pid":"<0.1128.0>",
"name":{
"type":"registered",
"name":"rabbit_mgmt_db_cache_connections"
},
"reduction_delta":0,
"memory":97079600,
"message_queue_len":0,
"reductions":2868528027,
"status":"waiting"
},
{
"pid":"<0.833.0>",
"name":{
"type":"registered",
"name":"connection_created_metrics_collector"
},
"reduction_delta":0,
"memory":4720656,
"message_queue_len":0,
"reductions":2555040101,
"status":"waiting"
},
{
"pid":"<0.842.0>",
"name":{
"type":"registered",
"name":"consumer_created_metrics_collector"
},
"reduction_delta":0,
"memory":2917920,
"message_queue_len":0,
"reductions":1335934025,
"status":"waiting"
},
{
"pid":"<0.192.0>",
"name":{
"type":"unknown",
"name":"{tls_connection,init,1}"
},
"reduction_delta":93,
"memory":47424,
"message_queue_len":0,
"reductions":1097245854,
"status":"waiting"
},
{
"pid":"<0.182.0>",
"name":{
"type":"unknown",
"name":"{tls_connection,init,1}"
},
"reduction_delta":57438,
"memory":101608,
"message_queue_len":0,
"reductions":1072815571,
"status":"waiting"
},
{
"pid":"<0.187.0>",
"name":{
"type":"unknown",
"name":"{tls_connection,init,1}"
},
"reduction_delta":88,
"memory":101608,
"message_queue_len":0,
"reductions":1064337543,
"status":"waiting"
},
{
"pid":"<0.826.0>",
"name":{
"type":"registered",
"name":"delegate_management_0"
},
"buffer_len":0,
"reduction_delta":298027,
"memory":9571296,
"message_queue_len":0,
"reductions":647296012,
"status":"waiting"
},
{
"pid":"<0.836.0>",
"name":{
"type":"registered",
"name":"channel_created_metrics_collector"
},
"reduction_delta":0,
"memory":1803768,
"message_queue_len":0,
"reductions":623961165,
"status":"waiting"
},
{
"pid":"<0.827.0>",
"name":{
"type":"registered",
"name":"delegate_management_1"
},
"buffer_len":0,
"reduction_delta":0,
"memory":2416,
"message_queue_len":0,
"reductions":621485421,
"status":"waiting"
},
{
"pid":"<0.830.0>",
"name":{
"type":"registered",
"name":"delegate_management_4"
},
"buffer_len":0,
"reduction_delta":0,
"memory":2416,
"message_queue_len":0,
"reductions":620203956,
"status":"waiting"
},
{
"pid":"<0.776.0>",
"name":{
"type":"registered",
"name":"rabbit_core_metrics_gc"
},
"reduction_delta":0,
"memory":89682560,
"message_queue_len":0,
"reductions":606140310,
"status":"waiting"
},
{
"pid":"<0.841.0>",
"name":{
"type":"registered",
"name":"channel_process_metrics_metrics_collector"
},
"reduction_delta":0,
"memory":1115184,
"message_queue_len":0,
"reductions":577270344,
"status":"waiting"
},
{
"pid":"<0.834.0>",
"name":{
"type":"registered",
"name":"connection_metrics_metrics_collector"
},
"reduction_delta":0,
"memory":4720656,
"message_queue_len":0,
"reductions":503582853,
"status":"waiting"
},
{
"pid":"<0.832.0>",
"name":{
"type":"registered",
"name":"rabbit_mgmt_gc"
},
"reduction_delta":0,
"memory":16365432,
"message_queue_len":0,
"reductions":503043783,
"status":"waiting"
},
{
"pid":"<0.11200.21>",
"name":{
"type":"registered",
"name":"rabbit_top_worker"
},
"reduction_delta":1307433,
"memory":289875912,
"message_queue_len":0,
"reductions":449649621,
"status":"running"
},
{
"pid":"<0.186.0>",
"name":{
"type":"unknown",
"name":"{erlang,apply,2}"
},
"reduction_delta":14,
"memory":978208,
"message_queue_len":0,
"reductions":111578590,
"status":"waiting"
},
{
"pid":"<0.191.0>",
"name":{
"type":"unknown",
"name":"{erlang,apply,2}"
},
"reduction_delta":14,
"memory":251248,
"message_queue_len":0,
"reductions":109594594,
"status":"waiting"
}
]
}
Report:
Status of node 'rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s'
[{pid,772},
{running_applications,
[{rabbitmq_management,"RabbitMQ Management Console","3.6.12"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.12"},
{rabbitmq_web_stomp,"Rabbit WEB-STOMP - WebSockets to Stomp adapter",
"3.6.12"},
{rabbitmq_stomp,"RabbitMQ STOMP plugin","3.6.12"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.12"},
{rabbit,"RabbitMQ","3.6.12"},
{mnesia,"MNESIA CXC 138 12","4.15.1"},
{cowboy,"Small, fast, modular HTTP server.","1.0.4"},
{ranch,"Socket acceptor pool for TCP protocols.","1.3.0"},
{autocluster,
"Forms RabbitMQ clusters using a variety of backends (AWS EC2, DNS, Consul, Kubernetes, etc)",
"0.8.0"},
{rabbitmq_aws,
"A minimalistic AWS API interface used by rabbitmq-autocluster (3.6.x) and other RabbitMQ plugins",
"3.6.11.milestone1"},
{ssl,"Erlang/OTP SSL application","8.2.2"},
{public_key,"Public key infrastructure","1.5.1"},
{asn1,"The Erlang ASN1 compiler version 5.0.3","5.0.3"},
{sockjs,"SockJS","0.3.4"},
{cowlib,"Support library for manipulating Web protocols.","1.0.2"},
{crypto,"CRYPTO","4.1"},
{amqp_client,"RabbitMQ AMQP Client","3.6.12"},
{rabbit_common,
"Modules shared by rabbitmq-server and rabbitmq-erlang-client",
"3.6.12"},
{os_mon,"CPO CXC 138 46","2.4.3"},
{compiler,"ERTS CXC 138 10","7.1.3"},
{xmerl,"XML parser","1.3.15"},
{syntax_tools,"Syntax tools","2.1.3"},
{inets,"INETS CXC 138 49","6.4.4"},
{sasl,"SASL CXC 138 11","3.1"},
{stdlib,"ERTS CXC 138 10","3.4.2"},
{kernel,"ERTS CXC 138 10","5.4"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang/OTP 20 [erts-9.1.5] [source] [64-bit] [smp:36:36] [ds:36:36:10] [async-threads:576] [hipe] [kernel-poll:true]\n"},
{memory,
[{connection_readers,0},
{connection_writers,0},
{connection_channels,0},
{connection_other,170066024},
{queue_procs,286836640},
{queue_slave_procs,116755456},
{plugins,459118512},
{other_proc,299009856},
{metrics,28850824},
{mgmt_db,121183688},
{mnesia,386393296},
{other_ets,23833408},
{binary,114050360},
{msg_index,11992120},
{code,25416876},
{atom,1041593},
{other_system,46904203},
{total,1024}]},
{alarms,[]},
{listeners,
[{clustering,25672,"::"},
{'http/web-stomp',15674,"::"},
{'https/web-stomp',9451,"::"},
{https,9450,"::"}]},
{vm_memory_calculation_strategy,rss},
{vm_memory_high_watermark,0.75},
{vm_memory_limit,47157202944},
{disk_free_limit,50000000},
{disk_free,12298194944},
{file_descriptors,
[{total_limit,65436},
{total_used,1935},
{sockets_limit,58890},
{sockets_used,1933}]},
{processes,[{limit,1048576},{used,87168}]},
{run_queue,0},
{uptime,137737},
{kernel,{net_ticktime,60}}]
Cluster status of node 'rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s'
[{nodes,[{disc,['rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld']}]},
{running_nodes,['rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s']},
{cluster_name,<<"rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh">>},
{partitions,[]},
{alarms,[{'rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld',[]},
{'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh',[]},
{'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn',[]},
{'rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s',[]}]}]
Application environment of node 'rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s'
[{amqp_client,[{prefer_ipv6,false},{ssl_options,[]}]},
{asn1,[]},
{autocluster,
[{autocluster_failure,ignore},
{autocluster_log_level,info},
{backend,consul},
{cleanup_warn_only,false},
{cluster_cleanup,true},
{consul_port,8500},
{consul_svc_addr_auto,true},
{consul_svc_port,9450},
{startup_steps_state,
{startup_state,consul,autocluster_consul,
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn',
"65b23fe5-ea9b-e3e3-f669-e5bd19ddea2a"}}]},
{compiler,[]},
{cowboy,[]},
{cowlib,[]},
{crypto,[{fips_mode,false}]},
{inets,[]},
{kernel,
[{error_logger,tty},
{inet_default_connect_options,[{nodelay,true}]},
{inet_dist_listen_max,25672},
{inet_dist_listen_min,25672}]},
{mnesia,
[{dir,
"/srv/rabbitmq_server-3.6.12/var/lib/rabbitmq/mnesia/rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s"},
{dump_log_write_threshold,256}]},
{os_mon,
[{start_cpu_sup,false},
{start_disksup,false},
{start_memsup,false},
{start_os_sup,false}]},
{public_key,[]},
{rabbit,
[{auth_backends,[rabbit_auth_backend_internal]},
{auth_mechanisms,['PLAIN','AMQPLAIN']},
{background_gc_enabled,false},
{background_gc_target_interval,60000},
{backing_queue_module,rabbit_priority_queue},
{channel_max,0},
{channel_operation_timeout,15000},
{cluster_keepalive_interval,10000},
{cluster_nodes,{[],disc}},
{cluster_partition_handling,autoheal},
{collect_statistics,coarse},
{collect_statistics_interval,60000},
{config_entry_decoder,
[{cipher,aes_cbc256},
{hash,sha512},
{iterations,1000},
{passphrase,undefined}]},
{credit_flow_default_credit,{400,200}},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
{default_user,<<"guest">>},
{default_user_tags,[administrator]},
{default_vhost,<<"/">>},
{delegate_count,16},
{disk_free_limit,50000000},
{disk_monitor_failure_retries,10},
{disk_monitor_failure_retry_interval,120000},
{enabled_plugins_file,
"/srv/rabbitmq_server-3.6.12/etc/rabbitmq/enabled_plugins"},
{error_logger,tty},
{fhc_read_buffering,false},
{fhc_write_buffering,true},
{frame_max,131072},
{halt_on_upgrade_failure,true},
{handshake_timeout,10000},
{heartbeat,60},
{hipe_compile,false},
{hipe_modules,
[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
rabbit_command_assembler,rabbit_framing_amqp_0_9_1,rabbit_basic,
rabbit_event,lists,queue,priority_queue,rabbit_router,rabbit_trace,
rabbit_misc,rabbit_binary_parser,rabbit_exchange_type_direct,
rabbit_guid,rabbit_net,rabbit_amqqueue_process,
rabbit_variable_queue,rabbit_binary_generator,rabbit_writer,
delegate,gb_sets,lqueue,sets,orddict,rabbit_amqqueue,
rabbit_limiter,gb_trees,rabbit_queue_index,
rabbit_exchange_decorator,gen,dict,ordsets,file_handle_cache,
rabbit_msg_store,array,rabbit_msg_store_ets_index,rabbit_msg_file,
rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,
mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,
ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,ssl]},
{lazy_queue_explicit_gc_run_operation_threshold,1000},
{log_levels,[{connection,info},{autocluster,info}]},
{loopback_users,[<<"guest">>]},
{memory_monitor_interval,2500},
{mirroring_flow_control,true},
{mirroring_sync_batch_size,4096},
{mnesia_table_loading_retry_limit,10},
{mnesia_table_loading_retry_timeout,30000},
{msg_store_credit_disc_bound,{4000,800}},
{msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index},
{msg_store_io_batch_size,4096},
{num_ssl_acceptors,1},
{num_tcp_acceptors,10},
{password_hashing_module,rabbit_password_hashing_sha256},
{plugins_dir,"/srv/rabbitmq_server-3.6.12/plugins"},
{plugins_expand_dir,
"/srv/rabbitmq_server-3.6.12/var/lib/rabbitmq/mnesia/rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s-plugins-expand"},
{queue_explicit_gc_run_operation_threshold,1000},
{queue_index_embed_msgs_below,4096},
{queue_index_max_journal_entries,32768},
{reverse_dns_lookups,false},
{sasl_error_logger,tty},
{server_properties,[]},
{ssl_allow_poodle_attack,false},
{ssl_apps,[asn1,crypto,public_key,ssl]},
{ssl_cert_login_from,distinguished_name},
{ssl_handshake_timeout,5000},
{ssl_listeners,[]},
{ssl_options,[]},
{tcp_listen_options,
[{backlog,128},
{nodelay,true},
{linger,{true,0}},
{exit_on_close,false}]},
{tcp_listeners,[]},
{trace_vhosts,[]},
{vm_memory_calculation_strategy,rss},
{vm_memory_high_watermark,0.75},
{vm_memory_high_watermark_paging_ratio,0.9}]},
{rabbit_common,[]},
{rabbitmq_aws,[]},
{rabbitmq_management,
[{cors_allow_origins,[]},
{cors_max_age,1800},
{http_log_dir,none},
{listener,
[{port,9450},
{ssl,true},
{ssl_opts,
[{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
{certfile,"/etc/rabbitmq/ssl/server.pem"},
{keyfile,"/etc/rabbitmq/ssl/server.key"},
{versions,['tlsv1.2','tlsv1.1']},
{dhfile,"/etc/rabbitmq/ssl/dh-params.pem"},
{ciphers,
[{ecdhe_ecdsa,aes_128_gcm,null,sha256},
{ecdhe_rsa,aes_128_gcm,null,sha256}]}]}]},
{load_definitions,none},
{management_db_cache_multiplier,5},
{process_stats_gc_timeout,300000},
{rates_mode,none},
{sample_retention_policies,
[{global,[{60,5}]},{basic,[{60,20}]},{detailed,[{5,5}]}]},
{stats_event_max_backlog,250}]},
{rabbitmq_management_agent,
[{rates_mode,basic},
{sample_retention_policies,
[{global,[{605,5},{3660,60},{29400,600},{86400,1800}]},
{basic,[{605,5},{3600,60}]},
{detailed,[{605,5}]}]}]},
{rabbitmq_stomp,
[{default_user,[{login,<<"guest">>},{passcode,<<"guest">>}]},
{default_vhost,<<"/">>},
{hide_server_info,false},
{implicit_connect,false},
{num_ssl_acceptors,1},
{num_tcp_acceptors,10},
{ssl_cert_login,false},
{ssl_listeners,[]},
{tcp_listen_options,[{backlog,128},{nodelay,true}]},
{tcp_listeners,[]},
{trailing_lf,true}]},
{rabbitmq_web_dispatch,[]},
{rabbitmq_web_stomp,
[{cowboy_opts,[]},
{num_ssl_acceptors,1},
{num_tcp_acceptors,10},
{port,15674},
{sockjs_opts,[]},
{ssl_config,
[{port,9451},
{backlog,1024},
{recbuf,32768},
{sndbuf,32768},
{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
{certfile,"/etc/rabbitmq/ssl/server.pem"},
{keyfile,"/etc/rabbitmq/ssl/server.key"},
{versions,['tlsv1.2','tlsv1.1']},
{dhfile,"/etc/rabbitmq/ssl/dh-params.pem"},
{ciphers,
[{ecdhe_ecdsa,aes_128_gcm,null,sha256},
{ecdhe_rsa,aes_128_gcm,null,sha256}]}]},
{tcp_config,[]},
{use_http_auth,true},
{ws_frame,text}]},
{ranch,[]},
{sasl,[{errlog_type,error},{sasl_error_logger,tty}]},
{sockjs,[]},
{ssl,
[{ciphers,
[{ecdhe_ecdsa,aes_128_gcm,null,sha256},
{ecdhe_rsa,aes_128_gcm,null,sha256}]},
{protocol_version,['tlsv1.2','tlsv1.1',tlsv1]},
{versions,['tlsv1.2','tlsv1.1']}]},
{stdlib,[]},
{syntax_tools,[]},
{xmerl,[]}]
Status of node 'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn'
[{pid,772},
{running_applications,
[{rabbitmq_management,"RabbitMQ Management Console","3.6.12"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.12"},
{rabbitmq_web_stomp,"Rabbit WEB-STOMP - WebSockets to Stomp adapter",
"3.6.12"},
{rabbitmq_stomp,"RabbitMQ STOMP plugin","3.6.12"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.12"},
{rabbit,"RabbitMQ","3.6.12"},
{mnesia,"MNESIA CXC 138 12","4.15.1"},
{cowboy,"Small, fast, modular HTTP server.","1.0.4"},
{ranch,"Socket acceptor pool for TCP protocols.","1.3.0"},
{autocluster,
"Forms RabbitMQ clusters using a variety of backends (AWS EC2, DNS, Consul, Kubernetes, etc)",
"0.8.0"},
{rabbitmq_aws,
"A minimalistic AWS API interface used by rabbitmq-autocluster (3.6.x) and other RabbitMQ plugins",
"3.6.11.milestone1"},
{ssl,"Erlang/OTP SSL application","8.2.2"},
{public_key,"Public key infrastructure","1.5.1"},
{asn1,"The Erlang ASN1 compiler version 5.0.3","5.0.3"},
{sockjs,"SockJS","0.3.4"},
{cowlib,"Support library for manipulating Web protocols.","1.0.2"},
{crypto,"CRYPTO","4.1"},
{amqp_client,"RabbitMQ AMQP Client","3.6.12"},
{rabbit_common,
"Modules shared by rabbitmq-server and rabbitmq-erlang-client",
"3.6.12"},
{os_mon,"CPO CXC 138 46","2.4.3"},
{compiler,"ERTS CXC 138 10","7.1.3"},
{xmerl,"XML parser","1.3.15"},
{syntax_tools,"Syntax tools","2.1.3"},
{inets,"INETS CXC 138 49","6.4.4"},
{sasl,"SASL CXC 138 11","3.1"},
{stdlib,"ERTS CXC 138 10","3.4.2"},
{kernel,"ERTS CXC 138 10","5.4"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang/OTP 20 [erts-9.1.5] [source] [64-bit] [smp:36:36] [ds:36:36:10] [async-threads:576] [hipe] [kernel-poll:true]\n"},
{memory,
[{connection_readers,0},
{connection_writers,0},
{connection_channels,0},
{connection_other,176252672},
{queue_procs,291127808},
{queue_slave_procs,116460608},
{plugins,489360680},
{other_proc,220165456},
{metrics,29568008},
{mgmt_db,157838576},
{mnesia,386373416},
{other_ets,23730304},
{binary,114405648},
{msg_index,12100536},
{code,25416876},
{atom,1041593},
{other_system,47343235},
{total,1024}]},
{alarms,[]},
{listeners,
[{clustering,25672,"::"},
{'http/web-stomp',15674,"::"},
{'https/web-stomp',9451,"::"},
{https,9450,"::"}]},
{vm_memory_calculation_strategy,rss},
{vm_memory_high_watermark,0.75},
{vm_memory_limit,47157202944},
{disk_free_limit,50000000},
{disk_free,12296790016},
{file_descriptors,
[{total_limit,65436},
{total_used,2009},
{sockets_limit,58890},
{sockets_used,2007}]},
{processes,[{limit,1048576},{used,89070}]},
{run_queue,0},
{uptime,137749},
{kernel,{net_ticktime,60}}]
Cluster status of node 'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn'
[{nodes,[{disc,['rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld']}]},
{running_nodes,['rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn']},
{cluster_name,<<"rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh">>},
{partitions,[]},
{alarms,[{'rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld',[]},
{'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh',[]},
{'rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s',[]},
{'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn',[]}]}]
Application environment of node 'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn'
[{amqp_client,[{prefer_ipv6,false},{ssl_options,[]}]},
{asn1,[]},
{autocluster,
[{autocluster_failure,ignore},
{autocluster_log_level,info},
{backend,consul},
{cleanup_warn_only,false},
{cluster_cleanup,true},
{consul_port,8500},
{consul_svc_addr_auto,true},
{consul_svc_port,9450},
{startup_steps_state,
{startup_state,consul,autocluster_consul,undefined,
"288dd426-4c02-2575-5dd2-19ba29b6c153"}}]},
{compiler,[]},
{cowboy,[]},
{cowlib,[]},
{crypto,[{fips_mode,false}]},
{inets,[]},
{kernel,
[{error_logger,tty},
{inet_default_connect_options,[{nodelay,true}]},
{inet_dist_listen_max,25672},
{inet_dist_listen_min,25672}]},
{mnesia,
[{dir,
"/srv/rabbitmq_server-3.6.12/var/lib/rabbitmq/mnesia/rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn"},
{dump_log_write_threshold,256}]},
{os_mon,
[{start_cpu_sup,false},
{start_disksup,false},
{start_memsup,false},
{start_os_sup,false}]},
{public_key,[]},
{rabbit,
[{auth_backends,[rabbit_auth_backend_internal]},
{auth_mechanisms,['PLAIN','AMQPLAIN']},
{background_gc_enabled,false},
{background_gc_target_interval,60000},
{backing_queue_module,rabbit_priority_queue},
{channel_max,0},
{channel_operation_timeout,15000},
{cluster_keepalive_interval,10000},
{cluster_nodes,{[],disc}},
{cluster_partition_handling,autoheal},
{collect_statistics,coarse},
{collect_statistics_interval,60000},
{config_entry_decoder,
[{cipher,aes_cbc256},
{hash,sha512},
{iterations,1000},
{passphrase,undefined}]},
{credit_flow_default_credit,{400,200}},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
{default_user,<<"guest">>},
{default_user_tags,[administrator]},
{default_vhost,<<"/">>},
{delegate_count,16},
{disk_free_limit,50000000},
{disk_monitor_failure_retries,10},
{disk_monitor_failure_retry_interval,120000},
{enabled_plugins_file,
"/srv/rabbitmq_server-3.6.12/etc/rabbitmq/enabled_plugins"},
{error_logger,tty},
{fhc_read_buffering,false},
{fhc_write_buffering,true},
{frame_max,131072},
{halt_on_upgrade_failure,true},
{handshake_timeout,10000},
{heartbeat,60},
{hipe_compile,false},
{hipe_modules,
[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
rabbit_command_assembler,rabbit_framing_amqp_0_9_1,rabbit_basic,
rabbit_event,lists,queue,priority_queue,rabbit_router,rabbit_trace,
rabbit_misc,rabbit_binary_parser,rabbit_exchange_type_direct,
rabbit_guid,rabbit_net,rabbit_amqqueue_process,
rabbit_variable_queue,rabbit_binary_generator,rabbit_writer,
delegate,gb_sets,lqueue,sets,orddict,rabbit_amqqueue,
rabbit_limiter,gb_trees,rabbit_queue_index,
rabbit_exchange_decorator,gen,dict,ordsets,file_handle_cache,
rabbit_msg_store,array,rabbit_msg_store_ets_index,rabbit_msg_file,
rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,
mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,
ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,ssl]},
{lazy_queue_explicit_gc_run_operation_threshold,1000},
{log_levels,[{connection,info},{autocluster,info}]},
{loopback_users,[<<"guest">>]},
{memory_monitor_interval,2500},
{mirroring_flow_control,true},
{mirroring_sync_batch_size,4096},
{mnesia_table_loading_retry_limit,10},
{mnesia_table_loading_retry_timeout,30000},
{msg_store_credit_disc_bound,{4000,800}},
{msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index},
{msg_store_io_batch_size,4096},
{num_ssl_acceptors,1},
{num_tcp_acceptors,10},
{password_hashing_module,rabbit_password_hashing_sha256},
{plugins_dir,"/srv/rabbitmq_server-3.6.12/plugins"},
{plugins_expand_dir,
"/srv/rabbitmq_server-3.6.12/var/lib/rabbitmq/mnesia/rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn-plugins-expand"},
{queue_explicit_gc_run_operation_threshold,1000},
{queue_index_embed_msgs_below,4096},
{queue_index_max_journal_entries,32768},
{reverse_dns_lookups,false},
{sasl_error_logger,tty},
{server_properties,[]},
{ssl_allow_poodle_attack,false},
{ssl_apps,[asn1,crypto,public_key,ssl]},
{ssl_cert_login_from,distinguished_name},
{ssl_handshake_timeout,5000},
{ssl_listeners,[]},
{ssl_options,[]},
{tcp_listen_options,
[{backlog,128},
{nodelay,true},
{linger,{true,0}},
{exit_on_close,false}]},
{tcp_listeners,[]},
{trace_vhosts,[]},
{vm_memory_calculation_strategy,rss},
{vm_memory_high_watermark,0.75},
{vm_memory_high_watermark_paging_ratio,0.9}]},
{rabbit_common,[]},
{rabbitmq_aws,[]},
{rabbitmq_management,
[{cors_allow_origins,[]},
{cors_max_age,1800},
{http_log_dir,none},
{listener,
[{port,9450},
{ssl,true},
{ssl_opts,
[{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
{certfile,"/etc/rabbitmq/ssl/server.pem"},
{keyfile,"/etc/rabbitmq/ssl/server.key"},
{versions,['tlsv1.2','tlsv1.1']},
{dhfile,"/etc/rabbitmq/ssl/dh-params.pem"},
{ciphers,
[{ecdhe_ecdsa,aes_128_gcm,null,sha256},
{ecdhe_rsa,aes_128_gcm,null,sha256}]}]}]},
{load_definitions,none},
{management_db_cache_multiplier,5},
{process_stats_gc_timeout,300000},
{rates_mode,none},
{sample_retention_policies,
[{global,[{60,5}]},{basic,[{60,20}]},{detailed,[{5,5}]}]},
{stats_event_max_backlog,250}]},
{rabbitmq_management_agent,
[{rates_mode,basic},
{sample_retention_policies,
[{global,[{605,5},{3660,60},{29400,600},{86400,1800}]},
{basic,[{605,5},{3600,60}]},
{detailed,[{605,5}]}]}]},
{rabbitmq_stomp,
[{default_user,[{login,<<"guest">>},{passcode,<<"guest">>}]},
{default_vhost,<<"/">>},
{hide_server_info,false},
{implicit_connect,false},
{num_ssl_acceptors,1},
{num_tcp_acceptors,10},
{ssl_cert_login,false},
{ssl_listeners,[]},
{tcp_listen_options,[{backlog,128},{nodelay,true}]},
{tcp_listeners,[]},
{trailing_lf,true}]},
{rabbitmq_web_dispatch,[]},
{rabbitmq_web_stomp,
[{cowboy_opts,[]},
{num_ssl_acceptors,1},
{num_tcp_acceptors,10},
{port,15674},
{sockjs_opts,[]},
{ssl_config,
[{port,9451},
{backlog,1024},
{recbuf,32768},
{sndbuf,32768},
{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
{certfile,"/etc/rabbitmq/ssl/server.pem"},
{keyfile,"/etc/rabbitmq/ssl/server.key"},
{versions,['tlsv1.2','tlsv1.1']},
{dhfile,"/etc/rabbitmq/ssl/dh-params.pem"},
{ciphers,
[{ecdhe_ecdsa,aes_128_gcm,null,sha256},
{ecdhe_rsa,aes_128_gcm,null,sha256}]}]},
{tcp_config,[]},
{use_http_auth,true},
{ws_frame,text}]},
{ranch,[]},
{sasl,[{errlog_type,error},{sasl_error_logger,tty}]},
{sockjs,[]},
{ssl,
[{ciphers,
[{ecdhe_ecdsa,aes_128_gcm,null,sha256},
{ecdhe_rsa,aes_128_gcm,null,sha256}]},
{protocol_version,['tlsv1.2','tlsv1.1',tlsv1]},
{versions,['tlsv1.2','tlsv1.1']}]},
{stdlib,[]},
{syntax_tools,[]},
{xmerl,[]}]
Status of node 'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh'
[{pid,772},
{running_applications,
[{rabbitmq_management,"RabbitMQ Management Console","3.6.12"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.12"},
{rabbitmq_web_stomp,"Rabbit WEB-STOMP - WebSockets to Stomp adapter",
"3.6.12"},
{rabbitmq_stomp,"RabbitMQ STOMP plugin","3.6.12"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.12"},
{rabbit,"RabbitMQ","3.6.12"},
{mnesia,"MNESIA CXC 138 12","4.15.1"},
{cowboy,"Small, fast, modular HTTP server.","1.0.4"},
{ranch,"Socket acceptor pool for TCP protocols.","1.3.0"},
{autocluster,
"Forms RabbitMQ clusters using a variety of backends (AWS EC2, DNS, Consul, Kubernetes, etc)",
"0.8.0"},
{rabbitmq_aws,
"A minimalistic AWS API interface used by rabbitmq-autocluster (3.6.x) and other RabbitMQ plugins",
"3.6.11.milestone1"},
{ssl,"Erlang/OTP SSL application","8.2.2"},
{public_key,"Public key infrastructure","1.5.1"},
{asn1,"The Erlang ASN1 compiler version 5.0.3","5.0.3"},
{sockjs,"SockJS","0.3.4"},
{cowlib,"Support library for manipulating Web protocols.","1.0.2"},
{crypto,"CRYPTO","4.1"},
{amqp_client,"RabbitMQ AMQP Client","3.6.12"},
{rabbit_common,
"Modules shared by rabbitmq-server and rabbitmq-erlang-client",
"3.6.12"},
{os_mon,"CPO CXC 138 46","2.4.3"},
{xmerl,"XML parser","1.3.15"},
{compiler,"ERTS CXC 138 10","7.1.3"},
{syntax_tools,"Syntax tools","2.1.3"},
{inets,"INETS CXC 138 49","6.4.4"},
{sasl,"SASL CXC 138 11","3.1"},
{stdlib,"ERTS CXC 138 10","3.4.2"},
{kernel,"ERTS CXC 138 10","5.4"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang/OTP 20 [erts-9.1.5] [source] [64-bit] [smp:36:36] [ds:36:36:10] [async-threads:576] [hipe] [kernel-poll:true]\n"},
{memory,
[{connection_readers,0},
{connection_writers,0},
{connection_channels,0},
{connection_other,176625752},
{queue_procs,291044992},
{queue_slave_procs,117476104},
{plugins,515800216},
{other_proc,278020328},
{metrics,29506536},
{mgmt_db,157739296},
{mnesia,386363968},
{other_ets,23808896},
{binary,117302056},
{msg_index,12141640},
{code,25416876},
{atom,1041593},
{other_system,47376795},
{total,1024}]},
{alarms,[]},
{listeners,
[{clustering,25672,"::"},
{'http/web-stomp',15674,"::"},
{'https/web-stomp',9451,"::"},
{https,9450,"::"}]},
{vm_memory_calculation_strategy,rss},
{vm_memory_high_watermark,0.75},
{vm_memory_limit,47157202944},
{disk_free_limit,50000000},
{disk_free,12296237056},
{file_descriptors,
[{total_limit,65436},
{total_used,2009},
{sockets_limit,58890},
{sockets_used,2007}]},
{processes,[{limit,1048576},{used,89198}]},
{run_queue,0},
{uptime,137736},
{kernel,{net_ticktime,60}}]
Cluster status of node 'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh'
[{nodes,[{disc,['rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld']}]},
{running_nodes,['rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh']},
{cluster_name,<<"rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh">>},
{partitions,[]},
{alarms,[{'rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld',[]},
{'rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s',[]},
{'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn',[]},
{'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh',[]}]}]
Application environment of node 'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh'
[{amqp_client,[{prefer_ipv6,false},{ssl_options,[]}]},
{asn1,[]},
{autocluster,
[{autocluster_failure,ignore},
{autocluster_log_level,info},
{backend,consul},
{cleanup_warn_only,false},
{cluster_cleanup,true},
{consul_port,8500},
{consul_svc_addr_auto,true},
{consul_svc_port,9450},
{startup_steps_state,
{startup_state,consul,autocluster_consul,
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s',
"d2d38f1f-e689-679d-1e37-98582923fb1e"}}]},
{compiler,[]},
{cowboy,[]},
{cowlib,[]},
{crypto,[{fips_mode,false}]},
{inets,[]},
{kernel,
[{error_logger,tty},
{inet_default_connect_options,[{nodelay,true}]},
{inet_dist_listen_max,25672},
{inet_dist_listen_min,25672}]},
{mnesia,
[{dir,
"/srv/rabbitmq_server-3.6.12/var/lib/rabbitmq/mnesia/rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh"},
{dump_log_write_threshold,256}]},
{os_mon,
[{start_cpu_sup,false},
{start_disksup,false},
{start_memsup,false},
{start_os_sup,false}]},
{public_key,[]},
{rabbit,
[{auth_backends,[rabbit_auth_backend_internal]},
{auth_mechanisms,['PLAIN','AMQPLAIN']},
{background_gc_enabled,false},
{background_gc_target_interval,60000},
{backing_queue_module,rabbit_priority_queue},
{channel_max,0},
{channel_operation_timeout,15000},
{cluster_keepalive_interval,10000},
{cluster_nodes,{[],disc}},
{cluster_partition_handling,autoheal},
{collect_statistics,coarse},
{collect_statistics_interval,60000},
{config_entry_decoder,
[{cipher,aes_cbc256},
{hash,sha512},
{iterations,1000},
{passphrase,undefined}]},
{credit_flow_default_credit,{400,200}},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
{default_user,<<"guest">>},
{default_user_tags,[administrator]},
{default_vhost,<<"/">>},
{delegate_count,16},
{disk_free_limit,50000000},
{disk_monitor_failure_retries,10},
{disk_monitor_failure_retry_interval,120000},
{enabled_plugins_file,
"/srv/rabbitmq_server-3.6.12/etc/rabbitmq/enabled_plugins"},
{error_logger,tty},
{fhc_read_buffering,false},
{fhc_write_buffering,true},
{frame_max,131072},
{halt_on_upgrade_failure,true},
{handshake_timeout,10000},
{heartbeat,60},
{hipe_compile,false},
{hipe_modules,
[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
rabbit_command_assembler,rabbit_framing_amqp_0_9_1,rabbit_basic,
rabbit_event,lists,queue,priority_queue,rabbit_router,rabbit_trace,
rabbit_misc,rabbit_binary_parser,rabbit_exchange_type_direct,
rabbit_guid,rabbit_net,rabbit_amqqueue_process,
rabbit_variable_queue,rabbit_binary_generator,rabbit_writer,
delegate,gb_sets,lqueue,sets,orddict,rabbit_amqqueue,
rabbit_limiter,gb_trees,rabbit_queue_index,
rabbit_exchange_decorator,gen,dict,ordsets,file_handle_cache,
rabbit_msg_store,array,rabbit_msg_store_ets_index,rabbit_msg_file,
rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,
mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,
ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,ssl]},
{lazy_queue_explicit_gc_run_operation_threshold,1000},
{log_levels,[{connection,info},{autocluster,info}]},
{loopback_users,[<<"guest">>]},
{memory_monitor_interval,2500},
{mirroring_flow_control,true},
{mirroring_sync_batch_size,4096},
{mnesia_table_loading_retry_limit,10},
{mnesia_table_loading_retry_timeout,30000},
{msg_store_credit_disc_bound,{4000,800}},
{msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index},
{msg_store_io_batch_size,4096},
{num_ssl_acceptors,1},
{num_tcp_acceptors,10},
{password_hashing_module,rabbit_password_hashing_sha256},
{plugins_dir,"/srv/rabbitmq_server-3.6.12/plugins"},
{plugins_expand_dir,
"/srv/rabbitmq_server-3.6.12/var/lib/rabbitmq/mnesia/rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh-plugins-expand"},
{queue_explicit_gc_run_operation_threshold,1000},
{queue_index_embed_msgs_below,4096},
{queue_index_max_journal_entries,32768},
{reverse_dns_lookups,false},
{sasl_error_logger,tty},
{server_properties,[]},
{ssl_allow_poodle_attack,false},
{ssl_apps,[asn1,crypto,public_key,ssl]},
{ssl_cert_login_from,distinguished_name},
{ssl_handshake_timeout,5000},
{ssl_listeners,[]},
{ssl_options,[]},
{tcp_listen_options,
[{backlog,128},
{nodelay,true},
{linger,{true,0}},
{exit_on_close,false}]},
{tcp_listeners,[]},
{trace_vhosts,[]},
{vm_memory_calculation_strategy,rss},
{vm_memory_high_watermark,0.75},
{vm_memory_high_watermark_paging_ratio,0.9}]},
{rabbit_common,[]},
{rabbitmq_aws,[]},
{rabbitmq_management,
[{cors_allow_origins,[]},
{cors_max_age,1800},
{http_log_dir,none},
{listener,
[{port,9450},
{ssl,true},
{ssl_opts,
[{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
{certfile,"/etc/rabbitmq/ssl/server.pem"},
{keyfile,"/etc/rabbitmq/ssl/server.key"},
{versions,['tlsv1.2','tlsv1.1']},
{dhfile,"/etc/rabbitmq/ssl/dh-params.pem"},
{ciphers,
[{ecdhe_ecdsa,aes_128_gcm,null,sha256},
{ecdhe_rsa,aes_128_gcm,null,sha256}]}]}]},
{load_definitions,none},
{management_db_cache_multiplier,5},
{process_stats_gc_timeout,300000},
{rates_mode,none},
{sample_retention_policies,
[{global,[{60,5}]},{basic,[{60,20}]},{detailed,[{5,5}]}]},
{stats_event_max_backlog,250}]},
{rabbitmq_management_agent,
[{rates_mode,basic},
{sample_retention_policies,
[{global,[{605,5},{3660,60},{29400,600},{86400,1800}]},
{basic,[{605,5},{3600,60}]},
{detailed,[{605,5}]}]}]},
{rabbitmq_stomp,
[{default_user,[{login,<<"guest">>},{passcode,<<"guest">>}]},
{default_vhost,<<"/">>},
{hide_server_info,false},
{implicit_connect,false},
{num_ssl_acceptors,1},
{num_tcp_acceptors,10},
{ssl_cert_login,false},
{ssl_listeners,[]},
{tcp_listen_options,[{backlog,128},{nodelay,true}]},
{tcp_listeners,[]},
{trailing_lf,true}]},
{rabbitmq_web_dispatch,[]},
{rabbitmq_web_stomp,
[{cowboy_opts,[]},
{num_ssl_acceptors,1},
{num_tcp_acceptors,10},
{port,15674},
{sockjs_opts,[]},
{ssl_config,
[{port,9451},
{backlog,1024},
{recbuf,32768},
{sndbuf,32768},
{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
{certfile,"/etc/rabbitmq/ssl/server.pem"},
{keyfile,"/etc/rabbitmq/ssl/server.key"},
{versions,['tlsv1.2','tlsv1.1']},
{dhfile,"/etc/rabbitmq/ssl/dh-params.pem"},
{ciphers,
[{ecdhe_ecdsa,aes_128_gcm,null,sha256},
{ecdhe_rsa,aes_128_gcm,null,sha256}]}]},
{tcp_config,[]},
{use_http_auth,true},
{ws_frame,text}]},
{ranch,[]},
{sasl,[{errlog_type,error},{sasl_error_logger,tty}]},
{sockjs,[]},
{ssl,
[{ciphers,
[{ecdhe_ecdsa,aes_128_gcm,null,sha256},
{ecdhe_rsa,aes_128_gcm,null,sha256}]},
{protocol_version,['tlsv1.2','tlsv1.1',tlsv1]},
{versions,['tlsv1.2','tlsv1.1']}]},
{stdlib,[]},
{syntax_tools,[]},
{xmerl,[]}]
Status of node 'rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld'
[{pid,771},
{running_applications,
[{rabbitmq_top,"RabbitMQ Top","3.6.12"},
{rabbitmq_management,"RabbitMQ Management Console","3.6.12"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.12"},
{rabbitmq_web_stomp,"Rabbit WEB-STOMP - WebSockets to Stomp adapter",
"3.6.12"},
{rabbitmq_stomp,"RabbitMQ STOMP plugin","3.6.12"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.12"},
{rabbit,"RabbitMQ","3.6.12"},
{mnesia,"MNESIA CXC 138 12","4.15.1"},
{cowboy,"Small, fast, modular HTTP server.","1.0.4"},
{ranch,"Socket acceptor pool for TCP protocols.","1.3.0"},
{autocluster,
"Forms RabbitMQ clusters using a variety of backends (AWS EC2, DNS, Consul, Kubernetes, etc)",
"0.8.0"},
{rabbitmq_aws,
"A minimalistic AWS API interface used by rabbitmq-autocluster (3.6.x) and other RabbitMQ plugins",
"3.6.11.milestone1"},
{ssl,"Erlang/OTP SSL application","8.2.2"},
{public_key,"Public key infrastructure","1.5.1"},
{asn1,"The Erlang ASN1 compiler version 5.0.3","5.0.3"},
{amqp_client,"RabbitMQ AMQP Client","3.6.12"},
{rabbit_common,
"Modules shared by rabbitmq-server and rabbitmq-erlang-client",
"3.6.12"},
{compiler,"ERTS CXC 138 10","7.1.3"},
{sockjs,"SockJS","0.3.4"},
{cowlib,"Support library for manipulating Web protocols.","1.0.2"},
{crypto,"CRYPTO","4.1"},
{os_mon,"CPO CXC 138 46","2.4.3"},
{xmerl,"XML parser","1.3.15"},
{syntax_tools,"Syntax tools","2.1.3"},
{inets,"INETS CXC 138 49","6.4.4"},
{sasl,"SASL CXC 138 11","3.1"},
{stdlib,"ERTS CXC 138 10","3.4.2"},
{kernel,"ERTS CXC 138 10","5.4"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang/OTP 20 [erts-9.1.5] [source] [64-bit] [smp:36:36] [ds:36:36:10] [async-threads:576] [hipe] [kernel-poll:true]\n"},
{memory,
[{connection_readers,0},
{connection_writers,0},
{connection_channels,0},
{connection_other,181516536},
{queue_procs,292753192},
{queue_slave_procs,119055752},
{plugins,652387208},
{other_proc,301441104},
{metrics,29774352},
{mgmt_db,158101272},
{mnesia,386350552},
{other_ets,23862592},
{binary,116490800},
{msg_index,12216488},
{code,25463824},
{atom,1049785},
{other_system,47880799},
{total,1024}]},
{alarms,[]},
{listeners,
[{clustering,25672,"::"},
{'http/web-stomp',15674,"::"},
{'https/web-stomp',9451,"::"},
{https,9450,"::"}]},
{vm_memory_calculation_strategy,rss},
{vm_memory_high_watermark,0.75},
{vm_memory_limit,47157202944},
{disk_free_limit,50000000},
{disk_free,12297879552},
{file_descriptors,
[{total_limit,65436},
{total_used,2065},
{sockets_limit,58890},
{sockets_used,2063}]},
{processes,[{limit,1048576},{used,90504}]},
{run_queue,2},
{uptime,137737},
{kernel,{net_ticktime,60}}]
Cluster status of node 'rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld'
[{nodes,[{disc,['rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld']}]},
{running_nodes,['rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh',
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld']},
{cluster_name,<<"rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh">>},
{partitions,[]},
{alarms,[{'rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s',[]},
{'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1r4wn',[]},
{'rabbit@mxt-mx02-wc-rabbitmq-2885423076-1z7rh',[]},
{'rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld',[]}]}]
Application environment of node 'rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld'
[{amqp_client,[{prefer_ipv6,false},{ssl_options,[]}]},
{asn1,[]},
{autocluster,
[{autocluster_failure,ignore},
{autocluster_log_level,info},
{backend,consul},
{cleanup_warn_only,false},
{cluster_cleanup,true},
{consul_port,8500},
{consul_svc_addr_auto,true},
{consul_svc_port,9450},
{startup_steps_state,
{startup_state,consul,autocluster_consul,
'rabbit@mxt-mx02-wc-rabbitmq-2885423076-16n4s',
"6f782773-ddc9-bfd6-0f17-2e18f94d7522"}}]},
{compiler,[]},
{cowboy,[]},
{cowlib,[]},
{crypto,[{fips_mode,false}]},
{inets,[]},
{kernel,
[{error_logger,tty},
{inet_default_connect_options,[{nodelay,true}]},
{inet_dist_listen_max,25672},
{inet_dist_listen_min,25672}]},
{mnesia,
[{dir,
"/srv/rabbitmq_server-3.6.12/var/lib/rabbitmq/mnesia/rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld"},
{dump_log_write_threshold,256}]},
{os_mon,
[{start_cpu_sup,false},
{start_disksup,false},
{start_memsup,false},
{start_os_sup,false}]},
{public_key,[]},
{rabbit,
[{auth_backends,[rabbit_auth_backend_internal]},
{auth_mechanisms,['PLAIN','AMQPLAIN']},
{background_gc_enabled,false},
{background_gc_target_interval,60000},
{backing_queue_module,rabbit_priority_queue},
{channel_max,0},
{channel_operation_timeout,15000},
{cluster_keepalive_interval,10000},
{cluster_nodes,{[],disc}},
{cluster_partition_handling,autoheal},
{collect_statistics,coarse},
{collect_statistics_interval,60000},
{config_entry_decoder,
[{cipher,aes_cbc256},
{hash,sha512},
{iterations,1000},
{passphrase,undefined}]},
{credit_flow_default_credit,{400,200}},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
{default_user,<<"guest">>},
{default_user_tags,[administrator]},
{default_vhost,<<"/">>},
{delegate_count,16},
{disk_free_limit,50000000},
{disk_monitor_failure_retries,10},
{disk_monitor_failure_retry_interval,120000},
{enabled_plugins_file,
"/srv/rabbitmq_server-3.6.12/etc/rabbitmq/enabled_plugins"},
{error_logger,tty},
{fhc_read_buffering,false},
{fhc_write_buffering,true},
{frame_max,131072},
{halt_on_upgrade_failure,true},
{handshake_timeout,10000},
{heartbeat,60},
{hipe_compile,false},
{hipe_modules,
[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
rabbit_command_assembler,rabbit_framing_amqp_0_9_1,rabbit_basic,
rabbit_event,lists,queue,priority_queue,rabbit_router,rabbit_trace,
rabbit_misc,rabbit_binary_parser,rabbit_exchange_type_direct,
rabbit_guid,rabbit_net,rabbit_amqqueue_process,
rabbit_variable_queue,rabbit_binary_generator,rabbit_writer,
delegate,gb_sets,lqueue,sets,orddict,rabbit_amqqueue,
rabbit_limiter,gb_trees,rabbit_queue_index,
rabbit_exchange_decorator,gen,dict,ordsets,file_handle_cache,
rabbit_msg_store,array,rabbit_msg_store_ets_index,rabbit_msg_file,
rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,
mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,
ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,ssl]},
{lazy_queue_explicit_gc_run_operation_threshold,1000},
{log_levels,[{connection,info},{autocluster,info}]},
{loopback_users,[<<"guest">>]},
{memory_monitor_interval,2500},
{mirroring_flow_control,true},
{mirroring_sync_batch_size,4096},
{mnesia_table_loading_retry_limit,10},
{mnesia_table_loading_retry_timeout,30000},
{msg_store_credit_disc_bound,{4000,800}},
{msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index},
{msg_store_io_batch_size,4096},
{num_ssl_acceptors,1},
{num_tcp_acceptors,10},
{password_hashing_module,rabbit_password_hashing_sha256},
{plugins_dir,"/srv/rabbitmq_server-3.6.12/plugins"},
{plugins_expand_dir,
"/srv/rabbitmq_server-3.6.12/var/lib/rabbitmq/mnesia/rabbit@mxt-mx02-wc-rabbitmq-2885423076-t8cld-plugins-expand"},
{queue_explicit_gc_run_operation_threshold,1000},
{queue_index_embed_msgs_below,4096},
{queue_index_max_journal_entries,32768},
{reverse_dns_lookups,false},
{sasl_error_logger,tty},
{server_properties,[]},
{ssl_allow_poodle_attack,false},
{ssl_apps,[asn1,crypto,public_key,ssl]},
{ssl_cert_login_from,distinguished_name},
{ssl_handshake_timeout,5000},
{ssl_listeners,[]},
{ssl_options,[]},
{tcp_listen_options,
[{backlog,128},
{nodelay,true},
{linger,{true,0}},
{exit_on_close,false}]},
{tcp_listeners,[]},
{trace_vhosts,[]},
{vm_memory_calculation_strategy,rss},
{vm_memory_high_watermark,0.75},
{vm_memory_high_watermark_paging_ratio,0.9}]},
{rabbit_common,[]},
{rabbitmq_aws,[]},
{rabbitmq_management,
[{cors_allow_origins,[]},
{cors_max_age,1800},
{http_log_dir,none},
{listener,
[{port,9450},
{ssl,true},
{ssl_opts,
[{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
{certfile,"/etc/rabbitmq/ssl/server.pem"},
{keyfile,"/etc/rabbitmq/ssl/server.key"},
{versions,['tlsv1.2','tlsv1.1']},
{dhfile,"/etc/rabbitmq/ssl/dh-params.pem"},
{ciphers,
[{ecdhe_ecdsa,aes_128_gcm,null,sha256},
{ecdhe_rsa,aes_128_gcm,null,sha256}]}]}]},
{load_definitions,none},
{management_db_cache_multiplier,5},
{process_stats_gc_timeout,300000},
{rates_mode,none},
{sample_retention_policies,
[{global,[{60,5}]},{basic,[{60,20}]},{detailed,[{5,5}]}]},
{stats_event_max_backlog,250}]},
{rabbitmq_management_agent,
[{rates_mode,basic},
{sample_retention_policies,
[{global,[{605,5},{3660,60},{29400,600},{86400,1800}]},
{basic,[{605,5},{3600,60}]},
{detailed,[{605,5}]}]}]},
{rabbitmq_stomp,
[{default_user,[{login,<<"guest">>},{passcode,<<"guest">>}]},
{default_vhost,<<"/">>},
{hide_server_info,false},
{implicit_connect,false},
{num_ssl_acceptors,1},
{num_tcp_acceptors,10},
{ssl_cert_login,false},
{ssl_listeners,[]},
{tcp_listen_options,[{backlog,128},{nodelay,true}]},
{tcp_listeners,[]},
{trailing_lf,true}]},
{rabbitmq_top,[]},
{rabbitmq_web_dispatch,[]},
{rabbitmq_web_stomp,
[{cowboy_opts,[]},
{num_ssl_acceptors,1},
{num_tcp_acceptors,10},
{port,15674},
{sockjs_opts,[]},
{ssl_config,
[{port,9451},
{backlog,1024},
{recbuf,32768},
{sndbuf,32768},
{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
{certfile,"/etc/rabbitmq/ssl/server.pem"},
{keyfile,"/etc/rabbitmq/ssl/server.key"},
{versions,['tlsv1.2','tlsv1.1']},
{dhfile,"/etc/rabbitmq/ssl/dh-params.pem"},
{ciphers,
[{ecdhe_ecdsa,aes_128_gcm,null,sha256},
{ecdhe_rsa,aes_128_gcm,null,sha256}]}]},
{tcp_config,[]},
{use_http_auth,true},
{ws_frame,text}]},
{ranch,[]},
{sasl,[{errlog_type,error},{sasl_error_logger,tty}]},
{sockjs,[]},
{ssl,
[{ciphers,
[{ecdhe_ecdsa,aes_128_gcm,null,sha256},
{ecdhe_rsa,aes_128_gcm,null,sha256}]},
{protocol_version,['tlsv1.2','tlsv1.1',tlsv1]},
{versions,['tlsv1.2','tlsv1.1']}]},
{stdlib,[]},
{syntax_tools,[]},
{xmerl,[]}]
Michael Klishin
Mar 15, 2018, 11:28:09 AM3/15/18
to rabbitm...@googlegroups.com
They collect and aggregate stats, one collector and ETS table per event type.
mgmt_db_cache maintains a cache of results retrieved from other nodes when serving
HTTP API requests. Fanning out to the entire cluster is expensive, aggregating, sorting and filtering all those results is not cheap either since sometimes moderately large
or fairly large data sets have to be transferred between nodes so that filtering + sorting can be applied.
Stats are emitted periodically and reductions is a cumulative metric, so on an idle
node those processes will not only be in the top 10 but also increasing the gap
in reduction count compared to other processes.
Just like before 3.6.7, the more connections/channels/queues there are, the more stats
are emitted and need processing. In fact, for the single node case things are not radically different.
HTH.
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Laurent Luce
Mar 15, 2018, 11:55:07 AM3/15/18
to rabbitmq-users
Hi Michael,
By collecting and aggregating stats, do you mean internal stats the system relies on to function properly or stats that are used only for monitoring? Is there a way to reduce some of those via config to lower CPU usage?
On our setup, we use 4 c4.4xlarge nodes (16 cores, 30 GB each) and with 10K idle connections distributed on those 4 nodes (web-stomp, TLS), we see a load of 6 and CPU at 20%.
Thanks,
Laurent
...
Michael Klishin
Mar 15, 2018, 12:02:13 PM3/15/18
to rabbitm...@googlegroups.com
The stats that make the HTTP API work.
You can increase stats emission interval, use a different rates_modeI'd definitely expect a lot of context switching that's not necessary. We cannot ship a default
that works for 2-4 cores as well as 32 or 64 or more.
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ
Laurent Luce
Mar 15, 2018, 1:01:29 PM3/15/18
to rabbitmq-users
I have collect statistics interval set to 60 seconds and rates mode = none so not sure why those metrics collectors are working so hard.
I will try a VM with less cores, same amount of memory.
Laurent
...
Michael Klishin
Mar 15, 2018, 2:16:18 PM3/15/18
to rabbitm...@googlegroups.com
You do realize that they are working "so hard" relative to everything else on an idle node, right? And that reductions
are not kilograms or meters, they are relative and fairly subjective units that make sense for the runtime to make
certain decisions, not an absolute unit that can be assumed to be constant, completely predictable between experiments and so on.
Try a PerfTest run with -x 4 -y 8 (well over a dozen of busy processes) and see if the collectors are still in the top 5:
http://www.rabbitmq.com/java-tools.html.are not kilograms or meters, they are relative and fairly subjective units that make sense for the runtime to make
certain decisions, not an absolute unit that can be assumed to be constant, completely predictable between experiments and so on.
Try a PerfTest run with -x 4 -y 8 (well over a dozen of busy processes) and see if the collectors are still in the top 5:
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Laurent Luce
Mar 15, 2018, 8:14:54 PM3/15/18
to rabbitmq-users
Understood.
Is the context switching metric something i can also access using the REST API?
I did more testing using 4 nodes - 8 cores / 60 GB each (r4.2xlarge):
10K connections (20K queues) - CPU 25% - 7 GB
20K connections (40K queues) - CPU 39% - 10 GB
30K connections (60K queues) - CPU 47% - 12 GB
40K connections (80K queues) - CPU 55% - 15 GB
Connections are web-stomp with TLS. Idle.
I read the Erlang core-to-scheduler binding strategy options. I was curious if there is a thread discussing those in relation to EC2 instance types.
Laurent
...
Michael Klishin
Mar 15, 2018, 11:32:16 PM3/15/18
to rabbitm...@googlegroups.com
Context switching chart is available on the node page.
See `GET /api/nodes/{node}`.--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Laurent Luce
Mar 21, 2018, 5:02:44 PM3/21/18
to rabbitmq-users
Using the core-to-scheduler binding strategy 'ts' instead of the default one, context switching issue went away.
Laurent
...
Michael Klishin
Mar 21, 2018, 5:04:11 PM3/21/18
to rabbitm...@googlegroups.com
Thank you for reporting back!
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
Michael Klishin
Apr 27, 2018, 7:13:35 AM4/27/18
to rabbitmq-users
This question in various forms comes up often enough over 1 year after the new management plugin shipped,
so since we keep linking to this thread I'd like to commend on this specific post.
For those upgrading from a pre-3.6.7 version to 3.6.15 or later: there are no reasons to set
stats collection interval to 60 seconds and rates_mode to `none`. Stats DB overload is not an issue any more,
based on more than 1 year of feedback on this list and elsewhere.
Reply all
Reply to author
Forward
0 new messages