CLOSE_WAIT connections are increasing in rabbitmq in rhel7 server

[balaji k]

Hi
I have installed rabbitmq-server-3.6.9 package in rhel7 server
my rabbitmq config file is as below mentioned
[
{ssl, [{versions, ['tlsv1.2', 'tlsv1.1']}]},
{rabbit, [
{cluster_partition_handling, pause_minority},
{ssl_listeners, [5671]},
{tcp_listeners, []},
{log_levels,[{connection, debug}]},
{ssl_allow_poodle_attack, false},
{ssl_apps,[asn1,crypto,public_key,ssl]},
{auth_mechanisms, [ 'EXTERNAL', 'PLAIN', 'AMQPLAIN' ]},
{ssl_cert_login_from, common_name},
{ssl_options, [{cacertfile,"/etc/sensu/ssl/cacerts/dev_ca.pem"},
{certfile,"/etc/sensu/ssl/host.cert"},
{keyfile,"/etc/sensu/ssl/host.key"},
{versions, ['tlsv1.2', 'tlsv1.1']},
{dhfile, "/etc/sensu/ssl/dh-params.pem"},
{ciphers, ["ECDHE-ECDSA-AES256-GCM-SHA384","ECDHE-RSA-AES256-GCM-SHA384",
"ECDHE-ECDSA-AES256-SHA384","ECDHE-RSA-AES256-SHA384", "ECDHE-ECDSA-DES-CBC3-SHA",
"ECDH-ECDSA-AES256-GCM-SHA384","ECDH-RSA-AES256-GCM-SHA384","ECDH-ECDSA-AES256-SHA384",
"ECDH-RSA-AES256-SHA384","DHE-DSS-AES256-GCM-SHA384","DHE-DSS-AES256-SHA256",
"AES256-GCM-SHA384","AES256-SHA256","ECDHE-ECDSA-AES128-GCM-SHA256",
"ECDHE-RSA-AES128-GCM-SHA256","ECDHE-ECDSA-AES128-SHA256","ECDHE-RSA-AES128-SHA256",
"ECDH-ECDSA-AES128-GCM-SHA256","ECDH-RSA-AES128-GCM-SHA256","ECDH-ECDSA-AES128-SHA256",
"ECDH-RSA-AES128-SHA256","DHE-DSS-AES128-GCM-SHA256","DHE-DSS-AES128-SHA256",
"AES128-GCM-SHA256","AES128-SHA256"]},
{depth, 2},
{verify,verify_peer},
{fail_if_no_peer_cert,true},
{secure_renegotiate, true},
{reuse_sessions, true},
{honor_cipher_order, true},
{max_connections, infinity}]}
]}
].

rabbitmq status is as below mentioned
Status of node 'rabbit@host' ...
[{pid,51019},
{running_applications,
[{rabbit,"RabbitMQ","3.6.9"},
{rabbit_common,
"Modules shared by rabbitmq-server and rabbitmq-erlang-client",
"3.6.9"},
{compiler,"ERTS CXC 138 10","7.0.4"},
{os_mon,"CPO CXC 138 46","2.4.2"},
{ranch,"Socket acceptor pool for TCP protocols.","1.3.0"},
{ssl,"Erlang/OTP SSL application","8.1.1"},
{public_key,"Public key infrastructure","1.4"},
{crypto,"CRYPTO","3.7.3"},
{asn1,"The Erlang ASN1 compiler version 4.0.4","4.0.4"},
{xmerl,"XML parser","1.3.13"},
{syntax_tools,"Syntax tools","2.1.1"},
{mnesia,"MNESIA CXC 138 12","4.14.3"},
{sasl,"SASL CXC 138 11","3.0.3"},
{stdlib,"ERTS CXC 138 10","3.3"},
{kernel,"ERTS CXC 138 10","5.2"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang/OTP 19 [erts-8.3] [source] [64-bit] [smp:4:4] [async-threads:64] [hipe] [kernel-poll:true]\n"},
{memory,
[{total,55273816},
{connection_readers,0},
{connection_writers,0},
{connection_channels,0},
{connection_other,0},
{queue_procs,2832},
{queue_slave_procs,0},
{plugins,0},
{other_proc,21434000},
{mnesia,73448},
{metrics,184192},
{mgmt_db,0},
{msg_index,42480},
{other_ets,2177168},
{binary,53528},
{code,21356615},
{atom,891849},
{other_system,9239064}]},
{alarms,[]},
{listeners,[{clustering,25672,"::"},{'amqp/ssl',5671,"::"}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,6660399104},
{disk_free_limit,50000000},
{disk_free,99994918912},
{file_descriptors,
[{total_limit,924},{total_used,2},{sockets_limit,829},{sockets_used,0}]},
{processes,[{limit,1048576},{used,147}]},
{run_queue,0},
{uptime,269},
{kernel,{net_ticktime,60}}]

I am able to start and stop rabbitmq. but my client is not connecting to it
when I see there are so many close wait connections as below

netstat -anlp | grep 5671
tcp6 4 0 :::5671 :::* LISTEN 51019/beam.smp
tcp6 518 0 serverIP:5671 clientIP:19260 CLOSE_WAIT -
tcp6 518 0 serverIP:5671 clientIP:19236 CLOSE_WAIT 51019/beam.smp

network settings related to rabbitmq are as below
fs.file-max = 1614110
net.core.somaxconn = 128
net.ipv4.conf.default.rp_filter = 1
net.ipv4.ip_local_port_range = 32768 60999
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_max_syn_backlog = 512
net.ipv4.tcp_tw_reuse = 0

server log as below
=INFO REPORT==== 28-Apr-2017::08:01:42 ===
Starting RabbitMQ 3.6.9 on Erlang 19.3
Copyright (C) 2007-2016 Pivotal Software, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/

=INFO REPORT==== 28-Apr-2017::08:01:42 ===
node : rabbit@sd-0aad-6587
home dir : /home/aimops
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash : yl7M4HTRmS+l2YF5Tc3RYg==
log : /var/log/rabbitmq/rab...@host.log
sasl log : /var/log/rabbitmq/rab...@host.log
database dir : /var/lib/rabbitmq/mnesia/rabbit@host

=INFO REPORT==== 28-Apr-2017::08:01:43 ===
Memory limit set to 6351MB of 15879MB total.

=INFO REPORT==== 28-Apr-2017::08:01:43 ===
Disk free limit set to 50MB

=INFO REPORT==== 28-Apr-2017::08:01:43 ===
Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 28-Apr-2017::08:01:43 ===
FHC read buffering: OFF
FHC write buffering: ON

=INFO REPORT==== 28-Apr-2017::08:01:43 ===
Waiting for Mnesia tables for 30000 ms, 9 retries left

=INFO REPORT==== 28-Apr-2017::08:01:43 ===
Waiting for Mnesia tables for 30000 ms, 9 retries left

=INFO REPORT==== 28-Apr-2017::08:01:43 ===
Priority queues enabled, real BQ is rabbit_variable_queue

=INFO REPORT==== 28-Apr-2017::08:01:43 ===
Starting rabbit_node_monitor

=INFO REPORT==== 28-Apr-2017::08:01:43 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 28-Apr-2017::08:01:43 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 28-Apr-2017::08:01:43 ===
started SSL Listener on [::]:5671

=INFO REPORT==== 28-Apr-2017::08:01:43 ===
Server startup complete; 0 plugins started.

this close wait connections is keep on increasing....what to do for this
this is happening only in rhel7 ..in rhel6 , its working fine

[Michael Klishin]

According to the log and status RabbitMQ doesn't have any connections. In case that's the entire log, it 

did not have any since last boot.

If your clients fail to connect, consider enabling net.ipv4.tcp_tw_reus.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Staff Software Engineer, Pivotal/RabbitMQ

[balaji k]

after enabling also, its same issue

same logs

same close wait connections

log : /var/log/rabbitmq/rabbit@host.log
sasl log : /var/log/rabbitmq/rabbit@host.log
database dir : /var/lib/rabbitmq/mnesia/rabbit@host

[Michael Klishin]

OK, so with a refresher on various states I recommend the following.

You can take a look at https://groups.google.com/forum/#!topic/rabbitmq-users/MsTnhP9O5YM and use the same

TCP listener option tweak used there (the exit_on_close one, the rest is not relevant for this thread).

Another thing I'd highly recommend checking is whether your apps have heartbeats disabled (it's not the case

with recent versions of most clients and RabbitMQ server defaults but there's one exception: the PHP client).

Heartbeats (or keepalives in MQTT) are the primary mechanism for messaging protocols for unavailable peer detection.

See http://www.rabbitmq.com/heartbeats.html. As explained in that doc guide, the kernel TCP settings still

can help but the defaults are absolutely inadequate for modern servers :(

So please make sure your clients can and so use heartbeats (values between 6 and 20 seconds is something we

recommend when connection churn is high; do not go lower as it will cause false positives).

I can't think of any differences between RHEL 6 and 7 other than: they can simply be two difference machines

with subtle configuration differences that are hard to pin point.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ