RabbitMQ sends RST packets

Fei Jiang

unread,

Apr 24, 2018, 4:29:34 AM4/24/18

to rabbitmq-users

hi， there.

Rabbitmq cluster : rabbitmqNode0 and rabbitmqNode1

Memory : 50GB

CPU: 40 Vcpu

RabbitMQ version : 3.5.7

Now I have a problem that server RabbitMQ sometimes sends to me the RST packets.

Scenario is as below:

I create about 1000 producer ，and 100 consumer

1000 producer send message to consumer， and wait consumer to response

I restart producer process sometimes

The RabbitMQ have some error about， rabbitmqNode0-sasl.log have no info about it，

rabbitmq log rab...@rabbitmqNode0.log ：

=WARNING REPORT==== 24-Apr-2018::12:24:28 ===

closing AMQP connection <0.25682.614> (172.28.1.122:60102 -> 172.28.8.220:5672):

connection_closed_abruptly

=INFO REPORT==== 24-Apr-2018::12:24:50 ===

accepting AMQP connection <0.19854.77> (172.28.1.122:33274 -> 172.28.8.220:5672)

I am sure oslo.messaging client have no error about to close the socke，and the tcpdump prove that the MQ-server send RST packets to client，

2018-04-24T12:24:46.656+08:00 localhost testtest DEBUG [pid:29857] [MainThread][tid:91998160] [impl_rabbit.py:766 on_error] Received recoverable error from kombu:

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit File "/home/jf/venv/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs)

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit File "/home/jf/venv/lib/python2.7/site-packages/kombu/connection.py", line 508, in __call__

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0]

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit File "/home/jf/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 815, in execute_method

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit method()

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit File "/home/jf/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1124, in _consume

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit self.connection.drain_events(timeout=poll_timeout)

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit File "/home/jf/venv/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit File "/home/jf/venv/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 95, in drain_events

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit File "/home/jf/venv/lib/python2.7/site-packages/amqp/connection.py", line 303, in drain_events

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit chanmap, None, timeout=timeout,

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit File "/home/jf/venv/lib/python2.7/site-packages/amqp/connection.py", line 366, in _wait_multiple

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit channel, method_sig, args, content = read_timeout(timeout)

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit File "/home/jf/venv/lib/python2.7/site-packages/amqp/connection.py", line 337, in read_timeout

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit return self.method_reader.read_method()

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit File "/home/jf/venv/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit raise m

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit error: [Errno 110] Connection timed out

2018-04-24 12:24:46.651 8393 ERROR oslo.messaging._drivers.impl_rabbit

Do you have any idea what is the problem? And in that cases server can send RST packets?

thank you.

Michael Klishin

unread,

Apr 24, 2018, 6:32:24 AM4/24/18

to rabbitm...@googlegroups.com

See server logs, https://www.rabbitmq.com/troubleshooting-networking.html and https://www.rabbitmq.com/networking.html

(specifically the part on open file handles and ports).

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

Message has been deleted

Fei Jiang

unread,

Apr 24, 2018, 9:11:44 PM4/24/18

to rabbitmq-users

my rabbitmq config :

[

{kernel,

[

{inet_dist_use_interface, {172,28,8,211}},

{inet_dist_listen_min, 50000},

{inet_dist_listen_max, 50001},

{net_ticktime, 60}

]

},

{rabbit,

[

{default_pass,<<"tmp">>},

{disk_free_limit,1024000000},

{vm_memory_high_watermark,0.4},

{vm_memory_high_watermark_paging_ratio, 0.9},

{heartbeat,180},

{default_user,<<"rabbitTest">>},

{cluster_partition_handling, autoheal},

{cluster_nodes, {[rabbit@rabbitmqNode1,rabbit@rabbitmqNode0], disc}},

{tcp_listeners, []},

{ssl_listeners, [{"0.0.0.0", 5672}]},

{tcp_listen_options, [binary,

{packet, raw},

{reuseaddr, true},

{backlog, 4096},

{nodelay, true},

{exit_on_close, false},

{keepalive, true}]},

{ssl_options, [

{certfile,"/etc/rabbitmq/rabbitmq-server.crt"},

{keyfile,"/etc/rabbitmq/rabbitmq-server.key"},

{versions, ['tlsv1.1', 'tlsv1.2']},

{ciphers, [

{dhe_rsa,aes_256_cbc,sha256},

{dhe_dss,aes_256_cbc,sha256},

{dhe_rsa,aes_128_cbc,sha256},

{dhe_dss,aes_128_cbc,sha256},

{dhe_dss,aes_256_cbc,sha},

{dhe_dss,aes_128_cbc,sha},

{ecdhe_ecdsa,aes_256_gcm,null,sha384},

{ecdhe_rsa,aes_256_gcm,null,sha384},

{ecdh_ecdsa,aes_256_gcm,null,sha384},

{ecdh_rsa,aes_256_gcm,null,sha384},

{dhe_rsa,aes_256_gcm,null,sha384},

{dhe_dss,aes_256_gcm,null,sha384},

{ecdhe_ecdsa,aes_128_gcm,null,sha256},

{ecdhe_rsa,aes_128_gcm,null,sha256},

{ecdh_ecdsa,aes_128_gcm,null,sha256},

{ecdh_rsa,aes_128_gcm,null,sha256},

{dhe_rsa,aes_128_gcm,null,sha256},

{dhe_dss,aes_128_gcm,null,sha256}

]}

]

},

{rabbitmq_management,

[

{listener,

[

{port, 15672},

{ip, "127.0.0.1"}

]

},

{redirect_old_port, false},

{rates_mode, none}

]

},

].

Fei Jiang

unread,

Apr 24, 2018, 10:08:22 PM4/24/18

to rabbitmq-users

details rabbitmqctl environment

Application environment of node rabbit@rabbitmqNode0 ...

[{amqp_client,[{prefer_ipv6,false},{ssl_options,[]}]},

{asn1,[]},

{crypto,[]},

{inets,[]},

{kernel,

[{error_logger,tty},

{inet_default_connect_options,

[{nodelay,true},{raw,6,18,<<152,58,0,0,0,0,0,0>>}]},

{inet_default_listen_options,[{raw,6,18,<<152,58,0,0,0,0,0,0>>}]},

{inet_dist_listen_max,50001},

{inet_dist_listen_min,50000},

{inet_dist_use_interface,{172,28,8,211}},

{net_ticktime,60}]},

{mnesia,[{dir,"/opt/data/rabbitmq/rabbit@rabbitmqNode0"}]},

{mochiweb,[]},

{os_mon,

[{start_cpu_sup,false},

{start_disksup,false},

{start_memsup,false},

{start_os_sup,false}]},

{public_key,[]},

{rabbit,

[{auth_backends,[rabbit_auth_backend_internal]},

{auth_mechanisms,['PLAIN','AMQPLAIN']},

{backing_queue_module,rabbit_priority_queue},

{channel_max,0},

{cluster_keepalive_interval,10000},

{cluster_nodes,{[rabbit@rabbitmqNode1,rabbit@rabbitmqNode0],disc}},

{cluster_partition_handling,autoheal},

{collect_statistics,coarse},

{collect_statistics_interval,60000},

{credit_flow_default_credit,{200,50}},

{default_permissions,[<<".*">>,<<".*">>,<<".*">>]},

{default_user,<<"rabbit">>},

{default_user_tags,[administrator]},

{default_vhost,<<"/">>},

{delegate_count,16},

{disk_free_limit,1024000000},

{enabled_plugins_file,

"/usr/local/lib/rabbitmq/sbin/../etc/rabbitmq/enabled_plugins"},

{error_logger,

{file,

"/var/log/rabbitmq/rab...@rabbitmqNode0.log"}},

{fhc_read_buffering,true},

{fhc_write_buffering,true},

{frame_max,131072},

{halt_on_upgrade_failure,true},

{handshake_timeout,10000},

{heartbeat,180},

{hipe_compile,false},

{hipe_modules,

[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,

rabbit_command_assembler,rabbit_framing_amqp_0_9_1,rabbit_basic,

rabbit_event,lists,queue,priority_queue,rabbit_router,rabbit_trace,

rabbit_misc,rabbit_binary_parser,rabbit_exchange_type_direct,

rabbit_guid,rabbit_net,rabbit_amqqueue_process,

rabbit_variable_queue,rabbit_binary_generator,rabbit_writer,

delegate,gb_sets,lqueue,sets,orddict,rabbit_amqqueue,

rabbit_limiter,gb_trees,rabbit_queue_index,

rabbit_exchange_decorator,gen,dict,ordsets,file_handle_cache,

rabbit_msg_store,array,rabbit_msg_store_ets_index,rabbit_msg_file,

rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,

mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,

ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,ssl]},

{log_levels,[{connection,info}]},

{loopback_users,[<<"guest">>]},

{mirroring_flow_control,true},

{mnesia_table_loading_timeout,30000},

{msg_store_credit_disc_bound,{2000,500}},

{msg_store_file_size_limit,16777216},

{msg_store_index_module,rabbit_msg_store_ets_index},

{msg_store_io_batch_size,2048},

{plugins_dir,"/usr/local/lib/rabbitmq/sbin/../plugins"},

{plugins_expand_dir,

"/opt/data/rabbitmq/rabbit@rabbitmqNode0-plugins-expand"},

{queue_index_embed_msgs_below,4096},

{queue_index_max_journal_entries,32768},

{reverse_dns_lookups,false},

{sasl_error_logger,

{file,

"/var/log/rabbitmq/rab...@rabbitmqNode0-sasl.log"}},

{server_properties,[]},

{ssl_allow_poodle_attack,false},

{ssl_apps,[asn1,crypto,public_key,ssl]},

{ssl_cert_login_from,distinguished_name},

{ssl_handshake_timeout,5000},

{ssl_listeners,[{"0.0.0.0",5672}]},

{ssl_options,

[{certfile,"/etc/rabbitmq/rabbitmq-server.crt"},

{keyfile,"/etc/rabbitmq/rabbitmq-server.key"},

{versions,['tlsv1.1','tlsv1.2']},

{ciphers,

[{dhe_rsa,aes_256_cbc,sha256},

{dhe_dss,aes_256_cbc,sha256},

{dhe_rsa,aes_128_cbc,sha256},

{dhe_dss,aes_128_cbc,sha256},

{dhe_dss,aes_256_cbc,sha},

{dhe_dss,aes_128_cbc,sha},

{ecdhe_ecdsa,aes_256_gcm,null,sha384},

{ecdhe_rsa,aes_256_gcm,null,sha384},

{ecdh_ecdsa,aes_256_gcm,null,sha384},

{ecdh_rsa,aes_256_gcm,null,sha384},

{dhe_rsa,aes_256_gcm,null,sha384},

{dhe_dss,aes_256_gcm,null,sha384},

{ecdhe_ecdsa,aes_128_gcm,null,sha256},

{ecdhe_rsa,aes_128_gcm,null,sha256},

{ecdh_ecdsa,aes_128_gcm,null,sha256},

{ecdh_rsa,aes_128_gcm,null,sha256},

{dhe_rsa,aes_128_gcm,null,sha256},

{dhe_dss,aes_128_gcm,null,sha256}]}]},

{stats_event_max_backlog,100},

{tcp_listen_options,

[binary,

{packet,raw},

{reuseaddr,true},

{backlog,4096},

{nodelay,true},

{exit_on_close,false},

{keepalive,true}]},

{tcp_listeners,[]},

{trace_vhosts,[]},

{vm_memory_high_watermark,0.454354862684},

{vm_memory_high_watermark_paging_ratio,0.9}]},

{rabbitmq_management,

[{http_log_dir,none},

{listener,[{port,15672},{ip,"127.0.0.1"}]},

{load_definitions,none},

{rates_mode,none},

{redirect_old_port,false},

{sample_retention_policies,

[{global,[{605,5},{3660,60},{29400,600},{86400,1800}]},

{basic,[{605,5},{3600,60}]},

{detailed,[{10,5}]}]}]},

{rabbitmq_management_agent,[]},

{rabbitmq_tracing,[{directory,"/var/log/rabbitmq"}]},

{rabbitmq_web_dispatch,[]},

{sasl,[{errlog_type,error},{sasl_error_logger,false}]},

{ssl,[]},

{stdlib,[]},

{webmachine,[{error_handler,rabbit_webmachine_error_handler}]},

{xmerl,[]}]

Michael Klishin

unread,

Apr 24, 2018, 11:33:38 PM4/24/18

to rabbitm...@googlegroups.com

Thanks but server logs and following the steps outlined in the guide we've linked to above is more important.

`rabbitmqctl status` will show how many file descriptors the node has available to it, a *very* common

reason why inbound connections may not be accepted.

"/var/log/rabbitmq/rabbit@rabbitmqNode0.log"}},

"/var/log/rabbitmq/rabbit@rabbitmqNode0-sasl.log"}},

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Fei Jiang

unread,

Apr 25, 2018, 9:26:25 PM4/25/18

to rabbitmq-users

Thank you ,Michael

There is mor info about it, FD is set 65535.

rabbitmqctl status

Status of node rabbit@rabbitmqNode0 ...

[{pid,47698},

{running_applications,

[{rabbitmq_tracing,"RabbitMQ message logging / tracing","3.5.6"},

{rabbitmq_management,"RabbitMQ Management Console","3.5.6"},

{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.5.6"},

{webmachine,"webmachine","1.10.3-rmq3.5.6-gite9359c7"},

{rabbitmq_management_agent,"RabbitMQ Management Agent","3.5.6"},

{rabbit,"RabbitMQ","3.5.6"},

{ssl,"Erlang/OTP SSL application","7.3.2"},

{public_key,"Public key infrastructure","1.1.1"},

{crypto,"CRYPTO","3.6.3"},

{asn1,"The Erlang ASN1 compiler version 4.0.2","4.0.2"},

{mnesia,"MNESIA CXC 138 12","4.13.4"},

{os_mon,"CPO CXC 138 46","2.4"},

{amqp_client,"RabbitMQ AMQP Client","3.5.6"},

{mochiweb,"MochiMedia Web Server","2.7.0-rmq3.5.6-git680dba8"},

{xmerl,"XML parser","1.3.10"},

{inets,"INETS CXC 138 49","6.2.3"},

{sasl,"SASL CXC 138 11","2.7"},

{stdlib,"ERTS CXC 138 10","2.8"},

{kernel,"ERTS CXC 138 10","4.2"}]},

{os,{unix,linux}},

{erlang_version,

"Erlang/OTP 18 [erts-7.3.1] [source] [64-bit] [smp:56:56] [async-threads:128] [hipe] [kernel-poll:true]\n"},

{memory,

[{total,7773910336},

{connection_readers,148229864},

{connection_writers,38187112},

{connection_channels,247834424},

{connection_other,419288744},

{queue_procs,550798400},

{queue_slave_procs,8896},

{plugins,84164512},

{other_proc,299389440},

{mnesia,64980160},

{mgmt_db,306382520},

{msg_index,27615480},

{other_ets,32891528},

{binary,5390836784},

{code,22754658},

{atom,4600369},

{other_system,135947445}]},

{alarms,[]},

{listeners,[{clustering,50000,"::"},{'amqp/ssl',5672,"0.0.0.0"}]},

{vm_memory_high_watermark,0.454354862684},

{vm_memory_limit,214748398298},

{disk_free_limit,1024000000},

{disk_free,19876384768},

{file_descriptors,

[{total_limit,65436},

{total_used,13910},

{sockets_limit,58890},

{sockets_used,13908}]},

{processes,[{limit,1048576},{used,235076}]},

{run_queue,0},

{uptime,1287565}]

add the tcpdump information of the previous time, Shortly thereafter, RST message was received from mq-server

Michael Klishin

unread,

Apr 25, 2018, 9:35:07 PM4/25/18

to rabbitm...@googlegroups.com

RabbitMQ logs when it accepts or closes a connection, successfully or not. Let's try one more time:

please take a look at and post server logs.

What steps from https://www.rabbitmq.com/troubleshooting-networking.html were followed and what was the outcome?

Please help others help you by collecting and providing data.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Michael Klishin

unread,

Apr 25, 2018, 9:40:26 PM4/25/18

to rabbitm...@googlegroups.com

We don't have enough information to confirm my next hypothesis but I can only respond to guesses with guesses.

You are running on Erlang 18.x, which has known bugs that makes node go into a state

where they cannot close existing and won't accept any new connections. This is

mentioned multiple times in http://www.rabbitmq.com/which-erlang.html. A node restart temporarily

makes the issue go away.

RabbitMQ 3.5.6 is not compatible with later Erlang versions, so in order to move away from

18.x upgrading RabbitMQ to at least 3.6.15 will be necessary:

* http://www.rabbitmq.com/changelog.html

* http://www.rabbitmq.com/upgrade.html

3.5.x has been out of support for over 6 months, Erlang 18.x likely won't see any new releases either

and has known issues that severely affect RabbitMQ.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Fei Jiang

unread,

Apr 26, 2018, 2:26:30 AM4/26/18

to rabbitmq-users

thank you Michael ,

/etc/hosts:

172.28.8.212 rabbitmqNode1

172.28.8.211 rabbitmqNode0

rabbitmqctl cluster_status

Cluster status of node rabbit@rabbitmqNode0 ...

[{nodes,[{disc,[rabbit@rabbitmqNode0,rabbit@rabbitmqNode1]}]},

{running_nodes,[rabbit@rabbitmqNode1,rabbit@rabbitmqNode0]},

{cluster_name,<<"rabbit@rabbitmq_test2">>},

{partitions,[]}]

rabbitmqnode0-sasl.log at that time no logging ,and the rab...@rabbitmqNode0.log :

=WARNING REPORT==== 24-Apr-2018::12:24:28 ===

closing AMQP connection <0.25682.614> (172.28.1.122:60102 -> 172.28.8.220:5672):

connection_closed_abruptly

=INFO REPORT==== 24-Apr-2018::12:24:50 ===

accepting AMQP connection <0.19854.77> (172.28.1.122:33274 -> 172.28.8.220:5672)

在 2018年4月26日星期四 UTC+8上午9:40:26，Michael Klishin写道：

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

rabbit@rabbitmqNode0.log

Michael Klishin

unread,

Apr 26, 2018, 7:18:20 AM4/26/18

to rabbitm...@googlegroups.com

So as far as RabbitMQ is concerned, there *was* a successful connection

and then the *client* (or an intermediary) closed its TCP connection:

> connection_closed_abruptly

See https://www.rabbitmq.com/logging.html#connection-lifecycle-events.

Two primary suspects now are the aforementioned Erlang 18.x bugs and a proxy/load balancer of some kind that closes

the connection. This is mentioned in http://www.rabbitmq.com/heartbeats.html#tcp-proxies.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Reply all

Reply to author

Forward