'badmatch' termination of rabbitMQ when previously working fine
415 views
Skip to first unread message
Kimberly McKinnis
Jul 17, 2017, 6:33:09 PM7/17/17
to rabbitmq-users
Hi. I've taken on the management of some Sensu servers running RabbitMQ, they all run with the same Chef recipe, and today one of refuses to properly run rabbitmq. I'm still learning details, so please bear with me. It starts fine, but quickly starts logging 'bad match'. It will list connections until you ask for other details as well. I've included some output and log snippet. The log's reason for termination is fairly unclear to me, the machine on the other end at 10.64.4.11 seems fine. We are doing everything through the CLI (and Chef) on machines running OL7. I've been through Google and mail list archives but nothing clarifies to me what 'bad match' might be. Thank you!
[root@ots-76b7ef13 rabbitmq]# less /var/log/rabbitmq/rab...@ots-76b7ef13.log
=INFO REPORT==== 17-Jul-2017::21:14:57 ===
accepting AMQP connection <0.690.0> (10.64.4.155:42359 -> 10.64.4.11:5671)
=INFO REPORT==== 17-Jul-2017::21:14:57 ===
accepting AMQP connection <0.749.0> (10.64.0.58:6302 -> 10.64.4.11:5671)
=ERROR REPORT==== 17-Jul-2017::21:15:49 ===
** Generic server <0.2040.0> terminating
** Last message in was connect
** When Server state == {<0.2039.0>,
{amqp_params_direct,<<"guest">>,<<"guest">>,<<"/">>,
'rabbit@ots-76b7ef13',none,[]}}
** Reason for termination ==
** {function_clause,
[{amqp_gen_connection,terminate,
[{{badmatch,
<<131,103,119,19,114,97,98,98,105,116,64,111,116,115,45,55,
54,98,55,101,102,49,51,0,0,7,248,0,0,0,0,1>>},
[{rabbit_misc,decompose_pid,1,
[{file,"src/rabbit_misc.erl"},{line,726}]},
{rabbit_misc,pid_to_string,1,
[{file,"src/rabbit_misc.erl"},{line,696}]},
{amqp_direct_connection,ensure_adapter_info,1,
[{file,"src/amqp_direct_connection.erl"},{line,164}]},
{amqp_direct_connection,connect,4,
[{file,"src/amqp_direct_connection.erl"},{line,132}]},
{amqp_gen_connection,handle_call,3,
[{file,"src/amqp_gen_connection.erl"},{line,171}]},
{gen_server,try_handle_call,4,
[{file,"gen_server.erl"},{line,636}]},
{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,665}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,247}]}]},
{<0.2039.0>,
{amqp_params_direct,<<"guest">>,<<"guest">>,<<"/">>,
'rabbit@ots-76b7ef13',none,[]}}],
[{file,"src/amqp_gen_connection.erl"},{line,230}]},
[{file,"src/amqp_gen_connection.erl"},{line,230}]},
{gen_server,try_terminate,3,[{file,"gen_server.erl"},{line,648}]},
{gen_server,terminate,10,[{file,"gen_server.erl"},{line,833}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
** Client <0.2020.0> stacktrace
** [{gen,do_call,4,[{file,"gen.erl"},{line,169}]},
{gen_server,call,3,[{file,"gen_server.erl"},{line,210}]},
{rabbit_mgmt_util,with_channel,5,
[{file,"src/rabbit_mgmt_util.erl"},{line,594}]},
{webmachine_resource,resource_call,3,
[{file,"src/webmachine_resource.erl"},{line,186}]},
{webmachine_resource,do,3,
[{file,"src/webmachine_resource.erl"},{line,142}]},
{webmachine_decision_core,resource_call,1,
[{file,"src/webmachine_decision_core.erl"},
{line,48}]},
{webmachine_decision_core,decision,1,
[{file,"src/webmachine_decision_core.erl"},
{line,558}]},
{webmachine_decision_core,handle_request,2,
[{file,"src/webmachine_decision_core.erl"},
{line,33}]}]
=ERROR REPORT==== 17-Jul-2017::21:15:49 ===
webmachine error: path="/api/aliveness-test/%2F"
{error,
{exit,
{{function_clause,
[{amqp_gen_connection,terminate,
[{{badmatch,
<<131,103,119,19,114,97,98,98,105,116,64,111,116,115,
45,55,54,98,55,101,102,49,51,0,0,7,248,0,0,0,0,1>>},
[{rabbit_misc,decompose_pid,1,
[{file,"src/rabbit_misc.erl"},{line,726}]},
{rabbit_misc,pid_to_string,1,
[{file,"src/rabbit_misc.erl"},{line,696}]},
{amqp_direct_connection,ensure_adapter_info,1,
[{file,"src/amqp_direct_connection.erl"},{line,164}]},
{amqp_direct_connection,connect,4,
[{file,"src/amqp_direct_connection.erl"},{line,132}]},
{amqp_gen_connection,handle_call,3,
[{file,"src/amqp_gen_connection.erl"},{line,171}]},
{gen_server,try_handle_call,4,
[{file,"gen_server.erl"},{line,636}]},
{gen_server,handle_msg,6,
[{file,"gen_server.erl"},{line,665}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,247}]}]},
{<0.2039.0>,
{amqp_params_direct,<<"guest">>,<<"guest">>,<<"/">>,
'rabbit@ots-76b7ef13',none,[]}}],
[{file,"src/amqp_gen_connection.erl"},{line,230}]},
{gen_server,try_terminate,3,
[{file,"gen_server.erl"},{line,648}]},
{gen_server,terminate,10,[{file,"gen_server.erl"},{line,833}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,247}]}]},
{gen_server,call,[<0.2040.0>,connect,infinity]}},
[{gen_server,call,3,[{file,"gen_server.erl"},{line,214}]},
{rabbit_mgmt_util,with_channel,5,
[{file,"src/rabbit_mgmt_util.erl"},{line,594}]},
{webmachine_resource,resource_call,3,
[{file,"src/webmachine_resource.erl"},{line,186}]},
{webmachine_resource,do,3,
[{file,"src/webmachine_resource.erl"},{line,142}]},
{webmachine_decision_core,resource_call,1,
[{file,"src/webmachine_decision_core.erl"},{line,48}]},
{webmachine_decision_core,decision,1,
[{file,"src/webmachine_decision_core.erl"},{line,558}]},
{webmachine_decision_core,handle_request,2,
[{file,"src/webmachine_decision_core.erl"},{line,33}]},
{rabbit_webmachine,'-makeloop/1-fun-0-',2,
[{file,"src/rabbit_webmachine.erl"},{line,55}]}]}}
[root@ots-76b7ef13 rabbitmq]# rabbitmqctl status
Status of node 'rabbit@ots-76b7ef13' ...
[{pid,39847},
{running_applications,
[{rabbitmq_management_visualiser,"RabbitMQ Visualiser","3.6.1"},
{rabbitmq_management,"RabbitMQ Management Console","3.6.1"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.1"},
{webmachine,"webmachine","1.10.3"},
{mochiweb,"MochiMedia Web Server","2.13.0"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.1"},
{rabbit,"RabbitMQ","3.6.1"},
{ssl,"Erlang/OTP SSL application","8.2"},
{public_key,"Public key infrastructure","1.4.1"},
{asn1,"The Erlang ASN1 compiler version 5.0","5.0"},
{amqp_client,"RabbitMQ AMQP Client","3.6.1"},
{xmerl,"XML parser","1.3.14"},
{inets,"INETS CXC 138 49","6.4"},
{rabbit_common,[],"3.6.1"},
{crypto,"CRYPTO","4.0"},
{compiler,"ERTS CXC 138 10","7.1"},
{os_mon,"CPO CXC 138 46","2.4.2"},
{ranch,"Socket acceptor pool for TCP protocols.","1.2.1"},
{mnesia,"MNESIA CXC 138 12","4.15"},
{syntax_tools,"Syntax tools","2.1.2"},
{sasl,"SASL CXC 138 11","3.0.4"},
{stdlib,"ERTS CXC 138 10","3.4"},
{kernel,"ERTS CXC 138 10","5.3"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang/OTP 20 [RELEASE CANDIDATE 2] [erts-9.0] [source] [64-bit] [smp:72:72] [ds:72:72:10] [async-threads:64] [hipe] [kernel-poll:true]\n"},
{memory,
[{total,127650576},
{connection_readers,603600},
{connection_writers,293960},
{connection_channels,140944},
{connection_other,546896},
{queue_procs,231016},
{queue_slave_procs,0},
{plugins,3206024},
{other_proc,23297056},
{mnesia,155720},
{mgmt_db,1841448},
{msg_index,53936},
{other_ets,1472688},
{binary,31042192},
{code,24420633},
{atom,1000625},
{other_system,39343838}]},
{alarms,[]},
{listeners,[{clustering,25672,"::"},{amqp,5672,"::"},{'amqp/ssl',5671,"::"}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,108025102336},
{disk_free_limit,50000000},
{disk_free,34508943360},
{file_descriptors,
[{total_limit,924},
{total_used,21},
{sockets_limit,829},
{sockets_used,19}]},
{processes,[{limit,1048576},{used,548}]},
{run_queue,0},
{uptime,3657},
{kernel,{net_ticktime,60}}]
[root@ots-76b7ef13 rabbitmq]# ps wgaux | grep rabbitmq
root 11215 0.0 0.0 112652 2360 pts/0 S+ 22:18 0:00 grep --color=auto rabbitmq
root 39156 0.0 0.0 115508 2156 ? S 21:14 0:00 /bin/sh /etc/rc.d/init.d/rabbitmq-server start
root 39158 0.0 0.0 115244 2876 ? S 21:14 0:00 /bin/bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/sbin/rabbitmq-server
root 39162 0.0 0.0 115244 3160 ? S 21:14 0:00 /bin/sh /usr/sbin/rabbitmq-server
root 39177 0.0 0.0 214492 4964 ? S 21:14 0:00 su rabbitmq -s /bin/sh -c /usr/lib/rabbitmq/bin/rabbitmq-server
rabbitmq 39596 0.0 0.0 113124 2992 ? Ss 21:14 0:00 /bin/sh -e /usr/lib/rabbitmq/bin/rabbitmq-server
rabbitmq 39847 12.6 0.1 16642432 392316 ? Sl 21:14 8:06 /usr/lib/erlang/erts-9.0/bin/beam.smp -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.1/ebin -noshell -noinput -s rabbit boot -sname rabbit@ots-76b7ef13 -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rab...@ots-76b7ef13.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rab...@ots-76b7ef13-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.1/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@ots-76b7ef13-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@ots-76b7ef13" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
rabbitmq 39920 0.0 0.0 4304 1344 ? Ss 21:14 0:00 erl_child_setup 1024
rabbitmq 40083 0.0 0.0 11540 928 ? Ss 21:14 0:00 inet_gethost 4
rabbitmq 40084 0.0 0.0 39396 2804 ? S 21:14 0:00 inet_gethost 4
rabbitmq 68531 0.0 0.0 13496 3292 ? S Jul14 0:03 /usr/lib/erlang/erts-9.0/bin/epmd -daemon
[root@ots-76b7ef13 rabbitmq]# rabbitmqctl list_connections
Listing connections ...
sensu 10.64.4.11 57390 running
sensu 10.64.0.59 44938 running
sensu 10.64.5.52 46674 running
sensu 10.64.0.107 48715 running
sensu 10.64.4.11 57391 running
sensu 10.64.0.60 23090 running
sensu 10.64.4.11 57392 running
sensu 10.64.0.154 60202 running
sensu 10.64.0.82 38270 running
sensu 10.64.4.106 36292 running
sensu 10.64.0.37 48232 running
sensu 10.64.5.50 40494 running
sensu 10.64.0.130 46285 running
sensu 10.64.5.51 33927 running
sensu 10.64.4.107 45651 running
sensu 10.64.0.11 53204 running
sensu 10.64.0.12 57955 running
sensu 10.64.4.155 42359 running
sensu 10.64.0.58 6302 running
root@ots-76b7ef13 rabbitmq]# rabbitmqctl list_connections pid
Listing connections ...
Error: {badmatch,<<131,103,119,19,114,97,98,98,105,116,64,111,116,115,45,55,
54,98,55,101,102,49,51,0,0,2,21,0,0,0,0,1>>}
Michael Klishin
Jul 17, 2017, 7:31:19 PM7/17/17
to rabbitm...@googlegroups.com
Hi Kimberly,
According to `rabbitmqctl status` and process 39847's process line you are running on Erlang/OTP 20 RC2. OTP 20 is not currently
supported by RabbitMQ and RC2 specifically has major breaking changes that affect RabbitMQ:
In fact, version 3.6.1 does not support even OTP 19 (3.6.4 is the first version that does: http://www.rabbitmq.com/changelog.html).
Are you using Erlang Solution's apt repo without apt pinning? Can it be that some or all of your nodes were upgraded to 20
unintentionally?
We have a section on apt pinning of Erlang packages in the docs now:
On Tue, Jul 18, 2017 at 1:33 AM, Kimberly McKinnis <k...@starjewel.org> wrote:
Hi. I've taken on the management of some Sensu servers running RabbitMQ, they all run with the same Chef recipe, and today one of refuses to properly run rabbitmq. I'm still learning details, so please bear with me. It starts fine, but quickly starts logging 'bad match'. It will list connections until you ask for other details as well. I've included some output and log snippet. The log's reason for termination is fairly unclear to me, the machine on the other end at 10.64.4.11 seems fine. We are doing everything through the CLI (and Chef) on machines running OL7. I've been through Google and mail list archives but nothing clarifies to me what 'bad match' might be. Thank you!
[root@ots-76b7ef13 rabbitmq]# less /var/log/rabbitmq/rabbit@ots-76b7ef13.log
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
MK
Staff Software Engineer, Pivotal/RabbitMQ
Kimberly McKinnis
Jul 17, 2017, 7:40:21 PM7/17/17
to rabbitmq-users
Thank you for the tip. I assumed we installed from the community Chef cookbook, so our wrappers may need pinned. I appreciate the information. I will investigate the versioning.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages