RabbitMQ service crashing and cannot restart the service without restarting the server

433 views

Skip to first unread message

Krishna M

unread,

Jul 31, 2018, 4:13:42 AM7/31/18

to rabbitmq-users

We are using RabitMQ 3.6.1 with Erland 19.2 on a Windows server 2012 R2. We have 3 servers in a cluster with an n+1 config as below policy

ha-mode:	exactly
ha-params:	2
ha-sync-mode:	automatic

Now since last couple of weeks one of the servers keeps crashing and going into a weird state. We cannot restart the rabbitmq service and the only way we can restart is by restarting the server itself. We have put in some alerting to note the time the server goes down and according to our alerts the rabbit server went down at around 21:10 and from sasl log, i can find the below entry. I am not sure what this entry means so any help is appreciated please.

Also one more pattern i am seeing is that our normal log file is reasonably big (in mb) and sasl log is always small (in kb). But when we see this rabbit crash we see sasl log grow to a big size (larger than the main log file). I dont know if this is useful for any investigation?

=CRASH REPORT==== 30-Jul-2018::21:16:53 ===

crasher:

initial call: rabbit_msg_store_gc:init/1

pid: <0.316.0>

registered_name: []

exception exit: {{badmatch,{error,eacces}},

[{rabbit_msg_store,safe_file_delete,3,

[{file,"src/rabbit_msg_store.erl"},{line,1411}]},

{rabbit_msg_store_gc,'-attempt_action/3-fun-0-',1,

[{file,"src/rabbit_msg_store_gc.erl"},{line,127}]},

{lists,'-filter/2-lc$^0/1-0-',2,

[{file,"lists.erl"},{line,1286}]},

{rabbit_msg_store_gc,attempt_action,3,

[{file,"src/rabbit_msg_store_gc.erl"},{line,126}]},

{rabbit_msg_store_gc,handle_cast,2,

[{file,"src/rabbit_msg_store_gc.erl"},{line,92}]},

{gen_server2,handle_msg,2,

[{file,"src/gen_server2.erl"},{line,1049}]},

{proc_lib,wake_up,3,

[{file,"proc_lib.erl"},{line,257}]}]}

in function gen_server2:terminate/3 (src/gen_server2.erl, line 1160)

ancestors: [msg_store_persistent,rabbit_sup,<0.168.0>]

messages: []

links: [<0.313.0>]

dictionary: [{fhc_age_tree,{0,nil}}]

trap_exit: false

status: running

heap_size: 4185

stack_size: 27

reductions: 891884789

neighbours:

=CRASH REPORT==== 30-Jul-2018::21:16:53 ===

crasher:

initial call: rabbit_msg_store:init/1

pid: <0.313.0>

registered_name: msg_store_persistent

exception exit: {noproc,{gen_server2,call,[<0.316.0>,stop,infinity]}}

in function gen_server2:terminate/3 (src/gen_server2.erl, line 1149)

ancestors: [rabbit_sup,<0.168.0>]

messages: []

links: [<0.169.0>,#Port<0.1747441>]

dictionary: [{{credit_to,<0.26403.72>},205},

{{#Ref<0.0.48496642.211232>,fhc_handle},

{handle,{file_descriptor,prim_file,{#Port<0.1747441>,600}},

#Ref<0.0.48496642.211232>,11005414,false,0,1048576,

[],<<>>,0,0,0,0,0,true,

"d:/RabbitMQ/db/rabbit@SERVERNAME-mnesia/msg_store_persistent/3756373.rdq",

[raw,binary,write],

[{write_buffer,1048576},{read_buffer,1048576}],

true,false,388120282522}},

{{credit_to,<0.26752.72>},481},

{{credit_to,<0.26774.72>},496},

{{credit_to,<0.26937.72>},423},

{{credit_to,<0.26782.72>},499},

{{credit_to,<0.26498.72>},448},

{{credit_to,<0.26468.72>},499},

{{credit_to,<0.26421.72>},489},

{{credit_to,<0.26450.72>},496},

{{credit_to,<0.26934.72>},499},

{{credit_to,<0.26657.72>},487},

{{"d:/RabbitMQ/db/rabbit@SERVERNAME-mnesia/msg_store_persistent/3756373.rdq",

fhc_file},

{file,0,true}},

{{credit_to,<0.26396.72>},42},

{{credit_to,<0.26638.72>},487},

{{credit_to,<0.26610.72>},332},

{{credit_to,<0.26342.72>},423},

{fhc_age_tree,{1,

{{388120282522,#Ref<0.0.48496642.211232>},

true,nil,nil}}},

{{credit_to,<0.27065.72>},130},

{{credit_to,<0.26431.72>},485},

{{credit_to,<0.26585.72>},332}]

trap_exit: true

status: running

heap_size: 2586

stack_size: 27

reductions: 2598438954

neighbours:

=SUPERVISOR REPORT==== 30-Jul-2018::21:16:53 ===

Supervisor: {local,rabbit_sup}

Context: child_terminated

Reason: {noproc,{gen_server2,call,[<0.316.0>,stop,infinity]}}

Offender: [{pid,<0.313.0>},

{id,msg_store_persistent},

{mfargs,

{rabbit_msg_store,start_link,

[msg_store_persistent,

"d:/RabbitMQ/db/rabbit@SERVERNAME-mnesia",[],

{#Fun<rabbit_queue_index.2.122150458>,

{start,[]}}]}},

{restart_type,transient},

{shutdown,4294967295},

{child_type,worker}]

There are a number of errors in the file after the above entries.

Krishna M

unread,

Jul 31, 2018, 4:20:06 AM7/31/18

to rabbitmq-users

Also to add to the above, I see below entries at the same time in the rabbit error log

=ERROR REPORT==== 30-Jul-2018::21:16:53 ===

** Generic server <0.316.0> terminating

** Last message in was {'$gen_cast',{delete,3756294}}

** When Server state == {state,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

[],

{gc_state,

"d:/RabbitMQ/db/rabbit@SERVERNAME-mnesia/msg_store_persistent",

rabbit_msg_store_ets_index,

{state,315470,

"d:/RabbitMQ/db/rabbit@SERVERNAME-mnesia/msg_store_persistent"},

311373,319567,<0.313.0>}}

** Reason for termination ==

** {{badmatch,{error,eacces}},

[{rabbit_msg_store,safe_file_delete,3,

[{file,"src/rabbit_msg_store.erl"},{line,1411}]},

{rabbit_msg_store_gc,'-attempt_action/3-fun-0-',1,

[{file,"src/rabbit_msg_store_gc.erl"},{line,127}]},

{lists,'-filter/2-lc$^0/1-0-',2,[{file,"lists.erl"},{line,1286}]},

{rabbit_msg_store_gc,attempt_action,3,

[{file,"src/rabbit_msg_store_gc.erl"},{line,126}]},

{rabbit_msg_store_gc,handle_cast,2,

[{file,"src/rabbit_msg_store_gc.erl"},{line,92}]},

{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1049}]},

{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,257}]}]}

=ERROR REPORT==== 30-Jul-2018::21:16:53 ===

** Generic server msg_store_persistent terminating

** Last message in was {'EXIT',<0.316.0>,

{{badmatch,{error,eacces}},

[{rabbit_msg_store,safe_file_delete,3,

[{file,"src/rabbit_msg_store.erl"},

{line,1411}]},

{rabbit_msg_store_gc,'-attempt_action/3-fun-0-',

[{file,"src/rabbit_msg_store_gc.erl"},

{line,127}]},

{lists,'-filter/2-lc$^0/1-0-',2,

[{file,"lists.erl"},{line,1286}]},

{rabbit_msg_store_gc,attempt_action,3,

[{file,"src/rabbit_msg_store_gc.erl"},

{line,126}]},

{rabbit_msg_store_gc,handle_cast,2,

[{file,"src/rabbit_msg_store_gc.erl"},

{line,92}]},

{gen_server2,handle_msg,2,

[{file,"src/gen_server2.erl"},{line,1049}]},

{proc_lib,wake_up,3,

[{file,"proc_lib.erl"},{line,257}]}]}}

** When Server state == {msstate,

"d:/RabbitMQ/db/rabbit@SERVERNAME-mnesia/msg_store_persistent",

rabbit_msg_store_ets_index,

{state,315470,

"d:/RabbitMQ/db/rabbit@SERVERNAME-mnesia/msg_store_persistent"},

3756373,#Ref<0.0.48496642.211232>,

{dict,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

undefined,241345996,455274441,[],<0.316.0>,319567,

311373,323664,327761,

{set,0,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[],[]}}},

{dict,122,25,32,16,125,75,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

[]},

{{[[<<244,14,10,66,208,77,71,76,190,34,30,168,

235,97,26,18>>|

{<0.26526.72>,