Virtual host is not accessible

Przemyslaw Karasek

unread,

Jun 12, 2020, 3:51:30 AM6/12/20

to rabbitmq-users

Hi all,
I setup rabbitmq in docker container with azure storage in order to support configuration
and have persistency of messages. From time to time (each 10-14 days) I have following
crash information in the logs:

2020-06-12 04:37:46 =CRASH REPORT====

crasher:

initial call: rabbit_msg_store:init/1

pid: <0.498.0>

registered_name: []

exception exit: {{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/93FSTXOHBKKAXMNOYHTBZS24V/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,608}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

ancestors: [<0.493.0>,<0.492.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.268.0>]

message_queue_len: 0

messages: []

links: [<0.493.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 1598

stack_size: 28

reductions: 2066

neighbours:

2020-06-12 04:37:46 =CRASH REPORT====

crasher:

initial call: rabbit_vhost_process:init/1

pid: <0.494.0>

registered_name: []

exception exit: {{error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/93FSTXOHBKKAXMNOYHTBZS24V/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/93FSTXOHBKKAXMNOYHTBZS24V",undefined,{#Fun<rabbit_variable_queue.0.43870038>,ok}]},transient,30000,worker,[rabbit_msg_store]}}},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,600}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

ancestors: [<0.492.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.268.0>]

message_queue_len: 0

messages: []

links: [<0.492.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 6772

stack_size: 28

reductions: 22995

neighbours:

2020-06-12 04:37:46 =SUPERVISOR REPORT====

Supervisor: {<0.492.0>,rabbit_vhost_sup_wrapper}

Context: start_error

Reason: {error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/93FSTXOHBKKAXMNOYHTBZS24V/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/93FSTXOHBKKAXMNOYHTBZS24V",undefined,{#Fun<rabbit_variable_queue.0.43870038>,ok}]},transient,30000,worker,[rabbit_msg_store]}}}

Offender: [{pid,undefined},{id,rabbit_vhost_process},{mfargs,{rabbit_vhost_process,start_link,[<<"0r0-dev-weu-poc">>]}},{restart_type,permanent},{shutdown,30000},{child_type,worker}]

2020-06-12 04:37:46 =CRASH REPORT====

crasher:

initial call: rabbit_msg_store:init/1

pid: <0.508.0>

registered_name: []

exception exit: {{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/BUPU1RFQENEABZ5PX4TENPR2Y/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,608}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

ancestors: [<0.502.0>,<0.501.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.268.0>]

message_queue_len: 0

messages: []

links: [<0.502.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 1598

stack_size: 28

reductions: 2063

neighbours:

2020-06-12 04:37:46 =CRASH REPORT====

crasher:

initial call: rabbit_vhost_process:init/1

pid: <0.503.0>

registered_name: []

exception exit: {{error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/BUPU1RFQENEABZ5PX4TENPR2Y/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/BUPU1RFQENEABZ5PX4TENPR2Y",undefined,{#Fun<rabbit_variable_queue.0.43870038>,ok}]},transient,30000,worker,[rabbit_msg_store]}}},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,600}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

ancestors: [<0.501.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.268.0>]

message_queue_len: 0

messages: []

links: [<0.501.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 6772

stack_size: 28

reductions: 23242

neighbours:

2020-06-12 04:37:46 =SUPERVISOR REPORT====

Supervisor: {<0.501.0>,rabbit_vhost_sup_wrapper}

Context: start_error

Reason: {error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/BUPU1RFQENEABZ5PX4TENPR2Y/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/BUPU1RFQENEABZ5PX4TENPR2Y",undefined,{#Fun<rabbit_variable_queue.0.43870038>,ok}]},transient,30000,worker,[rabbit_msg_store]}}}

Offender: [{pid,undefined},{id,rabbit_vhost_process},{mfargs,{rabbit_vhost_process,start_link,[<<"bqm-weu-v10qa-npr">>]}},{restart_type,permanent},{shutdown,30000},{child_type,worker}]

2020-06-12 04:37:47 =CRASH REPORT====

crasher:

initial call: rabbit_msg_store:init/1

pid: <0.517.0>

registered_name: []

exception exit: {{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/C1E1HSXENK4GHYRZGTCJUA69N/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,608}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

ancestors: [<0.512.0>,<0.511.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.268.0>]

message_queue_len: 0

messages: []

links: [<0.512.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 1598

stack_size: 28

reductions: 2063

neighbours:

2020-06-12 04:37:47 =CRASH REPORT====

crasher:

initial call: rabbit_vhost_process:init/1

pid: <0.513.0>

registered_name: []

exception exit: {{error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/C1E1HSXENK4GHYRZGTCJUA69N/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/C1E1HSXENK4GHYRZGTCJUA69N",undefined,{#Fun<rabbit_variable_queue.0.43870038>,ok}]},transient,30000,worker,[rabbit_msg_store]}}},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,600}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

ancestors: [<0.511.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.268.0>]

message_queue_len: 0

messages: []

links: [<0.511.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 6772

stack_size: 28

reductions: 22995

neighbours:

2020-06-12 04:37:47 =SUPERVISOR REPORT====

Supervisor: {<0.511.0>,rabbit_vhost_sup_wrapper}

Context: start_error

Reason: {error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/C1E1HSXENK4GHYRZGTCJUA69N/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/C1E1HSXENK4GHYRZGTCJUA69N",undefined,{#Fun<rabbit_variable_queue.0.43870038>,ok}]},transient,30000,worker,[rabbit_msg_store]}}}

Offender: [{pid,undefined},{id,rabbit_vhost_process},{mfargs,{rabbit_vhost_process,start_link,[<<"0r0-test-vhost1">>]}},{restart_type,permanent},{shutdown,30000},{child_type,worker}]

2020-06-12 04:37:47 =CRASH REPORT====

crasher:

initial call: rabbit_msg_store:init/1

pid: <0.527.0>

registered_name: []

exception exit: {{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/3WVXFJDG54XXY2YO0LV7C7249/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,608}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

ancestors: [<0.522.0>,<0.521.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.268.0>]

message_queue_len: 0

messages: []

links: [<0.522.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 1598

stack_size: 28

reductions: 2063

neighbours:

2020-06-12 04:37:47 =CRASH REPORT====

crasher:

initial call: rabbit_vhost_process:init/1

pid: <0.523.0>

registered_name: []

exception exit: {{error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/3WVXFJDG54XXY2YO0LV7C7249/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/3WVXFJDG54XXY2YO0LV7C7249",undefined,{#Fun<rabbit_variable_queue.0.43870038>,ok}]},transient,30000,worker,[rabbit_msg_store]}}},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,600}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

ancestors: [<0.521.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.268.0>]

message_queue_len: 0

messages: []

links: [<0.521.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 6772

stack_size: 28

reductions: 24359

neighbours:

2020-06-12 04:37:47 =SUPERVISOR REPORT====

Supervisor: {<0.521.0>,rabbit_vhost_sup_wrapper}

Context: start_error

Reason: {error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/3WVXFJDG54XXY2YO0LV7C7249/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/3WVXFJDG54XXY2YO0LV7C7249",undefined,{#Fun<rabbit_variable_queue.0.43870038>,ok}]},transient,30000,worker,[rabbit_msg_store]}}}

Offender: [{pid,undefined},{id,rabbit_vhost_process},{mfargs,{rabbit_vhost_process,start_link,[<<"localhost">>]}},{restart_type,permanent},{shutdown,30000},{child_type,worker}]

2020-06-12 04:37:48 =CRASH REPORT====

crasher:

initial call: rabbit_msg_store:init/1

pid: <0.536.0>

registered_name: []

exception exit: {{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/1J2GS4B20D9E9OWA0EJG7OU6X/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,608}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

ancestors: [<0.531.0>,<0.530.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.268.0>]

message_queue_len: 0

messages: []

links: [<0.531.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 1598

stack_size: 28

reductions: 2063

neighbours:

2020-06-12 04:37:48 =CRASH REPORT====

crasher:

initial call: rabbit_vhost_process:init/1

pid: <0.532.0>

registered_name: []

exception exit: {{error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/1J2GS4B20D9E9OWA0EJG7OU6X/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/1J2GS4B20D9E9OWA0EJG7OU6X",undefined,{#Fun<rabbit_variable_queue.0.43870038>,ok}]},transient,30000,worker,[rabbit_msg_store]}}},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,600}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

ancestors: [<0.530.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.268.0>]

message_queue_len: 0

messages: []

links: [<0.530.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 6772

stack_size: 28

reductions: 24422

neighbours:

2020-06-12 04:37:48 =SUPERVISOR REPORT====

Supervisor: {<0.530.0>,rabbit_vhost_sup_wrapper}

Context: start_error

Reason: {error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/1J2GS4B20D9E9OWA0EJG7OU6X/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/1J2GS4B20D9E9OWA0EJG7OU6X",undefined,{#Fun<rabbit_variable_queue.0.43870038>,ok}]},transient,30000,worker,[rabbit_msg_store]}}}

Offender: [{pid,undefined},{id,rabbit_vhost_process},{mfargs,{rabbit_vhost_process,start_link,[<<"0r0-dev-weu">>]}},{restart_type,permanent},{shutdown,30000},{child_type,worker}]

2020-06-12 04:37:48 =CRASH REPORT====

crasher:

initial call: rabbit_msg_store:init/1

pid: <0.546.0>

registered_name: []

exception exit: {{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,608}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

ancestors: [<0.541.0>,<0.540.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.268.0>]

message_queue_len: 0

messages: []

links: [<0.541.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 1598

stack_size: 28

reductions: 2063

neighbours:

2020-06-12 04:37:48 =CRASH REPORT====

crasher:

initial call: rabbit_vhost_process:init/1

pid: <0.542.0>

registered_name: []

exception exit: {{error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L",undefined,{#Fun<rabbit_variable_queue.0.43870038>,ok}]},transient,30000,worker,[rabbit_msg_store]}}},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,600}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

ancestors: [<0.540.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.268.0>]

message_queue_len: 0

messages: []

links: [<0.540.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 6772

stack_size: 28

reductions: 22806

neighbours:

2020-06-12 04:37:48 =SUPERVISOR REPORT====

Supervisor: {<0.540.0>,rabbit_vhost_sup_wrapper}

Context: start_error

Reason: {error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L",undefined,{#Fun<rabbit_variable_queue.0.43870038>,ok}]},transient,30000,worker,[rabbit_msg_store]}}}

Offender: [{pid,undefined},{id,rabbit_vhost_process},{mfargs,{rabbit_vhost_process,start_link,[<<"/">>]}},{restart_type,permanent},{shutdown,30000},{child_type,worker}]

2020-06-12 04:37:49 =CRASH REPORT====

crasher:

initial call: rabbit_msg_store:init/1

pid: <0.555.0>

registered_name: []

exception exit: {{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/4ZAYFZI8WN1K75LF8M67EFD5G/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,608}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

ancestors: [<0.550.0>,<0.549.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.268.0>]

message_queue_len: 0

messages: []

links: [<0.550.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 1598

stack_size: 28

reductions: 2063

neighbours:

2020-06-12 04:37:49 =CRASH REPORT====

crasher:

initial call: rabbit_vhost_process:init/1

pid: <0.551.0>

registered_name: []

exception exit: {{error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/4ZAYFZI8WN1K75LF8M67EFD5G/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/4ZAYFZI8WN1K75LF8M67EFD5G",undefined,{#Fun<rabbit_variable_queue.0.43870038>,ok}]},transient,30000,worker,[rabbit_msg_store]}}},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,600}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

ancestors: [<0.549.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.268.0>]

message_queue_len: 0

messages: []

links: [<0.549.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 6772

stack_size: 28

reductions: 24267

neighbours:

2020-06-12 04:37:49 =SUPERVISOR REPORT====

Supervisor: {<0.549.0>,rabbit_vhost_sup_wrapper}

Context: start_error

Reason: {error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/4ZAYFZI8WN1K75LF8M67EFD5G/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/4ZAYFZI8WN1K75LF8M67EFD5G",undefined,{#Fun<rabbit_variable_queue.0.43870038>,ok}]},transient,30000,worker,[rabbit_msg_store]}}}

Offender: [{pid,undefined},{id,rabbit_vhost_process},{mfargs,{rabbit_vhost_process,start_link,[<<"przemekk">>]}},{restart_type,permanent},{shutdown,30000},{child_type,worker}]

2020-06-12 04:37:49 =CRASH REPORT====

crasher:

initial call: rabbit_msg_store:init/1

pid: <0.565.0>

registered_name: []

exception exit: {{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/EC2YLRT4XCYQ7FPZA9XGI2K8/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,608}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

ancestors: [<0.560.0>,<0.559.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.268.0>]

message_queue_len: 0

messages: []

links: [<0.560.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 1598

stack_size: 28

reductions: 2051

neighbours:

2020-06-12 04:37:49 =CRASH REPORT====

crasher:

initial call: rabbit_vhost_process:init/1

pid: <0.561.0>

registered_name: []

exception exit: {{error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/EC2YLRT4XCYQ7FPZA9XGI2K8/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/EC2YLRT4XCYQ7FPZA9XGI2K8",undefined,{#Fun<rabbit_variable_queue.0.43870038>,ok}]},transient,30000,worker,[rabbit_msg_store]}}},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,600}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

ancestors: [<0.559.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.268.0>]

message_queue_len: 0

messages: []

links: [<0.559.0>]

dictionary: []

trap_exit: true

status: running

heap_size: 4185

stack_size: 28

reductions: 23198

neighbours:

2020-06-12 04:37:49 =SUPERVISOR REPORT====

Supervisor: {<0.559.0>,rabbit_vhost_sup_wrapper}

Context: start_error

Reason: {error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/EC2YLRT4XCYQ7FPZA9XGI2K8/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,731}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/EC2YLRT4XCYQ7FPZA9XGI2K8",undefined,{#Fun<rabbit_variable_queue.0.43870038>,ok}]},transient,30000,worker,[rabbit_msg_store]}}}

Offender: [{pid,undefined},{id,rabbit_vhost_process},{mfargs,{rabbit_vhost_process,start_link,[<<"bztest">>]}},{restart_type,permanent},{shutdown,30000},{child_type,worker}]

2020-06-12 04:38:46 =ERROR REPORT====

Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.1249283286.883687425.134436>}} from <0.593.0> to <0.13795.70> in an old incarnation (1591004445) of this node (1591936656)

2020-06-12 04:38:46 =ERROR REPORT====

Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.1249283286.883687425.134436>}} from <0.593.0> to <0.949.0> in an old incarnation (1591004445) of this node (1591936656)

2020-06-12 04:38:46 =ERROR REPORT====

Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.1249283286.883687425.134436>}} from <0.593.0> to <0.13787.70> in an old incarnation (1591004445) of this node (1591936656)

2020-06-12 04:38:46 =ERROR REPORT====

Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.1249283286.883687425.134436>}} from <0.593.0> to <0.1065.0> in an old incarnation (1591004445) of this node (1591936656)

2020-06-12 04:38:46 =ERROR REPORT====

Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.1249283286.883687425.134436>}} from <0.593.0> to <0.13815.70> in an old incarnation (1591004445) of this node (1591936656)

2020-06-12 04:38:46 =ERROR REPORT====

Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.1249283286.883687425.134436>}} from <0.593.0> to <0.952.0> in an old incarnation (1591004445) of this node (1591936656)

2020-06-12 04:38:46 =ERROR REPORT====

Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.1249283286.883687425.134436>}} from <0.593.0> to <0.1068.0> in an old incarnation (1591004445) of this node (1591936656)

I do not understand reason of this crash and not sure how to improve configuration in order to prevent from stopping vhosts.

Any suggestions?
Thank you in advance!

Warren Hua

unread,

Jun 12, 2020, 3:55:30 AM6/12/20

to rabbitm...@googlegroups.com

Przemyslaw Karasek wrote:
> I setup rabbitmq in docker container with azure storage in order to
> support configuration
> and have persistency of messages. From time to time (each 10-14 days) I
> have following
> crash information in the logs:

What are the server/erlang version?
Do you have filesystem issues such as FD exceeds or other IO error?

Regards
Warren

Przemyslaw Karasek

unread,

Jun 12, 2020, 4:13:28 AM6/12/20

to rabbitmq-users

Here are versions:

RabbitMQ 3.8.4Erlang 23.0.2

On azure storage traffic is low. But i can see some issues which I have to analize.

You think that this issues can force rabbitmq to stop.

Is there any way to force rabbitmq automatically restart to release virtual hosts after some time when we have issue with storage ?

Regards

Przemek

Warren Hua

unread,

Jun 12, 2020, 4:20:14 AM6/12/20

to rabbitm...@googlegroups.com

Przemyslaw Karasek wrote:
> You think that this issues can force rabbitmq to stop.
> Is there any way to force rabbitmq automatically restart to release
> virtual hosts after some time when we have issue with storage ?
>

From your statistics, the storage had some issues, which did influence
rabbitmq's stability. We once had the same case.

regards,
Warren

Przemyslaw Karasek

unread,

Jun 12, 2020, 4:38:13 AM6/12/20

to rabbitmq-users

Can you give me more details ?

You resolved issues with storage and then it was ok ?

Did you apply some changes in configuration of rabbitmq ?

Here problem is that almost at the same time I had this issue on two instances.

Regards,

Przemek

Reply all

Reply to author

Forward