rabbitmq server noproc errors on new sync of mirrored queue cluster

556 views

Skip to first unread message

Mike Zupan

unread,

Aug 28, 2016, 10:39:52 AM8/28/16

to rabbitmq-users

I have the a mirrored queue cluster right now with only a single cluster member active. The cluster has around 192 vhosts and in total around 3000 queues.

When I try to join a new member to the cluster I get the following example

This is for rabbitmq 3.6.3

I've tried to set the following on both nodes with no luck

RABBITMQ_SERVER_ERL_ARGS="+K true +A30 +P 15048576 -kernel inet_default_connect_options [{nodelay,true}]"

Below is the error

=CRASH REPORT==== 28-Aug-2016::14:28:44 ===

crasher:

initial call: gen:init_it/6

pid: <0.28893.1>

registered_name: []

exception exit: {noproc,

{gen_server2,call,

[msg_store_persistent,

{new_client_state,

<<127,9,176,203,223,47,49,235,159,250,16,96,

240,73,69,246>>,

<0.28893.1>,

#Fun<rabbit_variable_queue.1.95522769>,

#Fun<rabbit_variable_queue.20.95522769>},

infinity]}}

in function gen_server2:terminate/3 (src/gen_server2.erl, line 1143)

ancestors: [<0.3002.1>,rabbit_amqqueue_sup_sup,rabbit_sup,<0.219.0>]

messages: [{'EXIT',<0.3002.1>,shutdown}]

links: [<0.28899.1>]

dictionary: [{rand_seed,

{#{max => 288230376151711743,

next => #Fun<rand.8.45830090>,

type => exsplus,

uniform => #Fun<rand.9.45830090>,

uniform_n => #Fun<rand.10.45830090>},

[146366330958899148|286074369381688491]}},

{process_name,

{rabbit_mirror_queue_slave,

{resource,

<<"133cf901-7943-4d4e-8b5f-e7a5cc792f27">>,

queue,

<<"369ec7d5-b1ff-4164-ad7c-5ace8bc5895a">>}}},

{guid,{{2131341515,3744412139,2683965536,4031333878},0}}]

trap_exit: true

status: running

heap_size: 1598

stack_size: 27

reductions: 7462

neighbours:

neighbour: [{pid,<0.28899.1>},

{registered_name,[]},

{initial_call,

{gen,init_it,

['Argument__1','Argument__2','Argument__3',

'Argument__4','Argument__5','Argument__6']}},

{current_function,{gen_server2,process_next_msg,1}},

{ancestors,

[<0.28893.1>,<0.3002.1>,rabbit_amqqueue_sup_sup,

rabbit_sup,<0.219.0>]},

{messages,[]},

{links,[<0.28893.1>]},

{dictionary,

[{rand_seed,

{#{max => 288230376151711743,

next => #Fun<rand.8.45830090>,

type => exsplus,

uniform => #Fun<rand.9.45830090>,

uniform_n => #Fun<rand.10.45830090>},

[160604897944066945|149112999893747744]}},

{process_name,

{gm,{resource,

<<"133cf901-7943-4d4e-8b5f-e7a5cc792f27">>,

queue,

<<"369ec7d5-b1ff-4164-ad7c-5ace8bc5895a">>}}}]},

{trap_exit,false},

{status,waiting},

{heap_size,610},

{stack_size,7},

{reductions,741}]

=SUPERVISOR REPORT==== 28-Aug-2016::14:28:44 ===

Supervisor: {<0.3002.1>,rabbit_amqqueue_sup}

Context: shutdown_error

Reason: {noproc,

{gen_server2,call,

[msg_store_persistent,

{new_client_state,

<<127,9,176,203,223,47,49,235,159,250,16,96,240,

73,69,246>>,

<0.28893.1>,

#Fun<rabbit_variable_queue.1.95522769>,

#Fun<rabbit_variable_queue.20.95522769>},

infinity]}}

Offender: [{pid,<0.28893.1>},

{name,rabbit_amqqueue},

{mfargs,

{rabbit_prequeue,start_link,

[{amqqueue,

{resource,

<<"133cf901-7943-4d4e-8b5f-e7a5cc792f27">>,

queue,

<<"369ec7d5-b1ff-4164-ad7c-5ace8bc5895a">>},

true,false,none,[],<6601.5555.0>,[],[],

[rabbit@5b2197b15791cdfb29b37a724fc2b042,

rabbit@77ce2615e1f166650b32ad2b4e996679],

[{vhost,

<<"133cf901-7943-4d4e-8b5f-e7a5cc792f27">>},

{name,<<"ha-all">>},

{pattern,<<".*">>},

{'apply-to',<<"all">>},

{definition,

[{<<"ha-mode">>,<<"all">>},

{<<"ha-promote-on-shutdown">>,

<<"always">>},

{<<"ha-sync-mode">>,<<"automatic">>}]},

{priority,50}],

[{<6601.30959.0>,<6601.5555.0>}],

[],live},

slave,<0.3001.1>]}},

{restart_type,intrinsic},

{shutdown,30000},

{child_type,worker}]

Michael Klishin

unread,

Aug 29, 2016, 5:28:41 AM8/29/16

to rabbitm...@googlegroups.com

All this particular log says is that a process named msg_store_persistent has terminated. There are not traces of

the cause.

I have a feeling that this was reported via another (non-public) channel and so far it looks

like https://github.com/rabbitmq/rabbitmq-server/issues/803 is one of the issues involved.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Staff Software Engineer, Pivotal/RabbitMQ

Mike Zupan

unread,

Aug 29, 2016, 12:45:33 PM8/29/16

to rabbitmq-users

Looks like the issue.. Is the issue saying its milestone is only 3.7.0?

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.