Queue index issue
194 views
Skip to first unread message
jo...@cloudamqp.com
Apr 14, 2018, 11:36:08 AM4/14/18
to rabbitm...@googlegroups.com
Hi,
We've recently seen an increase in queue index issues. Below an excerpt of a log file when RabbitMQ is trying to start up.
This example is from RabbitMQ 3.7.3 / Erlang 20.1 (but have also seen it in Erlang 20.1/RabbitMQ 3.6.15 and Erlang 20.1/RabbitMQ 3.7.4)
Is there anyway to track down which queue index the issue relates to, so that we don't need to remove all of them?
Circumstantial evidence points to that this is more prone to happen after a cluster has been rebooted.
Similar to https://groups.google.com/d/msg/rabbitmq-users/_96tCMxPCTk/PX8W7YAYDAAJ
/Johan
2018-04-14 14:19:44.282 [info] <0.336.0> Message store "3SKE8T36OWL2LT39IJII04LEL/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2018-04-14 14:19:44.283 [warning] <0.336.0> Message store "3SKE8T36OWL2LT39IJII04LEL/msg_store_persistent": rebuilding indices from scratch
2018-04-14 14:19:46.408 [error] <0.274.0> ** Generic server <0.274.0> terminating
** Last message in was {'$gen_cast',{submit_async,#Fun<rabbit_queue_index.36.122888644>}}
** When Server state == undefined
** Reason for termination ==
** {function_clause,[{ ,journal_minus_segment1,[{no_pub,no_del,ack},{{true,<<183,76,252,216,188,134,145,120,4,107,92,115,245,235,18,91,0,5,105,153,138,25,118,17,0,0,5,145>>,<<>>},no_del,n
o_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.
erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,915}]},{lists,map,2,[{file,"lists
.erl"},{line,1239}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1039}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,906}]}]}
2018-04-14 14:19:46.422 [error] <0.274.0> CRASH REPORT Process <0.274.0> with 1 neighbours exited with reason: no function clause matching rabbit_queue_index:journal_minus_segment1({no_pub,no_del,ack}, {{
true,<<183,76,252,216,188,134,145,120,4,107,92,115,245,235,18,91,0,5,105,153,138,25,118,17,0,...>>,...},...}) line 1231 in gen_server2:terminate/3 line 1161
2018-04-14 14:19:46.423 [error] <0.336.0> CRASH REPORT Process <0.336.0> with 1 neighbours exited with reason: {{function_clause,[{rabbit_queue_index,journal_minus_segment1,[{no_pub,no_del,ack},{{true,<<1
83,76,252,216,188,134,145,120,4,107,92,115,245,235,18,91,0,5,105,153,138,25,118,17,0,0,5,145>>,<<>>},no_del,no_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_
segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_ind
ex,...},...]},...} in gen_server2:init_it/6 line 589
2018-04-14 14:19:46.424 [error] <0.272.0> Supervisor worker_pool_sup had child 1 started with worker_pool_worker:start_link(worker_pool) at <0.274.0> exit with reason no function clause matching rabbit_qu
eue_index:journal_minus_segment1({no_pub,no_del,ack}, {{true,<<183,76,252,216,188,134,145,120,4,107,92,115,245,235,18,91,0,5,105,153,138,25,118,17,0,...>>,...},...}) line 1231 in context child_terminated
2018-04-14 14:19:46.424 [error] <0.341.0> ** Generic server <0.341.0> terminating
** Last message in was {'$gen_cast',{submit_async,#Fun<rabbit_queue_index.36.122888644>}}
** When Server state == undefined
** Reason for termination ==
** {noproc,[{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,"src/rabbit_queue_index.erl"},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,"src/worker_pool_wo
rker.erl"},{line,122}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
2018-04-14 14:19:46.424 [error] <0.341.0> CRASH REPORT Process <0.341.0> with 0 neighbours exited with reason: no such process or port in call to [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,101,114,108]},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,112,111,111,108,95,119,111,114,107,101,114,46,101,114,108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in gen_server2:terminate/3 line 1161
2018-04-14 14:19:46.425 [error] <0.272.0> Supervisor worker_pool_sup had child 1 started with worker_pool_worker:start_link(worker_pool) at <0.341.0> exit with reason no such process or port in call to [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,101,114,108]},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,112,111,111,108,95,119,111,114,107,101,114,46,101,114,108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in context child_terminated
2018-04-14 14:19:46.425 [error] <0.342.0> ** Generic server <0.342.0> terminating
** Last message in was {'$gen_cast',{submit_async,#Fun<rabbit_queue_index.36.122888644>}}
** When Server state == undefined
** Reason for termination ==
** {noproc,[{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,"src/rabbit_queue_index.erl"},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,"src/worker_pool_worker.erl"},{line,122}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
2018-04-14 14:19:46.425 [error] <0.342.0> CRASH REPORT Process <0.342.0> with 0 neighbours exited with reason: no such process or port in call to [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,101,114,108]},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,112,111,111,108,95,119,111,114,107,101,114,46,101,114,108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in gen_server2:terminate/3 line 1161
2018-04-14 14:19:46.426 [error] <0.272.0> Supervisor worker_pool_sup had child 1 started with worker_pool_worker:start_link(worker_pool) at <0.342.0> exit with reason no such process or port in call to [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,101,114,108]},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,112,111,111,108,95,119,111,114,107,101,114,46,101,114,108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in context child_terminated
2018-04-14 14:19:46.427 [error] <0.343.0> ** Generic server <0.343.0> terminating
** Last message in was {'$gen_cast',{submit_async,#Fun<rabbit_queue_index.36.122888644>}}
** When Server state == undefined
** Reason for termination ==
** {noproc,[{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,"src/rabbit_queue_index.erl"},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,"src/worker_pool_worker.erl"},{line,122}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
2018-04-14 14:19:46.427 [error] <0.343.0> CRASH REPORT Process <0.343.0> with 0 neighbours exited with reason: no such process or port in call to [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,101,114,108]},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,112,111,111,108,95,119,111,114,107,101,114,46,101,114,108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in gen_server2:terminate/3 line 1161
2018-04-14 14:19:46.428 [error] <0.272.0> Supervisor worker_pool_sup had child 1 started with worker_pool_worker:start_link(worker_pool) at <0.343.0> exit with reason no such process or port in call to [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,101,114,108]},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,112,111,111,108,95,119,111,114,107,101,114,46,101,114,108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in context child_terminated
2018-04-14 14:19:46.430 [error] <0.344.0> ** Generic server <0.344.0> terminating
We've recently seen an increase in queue index issues. Below an excerpt of a log file when RabbitMQ is trying to start up.
This example is from RabbitMQ 3.7.3 / Erlang 20.1 (but have also seen it in Erlang 20.1/RabbitMQ 3.6.15 and Erlang 20.1/RabbitMQ 3.7.4)
Is there anyway to track down which queue index the issue relates to, so that we don't need to remove all of them?
Circumstantial evidence points to that this is more prone to happen after a cluster has been rebooted.
Similar to https://groups.google.com/d/msg/rabbitmq-users/_96tCMxPCTk/PX8W7YAYDAAJ
/Johan
2018-04-14 14:19:44.282 [info] <0.336.0> Message store "3SKE8T36OWL2LT39IJII04LEL/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2018-04-14 14:19:44.283 [warning] <0.336.0> Message store "3SKE8T36OWL2LT39IJII04LEL/msg_store_persistent": rebuilding indices from scratch
2018-04-14 14:19:46.408 [error] <0.274.0> ** Generic server <0.274.0> terminating
** Last message in was {'$gen_cast',{submit_async,#Fun<rabbit_queue_index.36.122888644>}}
** When Server state == undefined
** Reason for termination ==
** {function_clause,[{ ,journal_minus_segment1,[{no_pub,no_del,ack},{{true,<<183,76,252,216,188,134,145,120,4,107,92,115,245,235,18,91,0,5,105,153,138,25,118,17,0,0,5,145>>,<<>>},no_del,n
o_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.
erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,915}]},{lists,map,2,[{file,"lists
.erl"},{line,1239}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1039}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,906}]}]}
2018-04-14 14:19:46.422 [error] <0.274.0> CRASH REPORT Process <0.274.0> with 1 neighbours exited with reason: no function clause matching rabbit_queue_index:journal_minus_segment1({no_pub,no_del,ack}, {{
true,<<183,76,252,216,188,134,145,120,4,107,92,115,245,235,18,91,0,5,105,153,138,25,118,17,0,...>>,...},...}) line 1231 in gen_server2:terminate/3 line 1161
2018-04-14 14:19:46.423 [error] <0.336.0> CRASH REPORT Process <0.336.0> with 1 neighbours exited with reason: {{function_clause,[{rabbit_queue_index,journal_minus_segment1,[{no_pub,no_del,ack},{{true,<<1
83,76,252,216,188,134,145,120,4,107,92,115,245,235,18,91,0,5,105,153,138,25,118,17,0,0,5,145>>,<<>>},no_del,no_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_
segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_ind
ex,...},...]},...} in gen_server2:init_it/6 line 589
2018-04-14 14:19:46.424 [error] <0.272.0> Supervisor worker_pool_sup had child 1 started with worker_pool_worker:start_link(worker_pool) at <0.274.0> exit with reason no function clause matching rabbit_qu
eue_index:journal_minus_segment1({no_pub,no_del,ack}, {{true,<<183,76,252,216,188,134,145,120,4,107,92,115,245,235,18,91,0,5,105,153,138,25,118,17,0,...>>,...},...}) line 1231 in context child_terminated
2018-04-14 14:19:46.424 [error] <0.341.0> ** Generic server <0.341.0> terminating
** Last message in was {'$gen_cast',{submit_async,#Fun<rabbit_queue_index.36.122888644>}}
** When Server state == undefined
** Reason for termination ==
** {noproc,[{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,"src/rabbit_queue_index.erl"},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,"src/worker_pool_wo
rker.erl"},{line,122}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
2018-04-14 14:19:46.424 [error] <0.341.0> CRASH REPORT Process <0.341.0> with 0 neighbours exited with reason: no such process or port in call to [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,101,114,108]},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,112,111,111,108,95,119,111,114,107,101,114,46,101,114,108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in gen_server2:terminate/3 line 1161
2018-04-14 14:19:46.425 [error] <0.272.0> Supervisor worker_pool_sup had child 1 started with worker_pool_worker:start_link(worker_pool) at <0.341.0> exit with reason no such process or port in call to [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,101,114,108]},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,112,111,111,108,95,119,111,114,107,101,114,46,101,114,108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in context child_terminated
2018-04-14 14:19:46.425 [error] <0.342.0> ** Generic server <0.342.0> terminating
** Last message in was {'$gen_cast',{submit_async,#Fun<rabbit_queue_index.36.122888644>}}
** When Server state == undefined
** Reason for termination ==
** {noproc,[{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,"src/rabbit_queue_index.erl"},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,"src/worker_pool_worker.erl"},{line,122}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
2018-04-14 14:19:46.425 [error] <0.342.0> CRASH REPORT Process <0.342.0> with 0 neighbours exited with reason: no such process or port in call to [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,101,114,108]},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,112,111,111,108,95,119,111,114,107,101,114,46,101,114,108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in gen_server2:terminate/3 line 1161
2018-04-14 14:19:46.426 [error] <0.272.0> Supervisor worker_pool_sup had child 1 started with worker_pool_worker:start_link(worker_pool) at <0.342.0> exit with reason no such process or port in call to [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,101,114,108]},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,112,111,111,108,95,119,111,114,107,101,114,46,101,114,108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in context child_terminated
2018-04-14 14:19:46.427 [error] <0.343.0> ** Generic server <0.343.0> terminating
** Last message in was {'$gen_cast',{submit_async,#Fun<rabbit_queue_index.36.122888644>}}
** When Server state == undefined
** Reason for termination ==
** {noproc,[{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,"src/rabbit_queue_index.erl"},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,"src/worker_pool_worker.erl"},{line,122}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
2018-04-14 14:19:46.427 [error] <0.343.0> CRASH REPORT Process <0.343.0> with 0 neighbours exited with reason: no such process or port in call to [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,101,114,108]},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,112,111,111,108,95,119,111,114,107,101,114,46,101,114,108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in gen_server2:terminate/3 line 1161
2018-04-14 14:19:46.428 [error] <0.272.0> Supervisor worker_pool_sup had child 1 started with worker_pool_worker:start_link(worker_pool) at <0.343.0> exit with reason no such process or port in call to [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,101,114,108]},{line,693}]},{worker_pool_worker,handle_cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,112,111,111,108,95,119,111,114,107,101,114,46,101,114,108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in context child_terminated
2018-04-14 14:19:46.430 [error] <0.344.0> ** Generic server <0.344.0> terminating
Michael Klishin
Apr 14, 2018, 5:07:27 PM4/14/18
to rabbitm...@googlegroups.com
The [generic] worker pool might be exhausted: it returns a `noproc`:
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send an email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ
jo...@cloudamqp.com
Apr 16, 2018, 3:49:30 PM4/16/18
to rabbitm...@googlegroups.com
Would a copy of the queue index folder (or full data directory) be helpful?
>> 108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[
>> 115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,
>> 115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,
>> 115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,
>> 115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,
>> 115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,
>> 115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,
>> To post to this group, send an email to rabbitm...@googlegroups.com.
> To post to this group, send an email to rabbitm...@googlegroups.com.
> --000000000000bac03a0569d55f17
> Content-Type: text/html; charset="UTF-8"
> Content-Transfer-Encoding: quoted-printable
>
> <div dir=3D"ltr"><div>The [generic] worker pool might be exhausted: it retu=
> rns a `noproc`:</div><div><br></div><div>=C2=A0* <a href=3D"https://github.=
> com/rabbitmq/rabbitmq-server/blob/master/src/rabbit_queue_index.erl#L693">h=
> ttps://github.com/rabbitmq/rabbitmq-server/blob/master/src/rabbit_queue_ind=
> ex.erl#L693</a></div><div>=C2=A0* <a href=3D"https://github.com/rabbitmq/ra=
> bbitmq-common/blob/master/src/worker_pool_worker.erl#L122">https://github.c=
> om/rabbitmq/rabbitmq-common/blob/master/src/worker_pool_worker.erl#L122</a>=
> <br></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">O=
> n Sun, Apr 15, 2018 at 12:36 AM, <span dir=3D"ltr"><<a href=3D"mailto:j=
> oh...@cloudamqp.com" target=3D"_blank">jo...@cloudamqp.com</a>></span> wr=
> ote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border=
> -left:1px #ccc solid;padding-left:1ex">Hi,<br>
> <br>
> We've recently seen an increase in queue index issues. Below an excerpt=
> of a log file when RabbitMQ is trying to start up.<br>
> This example is from RabbitMQ 3.7.3 / Erlang 20.1 (but have also seen it in=
> Erlang 20.1/RabbitMQ 3.6.15 and Erlang 20.1/RabbitMQ 3.7.4)<br>
> <br>
> Is there anyway to track down which queue index the issue relates to, so th=
> at we don't need to remove all of them?<br>
> Circumstantial evidence points to that this is more prone to happen after a=
> cluster has been rebooted.<br>
> <br>
> Similar to <a href=3D"https://groups.google.com/d/msg/rabbitmq-users/_96tCM=
> xPCTk/PX8W7YAYDAAJ" rel=3D"noreferrer" target=3D"_blank">https://groups.goo=
> gle.com/d/<wbr>msg/rabbitmq-users/_<wbr>96tCMxPCTk/PX8W7YAYDAAJ</a><br>
> <br>
> /Johan<br>
> <br>
> 2018-04-14 14:19:44.282 [info] <0.336.0> Message store "3SKE8T36=
> OWL2LT39IJII04LEL/<wbr>msg_store_persistent": using rabbit_msg_store_e=
> ts_index to provide index<br>
> 2018-04-14 14:19:44.283 [warning] <0.336.0> Message store "3SKE8=
> T36OWL2LT39IJII04LEL/<wbr>msg_store_persistent": rebuilding indices fr=
> om scratch<br>
> 2018-04-14 14:19:46.408 [error] <0.274.0> ** Generic server <0.274=
> .0> terminating<br>
> ** Last message in was {'$gen_cast',{submit_async,#<wbr>Fun<rabb=
> it_queue_index.36.<wbr>122888644>}}<br>
> ** When Server state =3D=3D undefined<br>
> ** Reason for termination =3D=3D<br>
> ** {function_clause,[{=C2=A0 ,journal_minus_segment1,[{no_<wbr>pub,no_del,a=
> ck},{{true,<<183,<wbr>76,252,216,188,134,145,120,4,<wbr>107,92,115,24=
> 5,235,18,91,0,5,<wbr>105,153,138,25,118,17,0,0,5,<wbr>145>>,<<&=
> gt;>},no_del,n<br>
> o_ack}],[{file,"src/rabbit_<wbr>queue_index.erl"},{line,1231}]<wb=
> r>},{rabbit_queue_index,'-<wbr>journal_minus_segment/3-fun-0-<wbr>'=
> ,4,[{file,"src/rabbit_queue_<wbr>index.erl"},{line,1208}]},{<wbr>=
> array,sparse_foldl_3,7,[{file,<wbr>"array.<br>
> erl"},{line,1684}]},{array,<wbr>sparse_foldl_2,9,[{file,"<wbr>arr=
> ay.erl"},{line,1678}]},{<wbr>rabbit_queue_index,'-recover_<wbr>jou=
> rnal/1-fun-0-',1,[{file,"<wbr>src/rabbit_queue_index.erl"},{<=
> wbr>line,915}]},{lists,map,2,[{<wbr>file,"lists<br>
> .erl"},{line,1239}]},{rabbit_<wbr>queue_index,segment_map,2,[{<wbr>fil=
> e,"src/rabbit_queue_index.<wbr>erl"},{line,1039}]},{rabbit_<wbr>q=
> ueue_index,recover_journal,1,<wbr>[{file,"src/rabbit_queue_<wbr>index.=
> erl"},{line,906}]}]}<br>
> 2018-04-14 14:19:46.422 [error] <0.274.0> CRASH REPORT Process <0.=
> 274.0> with 1 neighbours exited with reason: no function clause matching=
> rabbit_queue_index:journal_<wbr>minus_segment1({no_pub,no_del,<wbr>ack}, {=
> {<br>
> true,<<183,76,252,216,188,134,<wbr>145,120,4,107,92,115,245,235,<wbr>=
> 18,91,0,5,105,153,138,25,118,<wbr>17,0,...>>,...},...}) line 1231 in =
> gen_server2:terminate/3 line 1161<br>
> 2018-04-14 14:19:46.423 [error] <0.336.0> CRASH REPORT Process <0.=
> 336.0> with 1 neighbours exited with reason: {{function_clause,[{rabbit_=
> <wbr>queue_index,journal_minus_<wbr>segment1,[{no_pub,no_del,ack},<wbr>{{tr=
> ue,<<1<br>
> 83,76,252,216,188,134,145,120,<wbr>4,107,92,115,245,235,18,91,0,<wbr>5,105,=
> 153,138,25,118,17,0,0,5,<wbr>145>>,<<>>},no_del,no_ack}],=
> [{<wbr>file,"src/rabbit_queue_index.<wbr>erl"},{line,1231}]},{rab=
> bit_<wbr>queue_index,'-journal_minus_<br>
> segment/3-fun-0-',4,[{file,"<wbr>src/rabbit_queue_index.erl"}=
> ,{<wbr>line,1208}]},{array,sparse_<wbr>foldl_3,7,[{file,"array.erl&quo=
> t;},<wbr>{line,1684}]},{array,sparse_<wbr>foldl_2,9,[{file,"array.erl&=
> quot;},<wbr>{line,1678}]},{rabbit_queue_<wbr>ind<br>
> ex,...},...]},...} in gen_server2:init_it/6 line 589<br>
> 2018-04-14 14:19:46.424 [error] <0.272.0> Supervisor worker_pool_sup =
> had child 1 started with worker_pool_worker:start_link(<wbr>worker_pool) at=
> <0.274.0> exit with reason no function clause matching rabbit_qu<br>
> eue_index:journal_minus_<wbr>segment1({no_pub,no_del,ack}, {{true,<<1=
> 83,76,252,216,188,<wbr>134,145,120,4,107,92,115,245,<wbr>235,18,91,0,5,105,=
> 153,138,25,<wbr>118,17,0,...>>,...},...}) line 1231 in context child_=
> terminated<br>
> 2018-04-14 14:19:46.424 [error] <0.341.0> ** Generic server <0.341=
> .0> terminating<br>
> ** Last message in was {'$gen_cast',{submit_async,#<wbr>Fun<rabb=
> it_queue_index.36.<wbr>122888644>}}<br>
> ** When Server state =3D=3D undefined<br>
> ** Reason for termination =3D=3D<br>
> ** {noproc,[{erlang,link,[<0.340.<wbr>0>],[]},{rabbit_queue_index,=
> 9;-<wbr>queue_index_walker/1-fun-0-',<wbr>2,[{file,"src/rabbit_que=
> ue_<wbr>index.erl"},{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>=
> cast,2,[{file,"src/worker_<wbr>pool_wo<br>
> rker.erl"},{line,122}]},{gen_<wbr>server2,handle_msg,2,[{file,"<w=
> br>src/gen_server2.erl"},{line,<wbr>1047}]},{proc_lib,init_p_do_<wbr>a=
> pply,3,[{file,"proc_lib.erl"}<wbr>,{line,247}]}]}<br>
> 2018-04-14 14:19:46.424 [error] <0.341.0> CRASH REPORT Process <0.=
> 341.0> with 0 neighbours exited with reason: no such process or port in =
> call to [{erlang,link,[<0.340.0>],[]},<wbr>{rabbit_queue_index,'-=
> queue_<wbr>index_walker/1-fun-0-',2,[{<wbr>file,[115,114,99,47,114,97,9=
> 8,<wbr>98,105,116,95,113,117,101,117,<wbr>101,95,105,110,100,101,120,46,<wb=
> r>101,114,108]},{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>cast,2,[{=
> file,[115,114,99,47,<wbr>119,111,114,107,101,114,95,<wbr>112,111,111,108,95=
> ,119,111,<wbr>114,107,101,114,46,101,114,<wbr>108]},{line,122}]},{gen_<wbr>=
> server2,handle_msg,2,[{file,[<wbr>115,114,99,47,103,101,110,95,<wbr>115,101=
> ,114,118,101,114,50,46,<wbr>101,114,108]},{line,1047}]},{<wbr>proc_lib,init=
> _p_do_apply,3,[{<wbr>file,[112,114,111,99,95,108,<wbr>105,98,46,101,114,108=
> ]},{line,<wbr>247}]}] in gen_server2:terminate/3 line 1161<br>
> 2018-04-14 14:19:46.425 [error] <0.272.0> Supervisor worker_pool_sup =
> had child 1 started with worker_pool_worker:start_link(<wbr>worker_pool) at=
> <0.341.0> exit with reason no such process or port in call to [{erla=
> ng,link,[<0.340.0>],[]},<wbr>{rabbit_queue_index,'-queue_<wbr>ind=
> ex_walker/1-fun-0-',2,[{<wbr>file,[115,114,99,47,114,97,98,<wbr>98,105,=
> 116,95,113,117,101,117,<wbr>101,95,105,110,100,101,120,46,<wbr>101,114,108]=
> },{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>cast,2,[{file,[115,114,=
> 99,47,<wbr>119,111,114,107,101,114,95,<wbr>112,111,111,108,95,119,111,<wbr>=
> 114,107,101,114,46,101,114,<wbr>108]},{line,122}]},{gen_<wbr>server2,handle=
> _msg,2,[{file,[<wbr>115,114,99,47,103,101,110,95,<wbr>115,101,114,118,101,1=
> 14,50,46,<wbr>101,114,108]},{line,1047}]},{<wbr>proc_lib,init_p_do_apply,3,=
> [{<wbr>file,[112,114,111,99,95,108,<wbr>105,98,46,101,114,108]},{line,<wbr>=
> 247}]}] in context child_terminated<br>
> 2018-04-14 14:19:46.425 [error] <0.342.0> ** Generic server <0.342=
> .0> terminating<br>
> ** Last message in was {'$gen_cast',{submit_async,#<wbr>Fun<rabb=
> it_queue_index.36.<wbr>122888644>}}<br>
> ** When Server state =3D=3D undefined<br>
> ** Reason for termination =3D=3D<br>
> ** {noproc,[{erlang,link,[<0.340.<wbr>0>],[]},{rabbit_queue_index,=
> 9;-<wbr>queue_index_walker/1-fun-0-',<wbr>2,[{file,"src/rabbit_que=
> ue_<wbr>index.erl"},{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>=
> cast,2,[{file,"src/worker_<wbr>pool_worker.erl"},{line,122}]}<wbr=
>> ,{gen_server2,handle_msg,2,[{<wbr>file,"src/gen_server2.erl"},{<=
> wbr>line,1047}]},{proc_lib,init_p_<wbr>do_apply,3,[{file,"proc_lib.<wb=
> r>erl"},{line,247}]}]}<br>
> 2018-04-14 14:19:46.425 [error] <0.342.0> CRASH REPORT Process <0.=
> 342.0> with 0 neighbours exited with reason: no such process or port in =
> call to [{erlang,link,[<0.340.0>],[]},<wbr>{rabbit_queue_index,'-=
> queue_<wbr>index_walker/1-fun-0-',2,[{<wbr>file,[115,114,99,47,114,97,9=
> 8,<wbr>98,105,116,95,113,117,101,117,<wbr>101,95,105,110,100,101,120,46,<wb=
> r>101,114,108]},{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>cast,2,[{=
> file,[115,114,99,47,<wbr>119,111,114,107,101,114,95,<wbr>112,111,111,108,95=
> ,119,111,<wbr>114,107,101,114,46,101,114,<wbr>108]},{line,122}]},{gen_<wbr>=
> server2,handle_msg,2,[{file,[<wbr>115,114,99,47,103,101,110,95,<wbr>115,101=
> ,114,118,101,114,50,46,<wbr>101,114,108]},{line,1047}]},{<wbr>proc_lib,init=
> _p_do_apply,3,[{<wbr>file,[112,114,111,99,95,108,<wbr>105,98,46,101,114,108=
> ]},{line,<wbr>247}]}] in gen_server2:terminate/3 line 1161<br>
> 2018-04-14 14:19:46.426 [error] <0.272.0> Supervisor worker_pool_sup =
> had child 1 started with worker_pool_worker:start_link(<wbr>worker_pool) at=
> <0.342.0> exit with reason no such process or port in call to [{erla=
> ng,link,[<0.340.0>],[]},<wbr>{rabbit_queue_index,'-queue_<wbr>ind=
> ex_walker/1-fun-0-',2,[{<wbr>file,[115,114,99,47,114,97,98,<wbr>98,105,=
> 116,95,113,117,101,117,<wbr>101,95,105,110,100,101,120,46,<wbr>101,114,108]=
> },{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>cast,2,[{file,[115,114,=
> 99,47,<wbr>119,111,114,107,101,114,95,<wbr>112,111,111,108,95,119,111,<wbr>=
> 114,107,101,114,46,101,114,<wbr>108]},{line,122}]},{gen_<wbr>server2,handle=
> _msg,2,[{file,[<wbr>115,114,99,47,103,101,110,95,<wbr>115,101,114,118,101,1=
> 14,50,46,<wbr>101,114,108]},{line,1047}]},{<wbr>proc_lib,init_p_do_apply,3,=
> [{<wbr>file,[112,114,111,99,95,108,<wbr>105,98,46,101,114,108]},{line,<wbr>=
> 247}]}] in context child_terminated<br>
> 2018-04-14 14:19:46.427 [error] <0.343.0> ** Generic server <0.343=
> .0> terminating<br>
> ** Last message in was {'$gen_cast',{submit_async,#<wbr>Fun<rabb=
> it_queue_index.36.<wbr>122888644>}}<br>
> ** When Server state =3D=3D undefined<br>
> ** Reason for termination =3D=3D<br>
> ** {noproc,[{erlang,link,[<0.340.<wbr>0>],[]},{rabbit_queue_index,=
> 9;-<wbr>queue_index_walker/1-fun-0-',<wbr>2,[{file,"src/rabbit_que=
> ue_<wbr>index.erl"},{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>=
> cast,2,[{file,"src/worker_<wbr>pool_worker.erl"},{line,122}]}<wbr=
>> ,{gen_server2,handle_msg,2,[{<wbr>file,"src/gen_server2.erl"},{<=
> wbr>line,1047}]},{proc_lib,init_p_<wbr>do_apply,3,[{file,"proc_lib.<wb=
> r>erl"},{line,247}]}]}<br>
> 2018-04-14 14:19:46.427 [error] <0.343.0> CRASH REPORT Process <0.=
> 343.0> with 0 neighbours exited with reason: no such process or port in =
> call to [{erlang,link,[<0.340.0>],[]},<wbr>{rabbit_queue_index,'-=
> queue_<wbr>index_walker/1-fun-0-',2,[{<wbr>file,[115,114,99,47,114,97,9=
> 8,<wbr>98,105,116,95,113,117,101,117,<wbr>101,95,105,110,100,101,120,46,<wb=
> r>101,114,108]},{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>cast,2,[{=
> file,[115,114,99,47,<wbr>119,111,114,107,101,114,95,<wbr>112,111,111,108,95=
> ,119,111,<wbr>114,107,101,114,46,101,114,<wbr>108]},{line,122}]},{gen_<wbr>=
> server2,handle_msg,2,[{file,[<wbr>115,114,99,47,103,101,110,95,<wbr>115,101=
> ,114,118,101,114,50,46,<wbr>101,114,108]},{line,1047}]},{<wbr>proc_lib,init=
> _p_do_apply,3,[{<wbr>file,[112,114,111,99,95,108,<wbr>105,98,46,101,114,108=
> ]},{line,<wbr>247}]}] in gen_server2:terminate/3 line 1161<br>
> 2018-04-14 14:19:46.428 [error] <0.272.0> Supervisor worker_pool_sup =
> had child 1 started with worker_pool_worker:start_link(<wbr>worker_pool) at=
> <0.343.0> exit with reason no such process or port in call to [{erla=
> ng,link,[<0.340.0>],[]},<wbr>{rabbit_queue_index,'-queue_<wbr>ind=
> ex_walker/1-fun-0-',2,[{<wbr>file,[115,114,99,47,114,97,98,<wbr>98,105,=
> 116,95,113,117,101,117,<wbr>101,95,105,110,100,101,120,46,<wbr>101,114,108]=
> },{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>cast,2,[{file,[115,114,=
> 99,47,<wbr>119,111,114,107,101,114,95,<wbr>112,111,111,108,95,119,111,<wbr>=
> 114,107,101,114,46,101,114,<wbr>108]},{line,122}]},{gen_<wbr>server2,handle=
> _msg,2,[{file,[<wbr>115,114,99,47,103,101,110,95,<wbr>115,101,114,118,101,1=
> 14,50,46,<wbr>101,114,108]},{line,1047}]},{<wbr>proc_lib,init_p_do_apply,3,=
> [{<wbr>file,[112,114,111,99,95,108,<wbr>105,98,46,101,114,108]},{line,<wbr>=
> 247}]}] in context child_terminated<br>
> 2018-04-14 14:19:46.430 [error] <0.344.0> ** Generic server <0.344=
> .0> terminating<br>
> <span class=3D"HOEnZb"><font color=3D"#888888"><br>
> -- <br>
> You received this message because you are subscribed to the Google Groups &=
> quot;rabbitmq-users" group.<br>
> To unsubscribe from this group and stop receiving emails from it, send an e=
> mail to <a href=3D"mailto:rabbitmq-users%2Bunsu...@googlegroups.com">ra=
> bbitmq-users+unsubscribe@<wbr>googlegroups.com</a>.<br>
> To post to this group, send an email to <a href=3D"mailto:rabbitmq-users@go=
> oglegroups.com">rabbitmq-users@googlegroups.<wbr>com</a>.<br>
> For more options, visit <a href=3D"https://groups.google.com/d/optout" rel=
> =3D"noreferrer" target=3D"_blank">https://groups.google.com/d/<wbr>optout</=
> a>.<br>
> </font></span></blockquote></div><br><br clear=3D"all"><br>-- <br><div clas=
> s=3D"gmail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr">M=
> K<div><br></div><div>Staff Software Engineer, Pivotal/RabbitMQ</div></div><=
> /div>
> </div>
>
> <p></p>
>
> -- <br />
> You received this message because you are subscribed to the Google Groups &=
> quot;rabbitmq-users" group.<br />
> To unsubscribe from this group and stop receiving emails from it, send an e=
> mail to <a href=3D"mailto:rabbitmq-user...@googlegroups.com">rabb=
> itmq-users+...@googlegroups.com</a>.<br />
> To post to this group, send email to <a href=3D"mailto:rabbitmq-users@googl=
> egroups.com">rabbitm...@googlegroups.com</a>.<br />
> For more options, visit <a href=3D"https://groups.google.com/d/optout">http=
> s://groups.google.com/d/optout</a>.<br />
>
> --000000000000bac03a0569d55f17--
>> 115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,
>> 101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{
>> file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in
>> gen_server2:terminate/3 line 1161
>> 2018-04-14 14:19:46.425 [error] <0.272.0> Supervisor worker_pool_sup had
>> child 1 started with worker_pool_worker:start_link(worker_pool) at
>> <0.341.0> exit with reason no such process or port in call to
>> [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_
>> index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,
>> 98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,
>> 101,114,108]},{line,693}]},{worker_pool_worker,handle_
>> cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,
>> 112,111,111,108,95,119,111,114,107,101,114,46,101,114,
>> 108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[
>> file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in
>> gen_server2:terminate/3 line 1161
>> 2018-04-14 14:19:46.425 [error] <0.272.0> Supervisor worker_pool_sup had
>> child 1 started with worker_pool_worker:start_link(worker_pool) at
>> <0.341.0> exit with reason no such process or port in call to
>> [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_
>> index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,
>> 98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,
>> 101,114,108]},{line,693}]},{worker_pool_worker,handle_
>> cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,
>> 112,111,111,108,95,119,111,114,107,101,114,46,101,114,
>> 115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,
>> 101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{
>> file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in
>> context child_terminated
>> 2018-04-14 14:19:46.425 [error] <0.342.0> ** Generic server <0.342.0>
>> terminating
>> ** Last message in was {'$gen_cast',{submit_async,#
>> file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in
>> context child_terminated
>> 2018-04-14 14:19:46.425 [error] <0.342.0> ** Generic server <0.342.0>
>> terminating
>> ** Last message in was {'$gen_cast',{submit_async,#
>> Fun<rabbit_queue_index.36.122888644>}}
>> ** When Server state == undefined
>> ** Reason for termination ==
>> ** {noproc,[{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-
>> queue_index_walker/1-fun-0-',2,[{file,"src/rabbit_queue_
>> index.erl"},{line,693}]},{worker_pool_worker,handle_
>> cast,2,[{file,"src/worker_pool_worker.erl"},{line,122}]}
>> ** When Server state == undefined
>> ** Reason for termination ==
>> ** {noproc,[{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-
>> queue_index_walker/1-fun-0-',2,[{file,"src/rabbit_queue_
>> index.erl"},{line,693}]},{worker_pool_worker,handle_
>> ,{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{
>> line,1047}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.
>> line,1047}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.
>> erl"},{line,247}]}]}
>> 2018-04-14 14:19:46.425 [error] <0.342.0> CRASH REPORT Process <0.342.0>
>> with 0 neighbours exited with reason: no such process or port in call to
>> [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_
>> index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,
>> 98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,
>> 101,114,108]},{line,693}]},{worker_pool_worker,handle_
>> cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,
>> 112,111,111,108,95,119,111,114,107,101,114,46,101,114,
>> 108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[
>> 2018-04-14 14:19:46.425 [error] <0.342.0> CRASH REPORT Process <0.342.0>
>> with 0 neighbours exited with reason: no such process or port in call to
>> [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_
>> index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,
>> 98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,
>> 101,114,108]},{line,693}]},{worker_pool_worker,handle_
>> cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,
>> 112,111,111,108,95,119,111,114,107,101,114,46,101,114,
>> 115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,
>> 101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{
>> file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in
>> gen_server2:terminate/3 line 1161
>> 2018-04-14 14:19:46.426 [error] <0.272.0> Supervisor worker_pool_sup had
>> child 1 started with worker_pool_worker:start_link(worker_pool) at
>> <0.342.0> exit with reason no such process or port in call to
>> [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_
>> index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,
>> 98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,
>> 101,114,108]},{line,693}]},{worker_pool_worker,handle_
>> cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,
>> 112,111,111,108,95,119,111,114,107,101,114,46,101,114,
>> 108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[
>> file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in
>> gen_server2:terminate/3 line 1161
>> 2018-04-14 14:19:46.426 [error] <0.272.0> Supervisor worker_pool_sup had
>> child 1 started with worker_pool_worker:start_link(worker_pool) at
>> <0.342.0> exit with reason no such process or port in call to
>> [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_
>> index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,
>> 98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,
>> 101,114,108]},{line,693}]},{worker_pool_worker,handle_
>> cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,
>> 112,111,111,108,95,119,111,114,107,101,114,46,101,114,
>> 115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,
>> 101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{
>> file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in
>> context child_terminated
>> 2018-04-14 14:19:46.427 [error] <0.343.0> ** Generic server <0.343.0>
>> terminating
>> ** Last message in was {'$gen_cast',{submit_async,#
>> file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in
>> context child_terminated
>> 2018-04-14 14:19:46.427 [error] <0.343.0> ** Generic server <0.343.0>
>> terminating
>> ** Last message in was {'$gen_cast',{submit_async,#
>> Fun<rabbit_queue_index.36.122888644>}}
>> ** When Server state == undefined
>> ** Reason for termination ==
>> ** {noproc,[{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-
>> queue_index_walker/1-fun-0-',2,[{file,"src/rabbit_queue_
>> index.erl"},{line,693}]},{worker_pool_worker,handle_
>> cast,2,[{file,"src/worker_pool_worker.erl"},{line,122}]}
>> ** When Server state == undefined
>> ** Reason for termination ==
>> ** {noproc,[{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-
>> queue_index_walker/1-fun-0-',2,[{file,"src/rabbit_queue_
>> index.erl"},{line,693}]},{worker_pool_worker,handle_
>> ,{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{
>> line,1047}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.
>> line,1047}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.
>> erl"},{line,247}]}]}
>> 2018-04-14 14:19:46.427 [error] <0.343.0> CRASH REPORT Process <0.343.0>
>> with 0 neighbours exited with reason: no such process or port in call to
>> [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_
>> index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,
>> 98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,
>> 101,114,108]},{line,693}]},{worker_pool_worker,handle_
>> cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,
>> 112,111,111,108,95,119,111,114,107,101,114,46,101,114,
>> 108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[
>> 2018-04-14 14:19:46.427 [error] <0.343.0> CRASH REPORT Process <0.343.0>
>> with 0 neighbours exited with reason: no such process or port in call to
>> [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_
>> index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,
>> 98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,
>> 101,114,108]},{line,693}]},{worker_pool_worker,handle_
>> cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,
>> 112,111,111,108,95,119,111,114,107,101,114,46,101,114,
>> 115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,
>> 101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{
>> file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in
>> gen_server2:terminate/3 line 1161
>> 2018-04-14 14:19:46.428 [error] <0.272.0> Supervisor worker_pool_sup had
>> child 1 started with worker_pool_worker:start_link(worker_pool) at
>> <0.343.0> exit with reason no such process or port in call to
>> [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_
>> index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,
>> 98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,
>> 101,114,108]},{line,693}]},{worker_pool_worker,handle_
>> cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,
>> 112,111,111,108,95,119,111,114,107,101,114,46,101,114,
>> 108]},{line,122}]},{gen_server2,handle_msg,2,[{file,[
>> file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in
>> gen_server2:terminate/3 line 1161
>> 2018-04-14 14:19:46.428 [error] <0.272.0> Supervisor worker_pool_sup had
>> child 1 started with worker_pool_worker:start_link(worker_pool) at
>> <0.343.0> exit with reason no such process or port in call to
>> [{erlang,link,[<0.340.0>],[]},{rabbit_queue_index,'-queue_
>> index_walker/1-fun-0-',2,[{file,[115,114,99,47,114,97,98,
>> 98,105,116,95,113,117,101,117,101,95,105,110,100,101,120,46,
>> 101,114,108]},{line,693}]},{worker_pool_worker,handle_
>> cast,2,[{file,[115,114,99,47,119,111,114,107,101,114,95,
>> 112,111,111,108,95,119,111,114,107,101,114,46,101,114,
>> 115,114,99,47,103,101,110,95,115,101,114,118,101,114,50,46,
>> 101,114,108]},{line,1047}]},{proc_lib,init_p_do_apply,3,[{
>> file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in
>> context child_terminated
>> 2018-04-14 14:19:46.430 [error] <0.344.0> ** Generic server <0.344.0>
>> terminating
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "rabbitmq-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to rabbitmq-user...@googlegroups.com.
>> file,[112,114,111,99,95,108,105,98,46,101,114,108]},{line,247}]}] in
>> context child_terminated
>> 2018-04-14 14:19:46.430 [error] <0.344.0> ** Generic server <0.344.0>
>> terminating
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "rabbitmq-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> To post to this group, send an email to rabbitm...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> MK
>
> Staff Software Engineer, Pivotal/RabbitMQ
>
>>
>
>
>
> --
> MK
>
> Staff Software Engineer, Pivotal/RabbitMQ
>
> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To post to this group, send an email to rabbitm...@googlegroups.com.
> --000000000000bac03a0569d55f17
> Content-Type: text/html; charset="UTF-8"
> Content-Transfer-Encoding: quoted-printable
>
> <div dir=3D"ltr"><div>The [generic] worker pool might be exhausted: it retu=
> rns a `noproc`:</div><div><br></div><div>=C2=A0* <a href=3D"https://github.=
> com/rabbitmq/rabbitmq-server/blob/master/src/rabbit_queue_index.erl#L693">h=
> ttps://github.com/rabbitmq/rabbitmq-server/blob/master/src/rabbit_queue_ind=
> ex.erl#L693</a></div><div>=C2=A0* <a href=3D"https://github.com/rabbitmq/ra=
> bbitmq-common/blob/master/src/worker_pool_worker.erl#L122">https://github.c=
> om/rabbitmq/rabbitmq-common/blob/master/src/worker_pool_worker.erl#L122</a>=
> <br></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">O=
> n Sun, Apr 15, 2018 at 12:36 AM, <span dir=3D"ltr"><<a href=3D"mailto:j=
> oh...@cloudamqp.com" target=3D"_blank">jo...@cloudamqp.com</a>></span> wr=
> ote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border=
> -left:1px #ccc solid;padding-left:1ex">Hi,<br>
> <br>
> We've recently seen an increase in queue index issues. Below an excerpt=
> of a log file when RabbitMQ is trying to start up.<br>
> This example is from RabbitMQ 3.7.3 / Erlang 20.1 (but have also seen it in=
> Erlang 20.1/RabbitMQ 3.6.15 and Erlang 20.1/RabbitMQ 3.7.4)<br>
> <br>
> Is there anyway to track down which queue index the issue relates to, so th=
> at we don't need to remove all of them?<br>
> Circumstantial evidence points to that this is more prone to happen after a=
> cluster has been rebooted.<br>
> <br>
> Similar to <a href=3D"https://groups.google.com/d/msg/rabbitmq-users/_96tCM=
> xPCTk/PX8W7YAYDAAJ" rel=3D"noreferrer" target=3D"_blank">https://groups.goo=
> gle.com/d/<wbr>msg/rabbitmq-users/_<wbr>96tCMxPCTk/PX8W7YAYDAAJ</a><br>
> <br>
> /Johan<br>
> <br>
> 2018-04-14 14:19:44.282 [info] <0.336.0> Message store "3SKE8T36=
> OWL2LT39IJII04LEL/<wbr>msg_store_persistent": using rabbit_msg_store_e=
> ts_index to provide index<br>
> 2018-04-14 14:19:44.283 [warning] <0.336.0> Message store "3SKE8=
> T36OWL2LT39IJII04LEL/<wbr>msg_store_persistent": rebuilding indices fr=
> om scratch<br>
> 2018-04-14 14:19:46.408 [error] <0.274.0> ** Generic server <0.274=
> .0> terminating<br>
> ** Last message in was {'$gen_cast',{submit_async,#<wbr>Fun<rabb=
> it_queue_index.36.<wbr>122888644>}}<br>
> ** When Server state =3D=3D undefined<br>
> ** Reason for termination =3D=3D<br>
> ** {function_clause,[{=C2=A0 ,journal_minus_segment1,[{no_<wbr>pub,no_del,a=
> ck},{{true,<<183,<wbr>76,252,216,188,134,145,120,4,<wbr>107,92,115,24=
> 5,235,18,91,0,5,<wbr>105,153,138,25,118,17,0,0,5,<wbr>145>>,<<&=
> gt;>},no_del,n<br>
> o_ack}],[{file,"src/rabbit_<wbr>queue_index.erl"},{line,1231}]<wb=
> r>},{rabbit_queue_index,'-<wbr>journal_minus_segment/3-fun-0-<wbr>'=
> ,4,[{file,"src/rabbit_queue_<wbr>index.erl"},{line,1208}]},{<wbr>=
> array,sparse_foldl_3,7,[{file,<wbr>"array.<br>
> erl"},{line,1684}]},{array,<wbr>sparse_foldl_2,9,[{file,"<wbr>arr=
> ay.erl"},{line,1678}]},{<wbr>rabbit_queue_index,'-recover_<wbr>jou=
> rnal/1-fun-0-',1,[{file,"<wbr>src/rabbit_queue_index.erl"},{<=
> wbr>line,915}]},{lists,map,2,[{<wbr>file,"lists<br>
> .erl"},{line,1239}]},{rabbit_<wbr>queue_index,segment_map,2,[{<wbr>fil=
> e,"src/rabbit_queue_index.<wbr>erl"},{line,1039}]},{rabbit_<wbr>q=
> ueue_index,recover_journal,1,<wbr>[{file,"src/rabbit_queue_<wbr>index.=
> erl"},{line,906}]}]}<br>
> 2018-04-14 14:19:46.422 [error] <0.274.0> CRASH REPORT Process <0.=
> 274.0> with 1 neighbours exited with reason: no function clause matching=
> rabbit_queue_index:journal_<wbr>minus_segment1({no_pub,no_del,<wbr>ack}, {=
> {<br>
> true,<<183,76,252,216,188,134,<wbr>145,120,4,107,92,115,245,235,<wbr>=
> 18,91,0,5,105,153,138,25,118,<wbr>17,0,...>>,...},...}) line 1231 in =
> gen_server2:terminate/3 line 1161<br>
> 2018-04-14 14:19:46.423 [error] <0.336.0> CRASH REPORT Process <0.=
> 336.0> with 1 neighbours exited with reason: {{function_clause,[{rabbit_=
> <wbr>queue_index,journal_minus_<wbr>segment1,[{no_pub,no_del,ack},<wbr>{{tr=
> ue,<<1<br>
> 83,76,252,216,188,134,145,120,<wbr>4,107,92,115,245,235,18,91,0,<wbr>5,105,=
> 153,138,25,118,17,0,0,5,<wbr>145>>,<<>>},no_del,no_ack}],=
> [{<wbr>file,"src/rabbit_queue_index.<wbr>erl"},{line,1231}]},{rab=
> bit_<wbr>queue_index,'-journal_minus_<br>
> segment/3-fun-0-',4,[{file,"<wbr>src/rabbit_queue_index.erl"}=
> ,{<wbr>line,1208}]},{array,sparse_<wbr>foldl_3,7,[{file,"array.erl&quo=
> t;},<wbr>{line,1684}]},{array,sparse_<wbr>foldl_2,9,[{file,"array.erl&=
> quot;},<wbr>{line,1678}]},{rabbit_queue_<wbr>ind<br>
> ex,...},...]},...} in gen_server2:init_it/6 line 589<br>
> 2018-04-14 14:19:46.424 [error] <0.272.0> Supervisor worker_pool_sup =
> had child 1 started with worker_pool_worker:start_link(<wbr>worker_pool) at=
> <0.274.0> exit with reason no function clause matching rabbit_qu<br>
> eue_index:journal_minus_<wbr>segment1({no_pub,no_del,ack}, {{true,<<1=
> 83,76,252,216,188,<wbr>134,145,120,4,107,92,115,245,<wbr>235,18,91,0,5,105,=
> 153,138,25,<wbr>118,17,0,...>>,...},...}) line 1231 in context child_=
> terminated<br>
> 2018-04-14 14:19:46.424 [error] <0.341.0> ** Generic server <0.341=
> .0> terminating<br>
> ** Last message in was {'$gen_cast',{submit_async,#<wbr>Fun<rabb=
> it_queue_index.36.<wbr>122888644>}}<br>
> ** When Server state =3D=3D undefined<br>
> ** Reason for termination =3D=3D<br>
> ** {noproc,[{erlang,link,[<0.340.<wbr>0>],[]},{rabbit_queue_index,=
> 9;-<wbr>queue_index_walker/1-fun-0-',<wbr>2,[{file,"src/rabbit_que=
> ue_<wbr>index.erl"},{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>=
> cast,2,[{file,"src/worker_<wbr>pool_wo<br>
> rker.erl"},{line,122}]},{gen_<wbr>server2,handle_msg,2,[{file,"<w=
> br>src/gen_server2.erl"},{line,<wbr>1047}]},{proc_lib,init_p_do_<wbr>a=
> pply,3,[{file,"proc_lib.erl"}<wbr>,{line,247}]}]}<br>
> 2018-04-14 14:19:46.424 [error] <0.341.0> CRASH REPORT Process <0.=
> 341.0> with 0 neighbours exited with reason: no such process or port in =
> call to [{erlang,link,[<0.340.0>],[]},<wbr>{rabbit_queue_index,'-=
> queue_<wbr>index_walker/1-fun-0-',2,[{<wbr>file,[115,114,99,47,114,97,9=
> 8,<wbr>98,105,116,95,113,117,101,117,<wbr>101,95,105,110,100,101,120,46,<wb=
> r>101,114,108]},{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>cast,2,[{=
> file,[115,114,99,47,<wbr>119,111,114,107,101,114,95,<wbr>112,111,111,108,95=
> ,119,111,<wbr>114,107,101,114,46,101,114,<wbr>108]},{line,122}]},{gen_<wbr>=
> server2,handle_msg,2,[{file,[<wbr>115,114,99,47,103,101,110,95,<wbr>115,101=
> ,114,118,101,114,50,46,<wbr>101,114,108]},{line,1047}]},{<wbr>proc_lib,init=
> _p_do_apply,3,[{<wbr>file,[112,114,111,99,95,108,<wbr>105,98,46,101,114,108=
> ]},{line,<wbr>247}]}] in gen_server2:terminate/3 line 1161<br>
> 2018-04-14 14:19:46.425 [error] <0.272.0> Supervisor worker_pool_sup =
> had child 1 started with worker_pool_worker:start_link(<wbr>worker_pool) at=
> <0.341.0> exit with reason no such process or port in call to [{erla=
> ng,link,[<0.340.0>],[]},<wbr>{rabbit_queue_index,'-queue_<wbr>ind=
> ex_walker/1-fun-0-',2,[{<wbr>file,[115,114,99,47,114,97,98,<wbr>98,105,=
> 116,95,113,117,101,117,<wbr>101,95,105,110,100,101,120,46,<wbr>101,114,108]=
> },{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>cast,2,[{file,[115,114,=
> 99,47,<wbr>119,111,114,107,101,114,95,<wbr>112,111,111,108,95,119,111,<wbr>=
> 114,107,101,114,46,101,114,<wbr>108]},{line,122}]},{gen_<wbr>server2,handle=
> _msg,2,[{file,[<wbr>115,114,99,47,103,101,110,95,<wbr>115,101,114,118,101,1=
> 14,50,46,<wbr>101,114,108]},{line,1047}]},{<wbr>proc_lib,init_p_do_apply,3,=
> [{<wbr>file,[112,114,111,99,95,108,<wbr>105,98,46,101,114,108]},{line,<wbr>=
> 247}]}] in context child_terminated<br>
> 2018-04-14 14:19:46.425 [error] <0.342.0> ** Generic server <0.342=
> .0> terminating<br>
> ** Last message in was {'$gen_cast',{submit_async,#<wbr>Fun<rabb=
> it_queue_index.36.<wbr>122888644>}}<br>
> ** When Server state =3D=3D undefined<br>
> ** Reason for termination =3D=3D<br>
> ** {noproc,[{erlang,link,[<0.340.<wbr>0>],[]},{rabbit_queue_index,=
> 9;-<wbr>queue_index_walker/1-fun-0-',<wbr>2,[{file,"src/rabbit_que=
> ue_<wbr>index.erl"},{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>=
> cast,2,[{file,"src/worker_<wbr>pool_worker.erl"},{line,122}]}<wbr=
>> ,{gen_server2,handle_msg,2,[{<wbr>file,"src/gen_server2.erl"},{<=
> wbr>line,1047}]},{proc_lib,init_p_<wbr>do_apply,3,[{file,"proc_lib.<wb=
> r>erl"},{line,247}]}]}<br>
> 2018-04-14 14:19:46.425 [error] <0.342.0> CRASH REPORT Process <0.=
> 342.0> with 0 neighbours exited with reason: no such process or port in =
> call to [{erlang,link,[<0.340.0>],[]},<wbr>{rabbit_queue_index,'-=
> queue_<wbr>index_walker/1-fun-0-',2,[{<wbr>file,[115,114,99,47,114,97,9=
> 8,<wbr>98,105,116,95,113,117,101,117,<wbr>101,95,105,110,100,101,120,46,<wb=
> r>101,114,108]},{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>cast,2,[{=
> file,[115,114,99,47,<wbr>119,111,114,107,101,114,95,<wbr>112,111,111,108,95=
> ,119,111,<wbr>114,107,101,114,46,101,114,<wbr>108]},{line,122}]},{gen_<wbr>=
> server2,handle_msg,2,[{file,[<wbr>115,114,99,47,103,101,110,95,<wbr>115,101=
> ,114,118,101,114,50,46,<wbr>101,114,108]},{line,1047}]},{<wbr>proc_lib,init=
> _p_do_apply,3,[{<wbr>file,[112,114,111,99,95,108,<wbr>105,98,46,101,114,108=
> ]},{line,<wbr>247}]}] in gen_server2:terminate/3 line 1161<br>
> 2018-04-14 14:19:46.426 [error] <0.272.0> Supervisor worker_pool_sup =
> had child 1 started with worker_pool_worker:start_link(<wbr>worker_pool) at=
> <0.342.0> exit with reason no such process or port in call to [{erla=
> ng,link,[<0.340.0>],[]},<wbr>{rabbit_queue_index,'-queue_<wbr>ind=
> ex_walker/1-fun-0-',2,[{<wbr>file,[115,114,99,47,114,97,98,<wbr>98,105,=
> 116,95,113,117,101,117,<wbr>101,95,105,110,100,101,120,46,<wbr>101,114,108]=
> },{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>cast,2,[{file,[115,114,=
> 99,47,<wbr>119,111,114,107,101,114,95,<wbr>112,111,111,108,95,119,111,<wbr>=
> 114,107,101,114,46,101,114,<wbr>108]},{line,122}]},{gen_<wbr>server2,handle=
> _msg,2,[{file,[<wbr>115,114,99,47,103,101,110,95,<wbr>115,101,114,118,101,1=
> 14,50,46,<wbr>101,114,108]},{line,1047}]},{<wbr>proc_lib,init_p_do_apply,3,=
> [{<wbr>file,[112,114,111,99,95,108,<wbr>105,98,46,101,114,108]},{line,<wbr>=
> 247}]}] in context child_terminated<br>
> 2018-04-14 14:19:46.427 [error] <0.343.0> ** Generic server <0.343=
> .0> terminating<br>
> ** Last message in was {'$gen_cast',{submit_async,#<wbr>Fun<rabb=
> it_queue_index.36.<wbr>122888644>}}<br>
> ** When Server state =3D=3D undefined<br>
> ** Reason for termination =3D=3D<br>
> ** {noproc,[{erlang,link,[<0.340.<wbr>0>],[]},{rabbit_queue_index,=
> 9;-<wbr>queue_index_walker/1-fun-0-',<wbr>2,[{file,"src/rabbit_que=
> ue_<wbr>index.erl"},{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>=
> cast,2,[{file,"src/worker_<wbr>pool_worker.erl"},{line,122}]}<wbr=
>> ,{gen_server2,handle_msg,2,[{<wbr>file,"src/gen_server2.erl"},{<=
> wbr>line,1047}]},{proc_lib,init_p_<wbr>do_apply,3,[{file,"proc_lib.<wb=
> r>erl"},{line,247}]}]}<br>
> 2018-04-14 14:19:46.427 [error] <0.343.0> CRASH REPORT Process <0.=
> 343.0> with 0 neighbours exited with reason: no such process or port in =
> call to [{erlang,link,[<0.340.0>],[]},<wbr>{rabbit_queue_index,'-=
> queue_<wbr>index_walker/1-fun-0-',2,[{<wbr>file,[115,114,99,47,114,97,9=
> 8,<wbr>98,105,116,95,113,117,101,117,<wbr>101,95,105,110,100,101,120,46,<wb=
> r>101,114,108]},{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>cast,2,[{=
> file,[115,114,99,47,<wbr>119,111,114,107,101,114,95,<wbr>112,111,111,108,95=
> ,119,111,<wbr>114,107,101,114,46,101,114,<wbr>108]},{line,122}]},{gen_<wbr>=
> server2,handle_msg,2,[{file,[<wbr>115,114,99,47,103,101,110,95,<wbr>115,101=
> ,114,118,101,114,50,46,<wbr>101,114,108]},{line,1047}]},{<wbr>proc_lib,init=
> _p_do_apply,3,[{<wbr>file,[112,114,111,99,95,108,<wbr>105,98,46,101,114,108=
> ]},{line,<wbr>247}]}] in gen_server2:terminate/3 line 1161<br>
> 2018-04-14 14:19:46.428 [error] <0.272.0> Supervisor worker_pool_sup =
> had child 1 started with worker_pool_worker:start_link(<wbr>worker_pool) at=
> <0.343.0> exit with reason no such process or port in call to [{erla=
> ng,link,[<0.340.0>],[]},<wbr>{rabbit_queue_index,'-queue_<wbr>ind=
> ex_walker/1-fun-0-',2,[{<wbr>file,[115,114,99,47,114,97,98,<wbr>98,105,=
> 116,95,113,117,101,117,<wbr>101,95,105,110,100,101,120,46,<wbr>101,114,108]=
> },{line,693}]},{<wbr>worker_pool_worker,handle_<wbr>cast,2,[{file,[115,114,=
> 99,47,<wbr>119,111,114,107,101,114,95,<wbr>112,111,111,108,95,119,111,<wbr>=
> 114,107,101,114,46,101,114,<wbr>108]},{line,122}]},{gen_<wbr>server2,handle=
> _msg,2,[{file,[<wbr>115,114,99,47,103,101,110,95,<wbr>115,101,114,118,101,1=
> 14,50,46,<wbr>101,114,108]},{line,1047}]},{<wbr>proc_lib,init_p_do_apply,3,=
> [{<wbr>file,[112,114,111,99,95,108,<wbr>105,98,46,101,114,108]},{line,<wbr>=
> 247}]}] in context child_terminated<br>
> 2018-04-14 14:19:46.430 [error] <0.344.0> ** Generic server <0.344=
> .0> terminating<br>
> <span class=3D"HOEnZb"><font color=3D"#888888"><br>
> -- <br>
> You received this message because you are subscribed to the Google Groups &=
> quot;rabbitmq-users" group.<br>
> To unsubscribe from this group and stop receiving emails from it, send an e=
> mail to <a href=3D"mailto:rabbitmq-users%2Bunsu...@googlegroups.com">ra=
> bbitmq-users+unsubscribe@<wbr>googlegroups.com</a>.<br>
> To post to this group, send an email to <a href=3D"mailto:rabbitmq-users@go=
> oglegroups.com">rabbitmq-users@googlegroups.<wbr>com</a>.<br>
> For more options, visit <a href=3D"https://groups.google.com/d/optout" rel=
> =3D"noreferrer" target=3D"_blank">https://groups.google.com/d/<wbr>optout</=
> a>.<br>
> </font></span></blockquote></div><br><br clear=3D"all"><br>-- <br><div clas=
> s=3D"gmail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr">M=
> K<div><br></div><div>Staff Software Engineer, Pivotal/RabbitMQ</div></div><=
> /div>
> </div>
>
> <p></p>
>
> -- <br />
> You received this message because you are subscribed to the Google Groups &=
> quot;rabbitmq-users" group.<br />
> To unsubscribe from this group and stop receiving emails from it, send an e=
> mail to <a href=3D"mailto:rabbitmq-user...@googlegroups.com">rabb=
> itmq-users+...@googlegroups.com</a>.<br />
> To post to this group, send email to <a href=3D"mailto:rabbitmq-users@googl=
> egroups.com">rabbitm...@googlegroups.com</a>.<br />
> For more options, visit <a href=3D"https://groups.google.com/d/optout">http=
> s://groups.google.com/d/optout</a>.<br />
>
> --000000000000bac03a0569d55f17--
Reply all
Reply to author
Forward
0 new messages