Issues with vHost in Rabbitmq

[Mohammed Reehan]

Hello All,

I have a 3 node cluster. since couple of days i am seeing one of the vhost is not accessible from a Node. ( Screenshot below) and error log on the node just says ampq_error,Internal_error during handshake. 

The log file is not even giving me alot of information. 

Please help me in investigating this. 

2019-08-28 16:24:29.463 [error] <0.12188.2> Error on AMQP connection <0.12188.2> (192.168.60.148:61765 -> 192.168.60.224:5672 - w3wp.exe, vhost: 'none', user: 'app_user', state: opening), channel 0:
 {handshake_error,opening,
                 {amqp_error,internal_error,
                             "access to vhost 'aerdata-main' refused for user 'app_user': vhost 'aerdata-main' is down",
                             'connection.open'}}

--

Mohammed Rehan

[Luke Bakken]

Hi Mohammed,

Could you please let us know what version of RabbitMQ and Erlang you're using, and on what operating system? Are you using any custom configuration?

 

Could you please share your entire log file? If it's large, just share the range of entries corresponding to these vhost errors.

Thanks,

Luke

[Mohammed Reehan]

Hi Luke, 

Thanks for your email, We are using RMQ version 3.7.16 with erlang 22.0.7 version running on Centos Linux machine. There is no custom configuration in place. 

it is a HA setup with 3 nodes. this is just happening on a single Node for two vhost. 

please see the logs below related to vhost. 

19-08-29 13:12:57.650 [error] <0.26074.8> ** Generic server <0.26074.8> terminating
** Last message in was {'$gen_cast',{sync_start,#Ref<6194.3564374521.1505230849.207809>,<6194.18339.373>}}
** When Server state == {state,{amqqueue,{resource,<<"aerdata-dev">>,queue,<<"Stream.AmosImport.Bus:ErrorQueue">>},true,false,none,[],<6194.29206.247>,[<6195.25612.209>],[<6195.25612.209>],['rabbit@i-aer-lin-rmq06'],[{vhost,<<"aerdata-dev">>},{name,<<"ha-all">>},{pattern,<<>>},{'apply-to',<<"all">>},{definition,[{<<"ha-mode">>,<<"all">>},{<<"ha-sync-mode">>,<<"automatic">>},{<<"max-length-bytes">>,1024000000},{<<"overflow">>,<<"reject-publish">>}]},{priority,0}],undefined,[{<6195.25616.209>,<6195.25612.209>},{<6194.29196.247>,<6194.29206.247>}],[],live,0,[],<<"aerdata-dev">>,#{user => <<"app_user">>}},<0.26075.8>,rabbit_priority_queue,{passthrough,rabbit_variable_queue,{vqstate,{0,{[],[]}},{0,{[],[]}},{delta,undefined,0,0,undefined},{0,{[],[]}},{0,{[],[]}},0,{0,nil},{0,nil},{0,nil},{qistate,"/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR/queues/CY6Z4M9HSN44IWO667417X7S5",{#{},[]},undefined,0,32768,#Fun<rabbit_variable_queue.10.38133315>,#Fun<rabbit_variable_queue.11.38133315>,{0,nil},{0,nil},[],[],{resource,<<"aerdata-dev">>,queue,<<"Stream.AmosImport.Bus:ErrorQueue">>}},{{client_msstate,<0.24342.8>,<<33,41,104,141,215,155,113,3,130,246,95,51,118,53,218,2>>,#{},{state,#Ref<0.876500829.2133458945.157322>,"/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR/msg_store_persistent"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR/msg_store_persistent",<0.24345.8>,#Ref<0.876500829.2133458945.157323>,#Ref<0.876500829.2133458945.157312>,#Ref<0.876500829.2133458945.157324>,#Ref<0.876500829.2133458945.157325>,{4000,800}},{client_msstate,<0.24339.8>,<<77,208,7,90,24,189,41,106,236,126,172,91,154,75,118,89>>,#{},{state,#Ref<0.876500829.2133458945.157273>,"/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZ..."},...}},...}},...}
** Reason for termination ==
** {bad_return_value,{bad_flying_ets_update,1,2,{<<141,221,190,68,105,228,161,126,154,179,139,214,249,216,207,216>>,<<33,41,104,141,215,155,113,3,130,246,95,51,118,53,218,2>>}}}
2019-08-29 13:12:57.651 [error] <0.26074.8> CRASH REPORT Process <0.26074.8> with 1 neighbours exited with reason: bad return value: {bad_flying_ets_update,1,2,{<<141,221,190,68,105,228,161,126,154,179,139,214,249,216,207,216>>,<<33,41,104,141,215,155,113,3,130,246,95,51,118,53,218,2>>}} in gen_server2:terminate/3 line 1172
2019-08-29 13:12:57.652 [error] <0.26073.8> Supervisor {<0.26073.8>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"aerdata-dev">>,queue,<<"Stream.AmosImport.Bus:ErrorQueue">>},true,false,none,...}, slave, <0.26072.8>) at <0.26074.8> exit with reason bad return value: {bad_flying_ets_update,1,2,{<<141,221,190,68,105,228,161,126,154,179,139,214,249,216,207,216>>,<<33,41,104,141,215,155,113,3,130,246,95,51,118,53,218,2>>}} in context child_terminated
2019-08-29 13:12:57.657 [info] <0.24342.8> Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR/msg_store_persistent'
2019-08-29 13:12:57.685 [info] <0.24342.8> Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR/msg_store_persistent' is stopped
2019-08-29 13:12:57.689 [error] <0.24342.8> ** Generic server <0.24342.8> terminating
** Last message in was {'$gen_cast',{write,<<33,41,104,141,215,155,113,3,130,246,95,51,118,53,218,2>>,<<141,221,190,68,105,228,161,126,154,179,139,214,249,216,207,216>>,flow}}
** When Server state == {msstate,"/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR/msg_store_persistent",rabbit_msg_store_ets_index,{state,#Ref<0.876500829.2133458945.157322>,"/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR/msg_store_persistent"},13,#Ref<0.876500829.2133327873.157349>,#{},undefined,2427326,2427326,#{},<0.24345.8>,#Ref<0.876500829.2133458945.157323>,#Ref<0.876500829.2133458945.157312>,#Ref<0.876500829.2133458945.157324>,#Ref<0.876500829.2133458945.157325>,#{<<33,41,104,141,215,155,113,3,130,246,95,51,118,53,218,2>> => {dying_client,<<33,41,104,141,215,155,113,3,130,246,95,51,118,53,218,2>>,13,2427326}},#{<<183,159,187,181,66,93,142,218,37,122,56,97,39,76,162,155>> => {<0.26058.8>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},<<7,106,87,234,153,104,221,246,202,103,97,35,172,115,232,221>> => {<0.26107.8>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},<<209,69,137,23,151,96,208,177,98,61,158,73,152,198,149,107>> => {<0.25465.8>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},<<64,195,235,102,79,65,183,83,224,49,134,78,49,25,19,248>> => {<0.25837.8>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},<<21,75,194,89,84,145,136,193,198,81,32,35,33,0,45,3>> => {<0.26214.8>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},<<114,244,44,15,207,141,156,38,83,184,171,140,127,94,120,17>> => {<0.26090.8>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},<<"Qi×vÓ¹[ßWnãúê·=.">> => {<0.26171.8>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},<<210,190,171,203,124,239,199,206,110,249,89,101,5,212,27,16>> => {<0.25832.8>,#Fun<rabbit_variable_queu..>,...},...},...}
** Reason for termination ==
** {bad_return_value,{bad_flying_ets_record,-1,2,{<<141,221,190,68,105,228,161,126,154,179,139,214,249,216,207,216>>,<<33,41,104,141,215,155,113,3,130,246,95,51,118,53,218,2>>}}}
2019-08-29 13:12:57.700 [error] <0.24342.8> CRASH REPORT Process <0.24342.8> with 0 neighbours exited with reason: bad return value: {bad_flying_ets_record,-1,2,{<<141,221,190,68,105,228,161,126,154,179,139,214,249,216,207,216>>,<<33,41,104,141,215,155,113,3,130,246,95,51,118,53,218,2>>}} in gen_server2:terminate/3 line 1172
2019-08-29 13:12:57.703 [error] <0.24334.8> Supervisor {<0.24334.8>,rabbit_vhost_sup} had child msg_store_persistent started with rabbit_msg_store:start_link(msg_store_persistent, "/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR", [], {#Fun<rabbit_queue_index.2.32138423>,{start,[]}}) at <0.24342.8> exit with reason bad return value: {bad_flying_ets_record,-1,2,{<<141,221,190,68,105,228,161,126,154,179,139,214,249,216,207,216>>,<<33,41,104,141,215,155,113,3,130,246,95,51,118,53,218,2>>}} in context child_terminated
2019-08-29 13:12:57.706 [error] <0.24334.8> Supervisor {<0.24334.8>,rabbit_vhost_sup} had child msg_store_persistent started with rabbit_msg_store:start_link(msg_store_persistent, "/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR", [], {#Fun<rabbit_queue_index.2.32138423>,{start,[]}}) at <0.24342.8> exit with reason reached_max_restart_intensity in context shutdown

2019-08-29 13:13:27.692 [error] <0.26073.8> Supervisor {<0.26073.8>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"aerdata-dev">>,queue,<<"Stream.AmosImport.Bus:ErrorQueue">>},true,false,none,...}, slave, <0.26072.8>) at <0.26842.8> exit with reason killed in context shutdown_error

2019-08-29 13:14:02.258 [info] <0.30148.8> Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR/msg_store_persistent'
2019-08-29 13:14:02.281 [info] <0.30148.8> Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR/msg_store_persistent' is stopped
2019-08-29 13:14:02.284 [error] <0.30148.8> ** Generic server <0.30148.8> terminating
** Last message in was {'$gen_cast',{write,<<34,153,151,137,127,133,60,31,45,118,34,198,206,140,157,110>>,<<141,221,190,68,105,228,161,126,154,179,139,214,249,216,207,216>>,flow}}
** When Server state == {msstate,"/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR/msg_store_persistent",rabbit_msg_store_ets_index,{state,#Ref<0.876500829.2133721089.102523>,"/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR/msg_store_persistent"},13,#Ref<0.876500829.2133590017.102566>,#{},undefined,1149592,1149592,#{},<0.30151.8>,#Ref<0.876500829.2133721089.102524>,#Ref<0.876500829.2133721089.102513>,#Ref<0.876500829.2133721089.102525>,#Ref<0.876500829.2133721089.102526>,#{<<34,153,151,137,127,133,60,31,45,118,34,198,206,140,157,110>> => {dying_client,<<34,153,151,137,127,133,60,31,45,118,34,198,206,140,157,110>>,13,1149592}},#{<<154,205,50,111,47,233,26,141,148,60,219,162,98,82,8,83>> => {<0.32031.8>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},<<155,151,214,191,250,233,80,107,188,165,234,152,6,142,183,64>> => {<0.31502.8>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},<<248,9,179,210,43,21,236,216,70,158,203,71,186,240,167,174>> => {<0.31755.8>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},<<9,120,152,146,200,215,108,178,121,68,192,180,45,141,58,95>> => {<0.31336.8>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},<<155,145,163,25,252,192,19,186,88,114,121,231,189,77,89,236>> => {<0.31961.8>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},<<29,117,98,229,253,193,138,146,113,101,20,164,141,155,34,202>> => {<0.31468.8>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},<<130,28,253,133,108,190,62,159,78,131,154,243,175,17,196,215>> => {<0.31146.8>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},<<187,191,168,136,240,137,152,208,...>> => ...,...},...}
** Reason for termination ==
** {bad_return_value,{bad_flying_ets_record,-1,2,{<<141,221,190,68,105,228,161,126,154,179,139,214,249,216,207,216>>,<<34,153,151,137,127,133,60,31,45,118,34,198,206,140,157,110>>}}}
2019-08-29 13:14:02.294 [error] <0.30148.8> CRASH REPORT Process <0.30148.8> with 0 neighbours exited with reason: bad return value: {bad_flying_ets_record,-1,2,{<<141,221,190,68,105,228,161,126,154,179,139,214,249,216,207,216>>,<<34,153,151,137,127,133,60,31,45,118,34,198,206,140,157,110>>}} in gen_server2:terminate/3 line 1172
2019-08-29 13:14:02.303 [error] <0.30140.8> Supervisor {<0.30140.8>,rabbit_vhost_sup} had child msg_store_persistent started with rabbit_msg_store:start_link(msg_store_persistent, "/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR", [], {#Fun<rabbit_queue_index.2.32138423>,{start,[]}}) at <0.30148.8> exit with reason bad return value: {bad_flying_ets_record,-1,2,{<<141,221,190,68,105,228,161,126,154,179,139,214,249,216,207,216>>,<<34,153,151,137,127,133,60,31,45,118,34,198,206,140,157,110>>}} in context child_terminated
2019-08-29 13:14:02.306 [error] <0.30140.8> Supervisor {<0.30140.8>,rabbit_vhost_sup} had child msg_store_persistent started with rabbit_msg_store:start_link(msg_store_persistent, "/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR", [], {#Fun<rabbit_queue_index.2.32138423>,{start,[]}}) at <0.30148.8> exit with reason reached_max_restart_intensity in context shutdown

2019-08-29 13:14:32.284 [error] <0.31876.8> Supervisor {<0.31876.8>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"aerdata-dev">>,queue,<<"Stream.AmosImport.Bus:ErrorQueue">>},true,false,none,...}, slave, <0.31875.8>) at <0.32626.8> exit with reason killed in context shutdown_error
2019-08-29 13:14:32.284 [info] <0.30145.8> Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR/msg_store_transient'
2019-08-29 13:14:32.296 [info] <0.30145.8> Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@i-aer-lin-rmq07/msg_stores/vhosts/4Z7XLED40TCLZJB3HTDL65TPR/msg_store_transient' is stopped
2019-08-29 13:14:32.300 [error] <0.24333.8> Supervisor {<0.24333.8>,rabbit_vhost_sup_wrapper} had child rabbit_vhost_sup started with rabbit_vhost_sup_wrapper:start_vhost_sup(<<"aerdata-dev">>) at <0.30140.8> exit with reason shutdown in context child_terminated
2019-08-29 13:14:32.300 [error] <0.24333.8> Supervisor {<0.24333.8>,rabbit_vhost_sup_wrapper} had child rabbit_vhost_sup started with rabbit_vhost_sup_wrapper:start_vhost_sup(<<"aerdata-dev">>) at <0.30140.8> exit with reason reached_max_restart_intensity in context shutdown
2019-08-29 13:14:32.300 [info] <0.433.0> Closing all connections in vhost 'aerdata-dev' on node 'rabbit@i-aer-lin-rmq07' because the vhost is stopping

2019-08-29 13:16:18.478 [error] <0.6048.9> ** Generic server <0.6048.9> terminating
** Last message in was {'$gen_cast',go}
** When Server state == {not_started,{amqqueue,{resource,<<"aerdata-main">>,queue,<<"ContentServer.Bus:ReconciliationResource(CMS.Messages.Reconciliations.ReconciliationRemoved)">>},true,false,none,[],<6195.8516.2>,[<6194.27365.2>],[<6194.27365.2>],['rabbit@i-aer-lin-rmq05'],[{vhost,<<"aerdata-main">>},{name,<<"ha-all">>},{pattern,<<>>},{'apply-to',<<"all">>},{definition,[{<<"ha-mode">>,<<"all">>},{<<"ha-sync-mode">>,<<"automatic">>},{<<"max-length-bytes">>,1024000000},{<<"overflow">>,<<"reject-publish">>}]},{priority,0}],undefined,[{<6194.27366.2>,<6194.27365.2>},{<6195.8517.2>,<6195.8516.2>}],[],live,0,[],<<"aerdata-main">>,#{user => <<"app_user">>}}}
** Reason for termination ==
** {{bad_return_value,{bad_flying_ets_record,-1,2,{<<230,28,89,135,175,85,181,136,88,72,209,102,209,185,194,191>>,<<175,190,124,226,143,76,173,171,75,36,175,165,216,52,68,137>>}}},{gen_server2,call,[<0.4111.9>,{new_client_state,<<122,146,75,178,251,200,131,28,80,146,36,246,11,251,7,101>>,<0.6048.9>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},infinity]}}
2019-08-29 13:16:18.480 [error] <0.6061.9> ** Generic server <0.6061.9> terminating
** Last message in was {'$gen_cast',go}
** When Server state == {not_started,{amqqueue,{resource,<<"aerdata-main">>,queue,<<"Stream.ImageManipulationProcessor.Bus:CleanupTempFileProcessor(Stream.Messages.ImageEditing.CleanupTempFileMessage)">>},true,false,none,[],<6195.8348.2>,[<6194.27369.2>],[<6194.27369.2>],['rabbit@i-aer-lin-rmq05'],[{vhost,<<"aerdata-main">>},{name,<<"ha-all">>},{pattern,<<>>},{'apply-to',<<"all">>},{definition,[{<<"ha-mode">>,<<"all">>},{<<"ha-sync-mode">>,<<"automatic">>},{<<"max-length-bytes">>,1024000000},{<<"overflow">>,<<"reject-publish">>}]},{priority,0}],undefined,[{<6194.27370.2>,<6194.27369.2>},{<6195.8349.2>,<6195.8348.2>}],[],live,0,[],<<"aerdata-main">>,#{user => <<"app_user">>}}}
** Reason for termination ==
** {{bad_return_value,{bad_flying_ets_record,-1,2,{<<230,28,89,135,175,85,181,136,88,72,209,102,209,185,194,191>>,<<175,190,124,226,143,76,173,171,75,36,175,165,216,52,68,137>>}}},{gen_server2,call,[<0.4111.9>,{new_client_state,<<3,45,17,6,82,24,91,245,180,57,186,24,100,227,200,214>>,<0.6061.9>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},infinity]}}
2019-08-29 13:16:18.485 [error] <0.6091.9> ** Generic server <0.6091.9> terminating
** Last message in was {'$gen_cast',go}
** When Server state == {not_started,{amqqueue,{resource,<<"aerdata-main">>,queue,<<"ContentServer.Bus:EtaResource(CMS.Messages.ETAs.EtaRemoved)">>},true,false,none,[],<6195.8078.2>,[<6194.27397.2>],[<6194.27397.2>],['rabbit@i-aer-lin-rmq05'],[{vhost,<<"aerdata-main">>},{name,<<"ha-all">>},{pattern,<<>>},{'apply-to',<<"all">>},{definition,[{<<"ha-mode">>,<<"all">>},{<<"ha-sync-mode">>,<<"automatic">>},{<<"max-length-bytes">>,1024000000},{<<"overflow">>,<<"reject-publish">>}]},{priority,0}],undefined,[{<6194.27398.2>,<6194.27397.2>},{<6195.8079.2>,<6195.8078.2>}],[],live,0,[],<<"aerdata-main">>,#{user => <<"app_user">>}}}
** Reason for termination ==
** {{bad_return_value,{bad_flying_ets_record,-1,2,{<<230,28,89,135,175,85,181,136,88,72,209,102,209,185,194,191>>,<<175,190,124,226,143,76,173,171,75,36,175,165,216,52,68,137>>}}},{gen_server2,call,[<0.4111.9>,{new_client_state,<<84,91,89,165,241,225,231,94,36,12,159,170,197,134,89,49>>,<0.6091.9>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},infinity]}}
2019-08-29 13:16:18.487 [error] <0.6095.9> ** Generic server <0.6095.9> terminating
** Last message in was {'$gen_cast',go}
** When Server state == {not_started,{amqqueue,{resource,<<"aerdata-main">>,queue,<<"Management.Web-SV-AD-IIS09(5720):JobQueueAuditClientUpdater(Management.Messages.Jobs.QueueEventRaised)">>},true,false,none,[{<<"x-expires">>,signedint,300000}],<6195.24187.422>,[<6194.4024.373>],[<6194.4024.373>],['rabbit@i-aer-lin-rmq05'],[{vhost,<<"aerdata-main">>},{name,<<"ha-all">>},{pattern,<<>>},{'apply-to',<<"all">>},{definition,[{<<"ha-mode">>,<<"all">>},{<<"ha-sync-mode">>,<<"automatic">>},{<<"max-length-bytes">>,1024000000},{<<"overflow">>,<<"reject-publish">>}]},{priority,0}],undefined,[{<6194.4029.373>,<6194.4024.373>},{<6195.24189.422>,<6195.24187.422>}],[],live,0,[],<<"aerdata-main">>,#{user => <<"app_user">>}}}
** Reason for termination ==
** {{bad_return_value,{bad_flying_ets_record,-1,2,{<<230,28,89,135,175,85,181,136,88,72,209,102,209,185,194,191>>,<<175,190,124,226,143,76,173,171,75,36,175,165,216,52,68,137>>}}},{gen_server2,call,[<0.4111.9>,{new_client_state,<<104,23,214,50,140,194,10,102,226,250,53,145,245,96,81,106>>,<0.6095.9>,#Fun<rabbit_variable_queue.9.38133315>,#Fun<rabbit_variable_queue.28.38133315>},infinity]}}
2019-08-29 13:16:18.488 [error] <0.6091.9> CRASH REPORT Process <0.6091.9> with 1 neighbours exited with reason: bad return value: {bad_flying_ets_record,-1,2,{<<230,28,89,135,175,85,181,136,88,72,209,102,209,185,194,191>>,<<175,190,124,226,143,76,173,171,75,36,175,165,216,52,68,137>>}} in gen_server2:call(<0.4111.9>, {new_client_state,<<84,91,89,165,241,225,231,94,36,12,159,170,197,134,89,49>>,<0.6091.9>,#Fun<rabbit..>,...}, infinity) in gen_server2:terminate/3 line 1172
2019-08-29 13:16:18.490 [error] <0.6048.9> CRASH REPORT Process <0.6048.9> with 1 neighbours exited with reason: bad return value: {bad_flying_ets_record,-1,2,{<<230,28,89,135,175,85,181,136,88,72,209,102,209,185,194,191>>,<<175,190,124,226,143,76,173,171,75,36,175,165,216,52,68,137>>}} in gen_server2:call(<0.4111.9>, {new_client_state,<<122,146,75,178,251,200,131,28,80,146,36,246,11,251,7,101>>,<0.6048.9>,#Fun<rabbi..>,...}, infinity) in gen_server2:terminate/3 line 1172
2019-08-29 13:16:18.492 [error] <0.6095.9> CRASH REPORT Process <0.6095.9> with 1 neighbours exited with reason: bad return value: {bad_flying_ets_record,-1,2,{<<230,28,89,135,175,85,181,136,88,72,209,102,209,185,194,191>>,<<175,190,124,226,143,76,173,171,75,36,175,165,216,52,68,137>>}} in gen_server2:call(<0.4111.9>, {new_client_state,<<104,23,214,50,140,194,10,102,226,250,53,145,245,96,81,106>>,<0.6095.9>,#Fun<rabb..>,...}, infinity) in gen_server2:terminate/3 line 1172
2019-08-29 13:16:18.495 [error] <0.6090.9> Supervisor {<0.6090.9>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"aerdata-main">>,queue,<<"ContentServer.Bus:EtaResource(CMS.Messages.ETAs.Et...">>},...}, slave, <0.6089.9>) at <0.6091.9> exit with reason bad return value: {bad_flying_ets_record,-1,2,{<<230,28,89,135,175,85,181,136,88,72,209,102,209,185,194,191>>,<<175,190,124,226,143,76,173,171,75,36,175,165,216,52,68,137>>}} in gen_server2:call(<0.4111.9>, {new_client_state,<<84,91,89,165,241,225,231,94,36,12,159,170,197,134,89,49>>,<0.6091.9>,#Fun<rabbit..>,...}, infinity) in context shutdown_error
2019-08-29 13:16:18.497 [error] <0.6047.9> Supervisor {<0.6047.9>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"aerdata-main">>,queue,<<"ContentServer.Bus:ReconciliationResource(CMS.Messa...">>},...}, slave, <0.6046.9>) at <0.6048.9> exit with reason bad return value: {bad_flying_ets_record,-1,2,{<<230,28,89,135,175,85,181,136,88,72,209,102,209,185,194,191>>,<<175,190,124,226,143,76,173,171,75,36,175,165,216,52,68,137>>}} in gen_server2:call(<0.4111.9>, {new_client_state,<<122,146,75,178,251,200,131,28,80,146,36,246,11,251,7,101>>,<0.6048.9>,#Fun<rabbi..>,...}, infinity) in context shutdown_error
2019-08-29 13:16:18.499 [error] <0.6061.9> CRASH REPORT Process <0.6061.9> with 1 neighbours exited with reason: bad return value: {bad_flying_ets_record,-1,2,{<<230,28,89,135,175,85,181,136,88,72,209,102,209,185,194,191>>,<<175,190,124,226,143,76,173,171,75,36,175,165,216,52,68,137>>}} in gen_server2:call(<0.4111.9>, {new_client_state,<<3,45,17,6,82,24,91,245,180,57,186,24,100,227,200,214>>,<0.6061.9>,#Fun<rabbit_va..>,...}, infinity) in gen_server2:terminate/3 line 1172
2019-08-29 13:16:18.501 [error] <0.6094.9> Supervisor {<0.6094.9>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"aerdata-main">>,queue,<<"Management.Web-SV-AD-IIS09(5720):JobQueueAuditClie...">>},...}, slave, <0.6093.9>) at <0.6095.9> exit with reason bad return value: {bad_flying_ets_record,-1,2,{<<230,28,89,135,175,85,181,136,88,72,209,102,209,185,194,191>>,<<175,190,124,226,143,76,173,171,75,36,175,165,216,52,68,137>>}} in gen_server2:call(<0.4111.9>, {new_client_state,<<104,23,214,50,140,194,10,102,226,250,53,145,245,96,81,106>>,<0.6095.9>,#Fun<rabb..>,...}, infinity) in context shutdown_error
2019-08-29 13:16:18.504 [error] <0.6060.9> Supervisor {<0.6060.9>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"aerdata-main">>,queue,<<"Stream.ImageManipulationProcessor.Bus:CleanupTempF...">>},...}, slave, <0.6059.9>) at <0.6061.9> exit with reason bad return value: {bad_flying_ets_record,-1,2,{<<230,28,89,135,175,85,181,136,88,72,209,102,209,185,194,191>>,<<175,190,124,226,143,76,173,171,75,36,175,165,216,52,68,137>>}} in gen_server2:call(<0.4111.9>, {new_client_state,<<3,45,17,6,82,24,91,245,180,57,186,24,100,227,200,214>>,<0.6061.9>,#Fun<rabbit_va..>,...}, infinity) in context shutdown_error

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/5f21de7c-307f-418b-8d7a-05dae7ca8055%40googlegroups.com.

--

Mohammed Rehan

[Luke Bakken]

Hi Mohammed,

You can see bad_flying_ets_record in as the source of the error, which is raised here - https://github.com/rabbitmq/rabbitmq-server/blob/master/src/rabbit_msg_store.erl#L1085

I've never seen this error nor has it been reported. All I can suggest at this point is to delete and re-create the vhost. Something has corrupted the message backing store file (I think).

Thanks -
Luke

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/5f21de7c-307f-418b-8d7a-05dae7ca8055%40googlegroups.com.

--

Mohammed Rehan

[Mohammed Reehan]

Hi Luke, 

Ok, I will delete and re-create the vhost on the node and monitor the situation. 

i shall keep you posted. 

Regards

Reehan

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/5f21de7c-307f-418b-8d7a-05dae7ca8055%40googlegroups.com.

--

Mohammed Rehan

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/e731a894-b1a4-4059-ab33-8f5d99621558%40googlegroups.com.

--

Mohammed Rehan