Issue while publishing to LVC exchange

[Ojha, Ashish]

Hi Team,

 

I get the below error when try to publish to a LVC exchange , the connections is closed by the broker .

any idea what wrong I am doing  ?

 

client logs ;

 

[20150710 10:32:39.842-S-OmaServerMessage-Thread-3] Unexpected exception in OmaServerMessage.process(): com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; protocol method: #method<connection.close>(reply-code=541, reply-text=INTERNAL_ERROR, class-id=0, method-id=0)

com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; protocol method: #method<connection.close>(reply-code=541, reply-text=INTERNAL_ERROR, class-id=0, method-id=0)
        at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:195)
        at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:296)
        at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:648)
        at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:631)
        at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:622)
        at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicPublish(AutorecoveringChannel.java:160)

Broker logs :

 

=ERROR REPORT==== 10-Jul-2015::09:43:46 ===

** Generic server <0.9385.62> terminating

** Last message in was {'$gen_cast',

                           {method,

                               {'queue.bind',0,

                                   <<"NONPROD.JETSTREAM.QA.JetStreamOma.jtdb">>,

                                   <<"NONPROD.JETSTREAM.QA.JSQA.Lvc">>,

                                   <<"JetStreamOma.jtdb">>,false,[]},

                               none,noflow}}

** When Server state == {ch,running,rabbit_framing_amqp_0_9_1,1,<0.9381.62>,

                            <0.9387.62>,<0.9381.62>,

                            <<"10.152.132.150:55490 -> 10.152.155.201:25009">>,

                            {lstate,<0.9386.62>,false},

                                                                                   

                            <<"10.152.132.150:55490 -> 10.152.155.201:25009">>,

                            {lstate,<0.9386.62>,false},

                            none,1,

                            {[],[]},

                            {user,<<"nonprod.jetstream-oma">>,[],

                                [{rabbit_auth_backend_internal,none}]},

                            <<"/">>,

                            <<"NONPROD.JETSTREAM.QA.JetStreamOma.jtdb">>,

                            {dict,0,16,16,8,80,48,

                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                 []},

                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                  [],[]}}},

                            {state,

                                {dict,0,16,16,8,80,48,

                                    {[],[],[],[],[],[],[],[],[],[],[],[],[],

                                     [],[],[]},

                                    {{[],[],[],[],[],[],[],[],[],[],[],[],[],

                                      [],[],[]}}},

                                erlang},

                            {dict,0,16,16,8,80,48,

                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                 []},

                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                  [],[]}}},

                            {dict,0,16,16,8,80,48,

                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                 []},

                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                  [],[]}}},

                            {set,0,16,16,8,80,48,

                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                 []},

                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],

                                  [],[]}}},

                            <0.9380.62>,

                            {state,fine,5000,#Ref<0.0.66.224148>},

                            false,1,

                            {{0,nil},{0,nil}},

                            [],

                            {{0,nil},{0,nil}},

                            [{<<"exchange_exchange_bindings">>,bool,true},

                             {<<"authentication_failure_close">>,bool,true},

                                                                                       

                            [{<<"exchange_exchange_bindings">>,bool,true},

                             {<<"authentication_failure_close">>,bool,true},

                             {<<"consumer_cancel_notify">>,bool,true},

                             {<<"basic.nack">>,bool,true},

                             {<<"publisher_confirms">>,bool,true},

                             {<<"connection.blocked">>,bool,true}],

                            none,5,none}

** Reason for termination ==

** {{aborted,

        {no_exists,

            [lvc,

             {cachekey,

                 {resource,<<"/">>,exchange,

                     <<"NONPROD.JETSTREAM.QA.JSQA.Lvc">>},

                 <<"JetStreamOma.jtdb">>}]}},

    [{mnesia,abort,1,[{file,"mnesia.erl"},{line,309}]},

     {rabbit_exchange_type_lvc,add_binding,3,

         [{file,"src/rabbit_exchange_type_lvc.erl"},{line,66}]},

     {rabbit_binding,x_callback,4,

         [{file,"src/rabbit_binding.erl"},{line,561}]},

     {rabbit_binding,'-add/3-fun-0-',3,

         [{file,"src/rabbit_binding.erl"},{line,193}]},

     {rabbit_channel,binding_action,9,

         [{file,"src/rabbit_channel.erl"},{line,1471}]},

     {rabbit_channel,handle_cast,2,

         [{file,"src/rabbit_channel.erl"},{line,326}]},

     {gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1034}]},

     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

 

=ERROR REPORT==== 10-Jul-2015::09:43:46 ===

Error on AMQP connection <0.9381.62> (10.152.132.150:55490 -> 10.152.155.201:25009, vhost: '/', user: 'nonprod.jetstream-oma', state: running), channel 1:

{{aborted,

     {no_exists,

         [lvc,

          {cachekey,

              {resource,<<"/">>,exchange,<<"NONPROD.JETSTREAM.QA.JSQA.Lvc">>},

              <<"JetStreamOma.jtdb">>}]}},

[{mnesia,abort,1,[{file,"mnesia.erl"},{line,309}]},

  {rabbit_exchange_type_lvc,add_binding,3,

      [{file,"src/rabbit_exchange_type_lvc.erl"},{line,66}]},

 {rabbit_binding,x_callback,4,[{file,"src/rabbit_binding.erl"},{line,561}]},

  {rabbit_binding,'-add/3-fun-0-',3,

      [{file,"src/rabbit_binding.erl"},{line,193}]},

                                                                                          

  {rabbit_binding,'-add/3-fun-0-',3,

      [{file,"src/rabbit_binding.erl"},{line,193}]},

  {rabbit_channel,binding_action,9,

      [{file,"src/rabbit_channel.erl"},{line,1471}]},

  {rabbit_channel,handle_cast,2,[{file,"src/rabbit_channel.erl"},{line,326}]},

  {gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1034}]},

  {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

 

=WARNING REPORT==== 10-Jul-2015::09:43:46 ===

Non-AMQP exit reason '{{aborted,

                        {no_exists,

                         [lvc,

                          {cachekey,

                           {resource,<<"/">>,exchange,

                            <<"NONPROD.JETSTREAM.QA.JSQA.Lvc">>},

                           <<"JetStreamOma.jtdb">>}]}},

                       [{mnesia,abort,1,[{file,"mnesia.erl"},{line,309}]},

                        {rabbit_exchange_type_lvc,add_binding,3,

                         [{file,"src/rabbit_exchange_type_lvc.erl"},

                          {line,66}]},

                        {rabbit_binding,x_callback,4,

                         [{file,"src/rabbit_binding.erl"},{line,561}]},

                        {rabbit_binding,'-add/3-fun-0-',3,

                         [{file,"src/rabbit_binding.erl"},{line,193}]},

                        {rabbit_channel,binding_action,9,

                         [{file,"src/rabbit_channel.erl"},{line,1471}]},

                        {rabbit_channel,handle_cast,2,

                         [{file,"src/rabbit_channel.erl"},{line,326}]},

                        {gen_server2,handle_msg,2,

                         [{file,"src/gen_server2.erl"},{line,1034}]},

                        {proc_lib,init_p_do_apply,3,

                         [{file,"proc_lib.erl"},{line,239}]}]}'

[Michael Klishin]

What RabbitMQ and LVC plugin versions do you use? Does this happen immediately after 1st message published?

MK

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ojha, Ashish]

version is 3.5.0 and yes this happens immediately after the first message is published..

[Michael Klishin]

On 10 July 2015 at 14:04:12, Ojha, Ashish (ashis...@gs.com) wrote:
> version is 3.5.0 and yes this happens immediately after the
> first message is published

What version of the plugin? Did you get it from our community plugins
page? [1]

The build there should be compatible with 3.5.x. 

1. http://www.rabbitmq.com/community-plugins.html
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Ojha, Ashish]

plugin version is also 3.5.0 and we get it from github https://github.com/rabbitmq/rabbitmq-lvc-plugin and build it from source ..

do you see incompatibility issues from the logs or is there some configuration issue ?

-----Original Message-----
From: Michael Klishin [mailto:mkli...@pivotal.io]
Sent: Friday, July 10, 2015 5:23 PM
To: Ojha, Ashish [Tech]
Cc: rabbitm...@googlegroups.com

[Michael Klishin]

 On 10 July 2015 at 15:02:08, Ojha, Ashish (ashis...@gs.com) wrote:
> plugin version is also 3.5.0 and we get it from github https://github.com/rabbitmq/rabbitmq-lvc-plugin
> and build it from source

Please use RabbitMQ 3.5.3 and the pre-compiled plugin version from our community
plugins page.

[Ojha, Ashish]

ok . Thanks Michael .

it will take some time for us to move to 3.5.3 ( I don't see any bug reported after 3.5.0 for lvc )

-----Original Message-----
From: Michael Klishin [mailto:mkli...@pivotal.io]
Sent: Friday, July 10, 2015 5:33 PM
To: Ojha, Ashish [Tech]
Cc: rabbitm...@googlegroups.com
Subject: RE: [rabbitmq-users] Issue while publishing to LVC exchange

[Alvaro Videla]

Otherwise use the RabbitMQ Recent History Exchange, which with a History value of 1 works the same as LVC

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send an email to rabbitm...@googlegroups.com.

[Ojha, Ashish]

Thanks .

 

from the below conversion , looks like either LVC is not supported or has some issues ?

or only the latest version is supported for rabbitmq .

 

is this true ?

[Ojha, Ashish]

Hi Michael ,

 

We had this issue with LVC exchange and we figured out a way to fix it . Is it possible if we can get some understanding of the issue please or the behavior of the LVC exchange

 

Scenario :

 

3 Node cluster ( A,B,C ) with Mirroring enabled for all the queues .

On Sunday , Node B and Node C were rebuilt from scratch ( mnesia was recreated ) meaning they went down and joined the cluster as virgin nodes .

Node A was untouched .

After Node B and Node C were back up , it clustered with Node A and everything worked fine .

Later we realized that LVC exchange is not working fine .

Any connection made to LVC exchange was closed down by the broker ( when the publisher invoked basicPublish() method )

All other exchanges were working fine ( topic , direct exchange )

 

 

Question :

 

- Does LVC exchange maintains any kind of state between the nodes in the cluster ?

- In our scenario , only Node A was 'unchanged' , so is it possible that only Node A knew about LVC exchange state and when Node B and Node C joined the cluster there was a conflict between the LVC exchange state in the mnesia LVC table  ?

 

How we resolved the issue

 

- disable LVC plugin .

-  shutdown all the nodes and start them in order ( last disc to go down was the first to come back up )

- enable the LVC plugin .

 

we tested LVC exchange after this resolution and everything was working fine .

 

 

-----Original Message-----
From: Michael Klishin [mailto:mkli...@pivotal.io]
Sent: Friday, July 10, 2015 5:33 PM
To: Ojha, Ashish [Tech]
Cc: rabbitm...@googlegroups.com
Subject: RE: [rabbitmq-users] Issue while publishing to LVC exchange

 

[Alvaro Videla]

The LVC exchange (and the recent history exchange as well) keeps a small table in Mnesia where they store the last N messages.

--

[Ojha, Ashish]

Thanks Alvaro , really appreciate your help on this one .

 

so my assumption and resolution looks fine to you if we hit this issue again ?

[Alvaro Videla]

It would be nice to get what errors were produced in the failing scenario to see if it's something we can fix on our side.

[Ojha, Ashish]

sure , I pasted the error in my first email .

adding the error again .

[Alvaro Videla]

Ah OK, sorry, I thought it was a different error this time.

So if while the cluster was let's call it "in error state", if you declared a new LVC exchange, with a different name from the one you were using, you still get those errors?

It looks like Mnesia is not replicating the contents for that particular LVC table. 

Looking at the code it seems LVC doesn't add table copies:

https://github.com/rabbitmq/rabbitmq-lvc-plugin/blob/master/src/rabbit_lvc_plugin.erl#L18

Which is a bug that we fixed for the recent history exchange plugin: https://github.com/videlalvaro/rabbitmq-recent-history-exchange/blob/master/src/rabbit_exchange_type_recent_history.erl#L119 

This was the related issue https://github.com/videlalvaro/rabbitmq-recent-history-exchange/issues/9

So I think using the LVC plugin you could run on the same issue as well. 

If you use the recent-history-exchange with a "x-recent-history-length" value of 1, then you should get the same behaviour as with the LVC plugin.

[Ojha, Ashish]

Some of the tests I did when we had this issue .

 

1.       Declaring a new LVC exchange is not an issue .

2.       Deleting a LVC exchange is a problem  , it gives below error when I try deleting a LVC exchange .

 

Got response code 500 with body {"error":"Internal Server Error","reason":"{exit,\n {{{error,{no_exists,lvc}},\n [{rabbit_misc,execute_mnesia_transaction,1,\n [{file,\"src/rabbit_misc.erl\"},{line,540}]},\n {rabbit_misc,execute_mnesia_tx_with_tail,1,\n [{file,\"src/rabbit_misc.erl\"},{line,562}]},\n {rabbit_channel,handle_method,3,\n [{file,\"src/rabbit_channel.erl\"},{line,1093}]},\n {rabbit_channel,handle_cast,2,\n [{file,\"src/rabbit_channel.erl\"},{line,326}]},\n {gen_server2,handle_msg,2,[{file,\"src/gen_server2.erl\"},{line,1034}]},\n {proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,239}]}]},\n {gen_server,call,\n [<0.23460.100>,\n {call,{'exchange.delete',0,<<\"ddd\">>,false,false},none,<0.7589.100>},\n infinity]}},\n [{gen_server,call,3,[{file,\"gen_server.erl\"},{line,188}]},\n {rabbit_mgmt_util,'-amqp_request/5-fun-0-',4,\n [{file,\"rabbitmq-management/src/rabbit_mgmt_util.erl\"},{line,416}]},\n {rabbit_mgmt_util,with_channel,5,\n [{file,\"rabbitmq-management/src/rabbit_mgmt_util.erl\"},{line,435}]},\n {webmachine_resource,resource_call,3,\n [{file,\n \"webmachine-wrapper/webmachine-git/src/webmachine_resource.erl\"},\n {line,186}]},\n {webmachine_resource,do,3,\n [{file,\n \"webmachine-wrapper/webmachine-git/src/webmachine_resource.erl\"},\n {line,142}]},\n {webmachine_decision_core,resource_call,1,\n [{file,\n \"webmachine-wrapper/webmachine-git/src/webmachine_decision_core.erl\"},\n {line,48}]},\n {webmachine_decision_core,decision,1,\n [{file,\n \"webmachine-wrapper/webmachine-git/src/webmachine_decision_core.erl\"},\n {line,440}]},\n {webmachine_decision_core,handle_request,2,\n [{file,\n \"webmachine-wrapper/webmachine-git/src/webmachine_decision_core.erl\"},\n {line,33}]}]}\n"}

 

3.       During this issue , publisher code always fails while consumers still can consumer from the queue which is bind to the LVC exchange .

4.       In 3 node cluster if we shut down one node , then publisher fails to publish to LVC exchange . Consumers still works fine .

 

 

Is it possible to fix the LVC exchange for the bug you have mentioned ?

or is LVC exchange no more supported and users should move off LVC and use Recent History Exchange ?

 

We have LVC exchange deployed in production so wondering if the LVC can be fixed . Moving to Recent History Exchange will need time/effort/testing ..

[Alvaro Videla]

Please file an issue on the LVC repo and we'll try to port the fixes to this exchange

[Ojha, Ashish]

Thanks Alvaro .

 

I have filed an issue here https://github.com/rabbitmq/rabbitmq-lvc-plugin/issues/2

 

let me know if you more details  . We will look forward for this fix .

[Alvaro Videla]

Thanks, you can follow progress on Github.

Cheers

[Ojha, Ashish]

Thanks Alvaro .

 

One more thing about LVC exchange , sometimes HTTP management interface says “Exchange type not found” for LVC exchanges  , however I see the LVC exchange is available and I can publish to it and subscribe to a queue bound to it ..

 

I have restarted all the brokers in the clusters but still the LVC exchanges are marked  “Not Found” on management interface .

 

thought of informing if it saves some effort while you fix the previous bug .

 

UI Snapshot

 

 

 

 

all the nodes has this which I think it means LVC is running on nodes .