Error '{{badmatch,{error,not_allowed}}' on shovel from 3.6.1 to 3.6.6

[egon....@ebanx.com]

Hey guys, I have two RabbitMQ cluster one running 3.6.1 and another running 3.6.6. I am trying to move the messages from the first to the latter but I keep having the following error no mather what.

Here is how I created the shovel:

rabbitmqctl set_parameter shovel -p "/myvhost" "test-shovel" '{
  "src-uri":  "amqp://user:pass...@10.1.1.101:5672/myvhost",
  "src-queue": "myqueue",
  "dest-uri": "amqp://user:pass...@10.1.1.102:5672/myvhost",
  "dest-queue": "myqueue"
}'

1. Both queues and vhosts already exist in both clusters;
2. User and passwords are correct and are not localhost users;
3. Both users have ".*" granted for all vhosts and have administrator profile;
4. Network connection is fine between both nodes (I can telnet on 5672 port with no problems);
5. I am not using SSL;
6. I have also tried a bunch of different URI styles and all had the same error;

However, here is the error message I keep getting:

{{badmatch,{error,not_allowed}},
 [{rabbit_shovel_worker,make_conn_and_chan,1,
                        [{file,"src/rabbit_shovel_worker.erl"},{line,236}]},
  {rabbit_shovel_worker,handle_cast,2,
                        [{file,"src/rabbit_shovel_worker.erl"},{line,62}]},
  {gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1049}]},
  {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

Some other observations:

1. These messages appear in the source server log files.
2. No log messages are shown in destination server which I believe tells us that the error may be related with the connection or message processing to the source server;
3. After this log message the shovel is terminated and then retries;

BTW, the servers are all EC2 in AWS, t2.large instances running Ubuntu 16.04.

Source server is running Erlan R16B03 and destination server is running Erlang 18.3.

Here is the source server rabbitmq.conf file:

[
  {kernel, [

  ]},
  {rabbitmq_management, [
    {listener, [{port, 8443},
                {ssl, false},
                {ssl_opts, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
                    {certfile,"/etc/rabbitmq/ssl/cert.pem"},
                    {keyfile,"/etc/rabbitmq/ssl/key.pem"}


                    ]}
              ]}
  ]},
  {rabbit, [
    {cluster_partition_handling,ignore},
    {ssl_listeners, [5671]},
    {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
                    {certfile,"/etc/rabbitmq/ssl/cert.pem"},
                    {keyfile,"/etc/rabbitmq/ssl/key.pem"},
                    {verify,verify_none},
                    {fail_if_no_peer_cert,false}
                    ]},
    {tcp_listen_options, [binary, {packet,raw},
                                  {reuseaddr,true},
                                  {backlog,128},
                                  {nodelay,true},
                                  {exit_on_close,false},
                                  {keepalive,false},
                                  {linger, {true,0}}]},

    {log_levels, [{ connection, info }]},
    {vm_memory_high_watermark, 0.4},

    {default_user, <<"guest">>},
    {default_pass, <<"guest">>},
    {heartbeat, 580}

  ]}
].

And here is the rabbitmq.conf file in the destination server:

[
  { kernel, [] },
  { rabbitmq_management, [{
    listener, [
      { port, 15672 },
    ]}
  ]},
  { rabbit, [
    { cluster_partition_handling, ignore},
    { ssl_listeners, [ 5671 ]},
    { ssl_options, [
      { cacertfile, "/etc/rabbitmq/ssl/ca.pem" },
      { certfile,   "/etc/rabbitmq/ssl/cert.pem" },
      { keyfile,    "/etc/rabbitmq/ssl/key.pem" }
      { verify, verify_none },
      { fail_if_no_peer_cert, false }
    ]},
    { tcp_listen_options, [
      binary,
      { packet, raw },
      { reuseaddr, true },
      { backlog, 128 },
      { nodelay, true },
      { exit_on_close, false },
      { keepalive, false },
      { linger, { true, 0 } }
    ]},
    {log_levels, [
      { connection, warning }
    ]},
    { vm_memory_high_watermark, 0.4 },
    { default_user, <<"guest">> },
    { default_pass, <<"guest">> },
    { heartbeat, 580 }
  ]}
].

It goes without saying but in any case the shovel plugin is enabled in both servers.

Thank you.

[Michael Klishin]

> {error,not_allowed}

The user used by Shovel doesn't have the permissions for the vhost
it is configured to use.

Shovel uses a RabbitMQ client internally and thus must be treated as a "regular"
app that needs permissions to consume and publish, see http://www.rabbitmq.com/access-control.html.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[egon....@ebanx.com]

Thanks for your reply Michael. Both users I am using are administratora and have all permissions on the vhosts I am using.

However, I am not sure if I should use any special grants when connection to localhost.

Is there any special grants I must use for shovel users? Or just having ".*" in read, write and configure should suffice?

Thank you.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

They need to be able to consumer from the source (typically a queue) and publish to destination.

With any 3.6.x version it's easy to see what client operation fails by looking at the server log.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[egon....@ebanx.com]

I am only seeing this log message I sent in the first mail of this thread nothing else. Maybe I should increase verbosity, I will do that.

Regarding the queues, if they do not exist on the target server... I read in the documentation that it will create the queues in the same manner that it was created in the origin server.

Is there any catch on that process that I may be missing?

Thank you again Michael.

[Michael Klishin]

You have two nodes. What log file are you looking at? The error will be logged

by the node this Shovel tries to connect to.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[egon....@ebanx.com]

I am checking all of them but that message I got on the origin server.

I have set the log levels now to be:

    {log_levels, [

      { channel, debug },

      { connection, debug },

      { federation, debug },

      { mirroring, debug }

    ]},

I saw that one of my users is having denied access to one of the vhosts.

I could've swear I granted that before. Anyway, let me fix that and see if now it works.

Thanks.

[egon....@ebanx.com]

Yeah, the user had the correct permissions indeed and here is the full message I am getting on the origin server.

Nothing new is popping up in destination servers:

=ERROR REPORT==== 28-Nov-2016::14:54:37 ===

** Generic server <0.7727.0> terminating

** Last message in was {'$gen_cast',init}

** When Server state == {state,undefined,undefined,undefined,undefined,

                            {<<"/myvhost">>,<<"test-shovel">>},

                            dynamic,

                            {shovel,

                                {endpoint,

                                    ["amqp://adminuser:password@/myvhost"],

                                    #Fun<rabbit_shovel_parameters.4.22841904>},

                                {endpoint,

                                    ["amqp://adminuser:password@server1:5672/myvhost",

                                     "amqp://adminuser:password@server2:5672/myvhost"],

                                    #Fun<rabbit_shovel_parameters.5.22841904>},

                                1000,on_confirm,

                                #Fun<rabbit_shovel_parameters.6.22841904>,

                                #Fun<rabbit_shovel_parameters.7.22841904>,

                                <<"staging-sandbox-payment-event-upsert-dashboard">>,

                                1,never},

                            undefined,undefined,undefined,undefined,undefined}

** Reason for termination ==

** {{badmatch,{error,not_allowed}},

    [{rabbit_shovel_worker,make_conn_and_chan,1,

                           [{file,"src/rabbit_shovel_worker.erl"},{line,236}]},

     {rabbit_shovel_worker,handle_cast,2,

                           [{file,"src/rabbit_shovel_worker.erl"},{line,62}]},

     {gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1049}]},

     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]

[Michael Klishin]

All that says that a Shovel Erlang process tried to init, was rejected access to a vhost and terminated

abnormally (and then probably was restarted, failed again, and so on for a certain number of retries).

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[egon....@ebanx.com]

Yeah, and I also see this message here n the logs:

access to vhost 'myvhost' refused for user 'adminuser'

However, this user exists and have the following access to /myvhost:

Configure regexp: .*

Write regexp: .*

Read regexp: .*

I am lost about this. =/

[Michael Klishin]

/myvhost is not the same value as myvhost.

Vhosts don't have to include slashes in the name and slashes must be percent-encoded in URI paths.

[egon....@ebanx.com]

Found the issue.

The problem is that on the connection URL I was using this:

amqp://adminuser:password@server:5672/myvhost

So, rabbitmq/shovel was using as vhost "myvhost" which is not correct because the vhost is actually "/vhost". It has a preceding slash.

So I had to build the URL to be:

amqp://adminuser:password@server:5672/%2fmyvhost

And that worked.

Do you think that this should be reported to rabbitmq?

[egon....@ebanx.com]

You replied to me at the same time with the solution as well.

Thank you very much Michael. =)

[Michael Klishin]

This is expected and documented behavior, see AMQP URI spec reference on rabbitmq.com.

[egon....@ebanx.com]

Understood! Again, thank you very much Michael for all the attention on this silly mistake on my part.