Problem creating federation-upstream-set on 3.7.3 & 3.7.4

[d.he...@safecard.cl]

Hello

I've been having problems since I upgraded from 3.6.10 to 3.7.3 when trying to create (or update) federation-upstream-sets. 

I'm using the versions provided on the rabbitmq docker image, including management pluggin (https://hub.docker.com/r/library/rabbitmq/) and the following extra plugins:

* rabbitmq_auth_mechanism_ssl

* rabbitmq_federation

* rabbitmq_federation_management

On the documentation there are 2 ways to create federation-upstream-sets, I had no problems using any of them on 3.6.10.

1.- Using rabbitmqctl (https://www.rabbitmq.com/federation-reference.html)
* rabbitmqctl set_parameter -p <vhost> federation-upstream-set <name> '[json-object, json-object, ...]'

2.- Trough the management API (https://rawcdn.githack.com/rabbitmq/rabbitmq-management/v3.7.4/priv/www/api/index.html)
* POST /api/parameters/federation-upstream-set/<vhost>/<name>

When trying to excecute any of them on 3.7.3, the service returned a validation error

rabbitmqctl set_parameter federation-upstream-set federation-set-barrier_sync '[{"upstream": "ls1-barrier_sync"},{"upstream": "ls3-barrier_sync"}]'
Setting runtime parameter "federation-upstream-set" for component "federation-set-barrier_sync" to "[{"upstream": "ls1-barrier_sync"},{"upstream": "ls3-barrier_sync"}]" in vhost "/" ...
Error:
Validation failed

federation-set-barrier_sync not a list #{<<"upstream">> =>
<<"ls1-barrier_sync">>}
federation-set-barrier_sync not a list #{<<"upstream">> =>
<<"ls3-barrier_sync">>}

I tried upgrading to 3.7.4 on a development enviorment to check if this issue is related to a fixed issue on the repo (https://github.com/rabbitmq/rabbitmq-federation/issues/67), but when creating a new federation-upstream-set I get this other error:

rabbitmqctl set_parameter federation-upstream-set federation-set-barrier_sync '[{"upstream": "ls1-barrier_sync"},{"upstream": "ls3-barrier_sync"}]'
Setting runtime parameter "federation-upstream-set" for component "federation-set-barrier_sync" to "[{"upstream": "ls1-barrier_sync"},{"upstream": "ls3-barrier_sync"}]" in vhost "/" ...
Error:
{:badarg, [{:lists, :keyfind, ["upstream", 1, %{"upstream" => "ls1-barrier_sync"}], []}, {:rabbit_misc, :pget, 3, [file: 'src/rabbit_misc.erl', line: 894]}, {:rabbit_federation_upstream, :bget, 4, [file: 'src/rabbit_federation_upstream.erl', line: 142]}, {:rabbit_federation_upstream, :from_set_element, 2, [file: 'src/rabbit_federation_upstream.erl', line: 106]}, {:rabbit_federation_upstream, :"-from_set_contents/2-lc$^0/1-0-", 2, [file: 'src/rabbit_federation_upstream.erl', line: 102]}, {:rabbit_federation_upstream, :from_set_contents, 2, [file: 'src/rabbit_federation_upstream.erl', line: 102]}, {:rabbit_federation_link_sup, :specs, 1, [file: 'src/rabbit_federation_link_sup.erl', line: 104]}, {:rabbit_federation_link_sup, :adjust, 3, [file: 'src/rabbit_federation_link_sup.erl', line: 35]}]}

Is there an issue with this feature? or i'm doing something wrong? should I rollback to 3.6.10 until this issue is fixed?

[Michael Klishin]

We will take a look, it could be another 3.7-specific issue that was previously hidden by federation#67.

You should move to at least 3.6.15.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Hi,

I tried with 3.7.4 and the tip of the v3.7.x branch (future 3.7.5) and I can't reproduce:

rabbitmqctl set_parameter federation-upstream-set federation-set-barrier_sync '[{"upstream": "ls1-barrier_sync"},{"upstream": "ls3-barrier_sync"}]'
> Setting runtime parameter "federation-upstream-set" for component "federation-set-barrier_sync" to "[{"upstream": "ls1-barrier_sync"},{"upstream": "ls3-barrier_sync"}]" in vhost "/" ...

rabbitmqctl clear_parameter federation-upstream-set federation-set-barrier_sync
> Clearing runtime parameter "federation-set-barrier_sync" for component "federation-upstream-set" on vhost "/" ...

I used `rabbitmqctl list_parameters` and a node shell to see if the value is actually being set and can confirm that it is.

I also tried a 3.6.14 version of rabbitmqctl (which must not be used against 3.7.x nodes). The set_parameter step succeeds

(clear_parameter fails but it is expected because many internal API functions now require one more argument).

I do have an idea of what can make the function in question be more defensive. We will apply that but

a fix without a way to reproduce is only worth so much :/

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[d.he...@safecard.cl]

Hi again

Thanks for helping me out, I ended up rolling back to 3.6.15 and everything is working ok :)

I also tried to reproduce the problem on a clean server. At first I thought that it was a general problem with federation-upstream-set, but it seems to fail on my specific use-case where I connect exchanges on different vhosts on the same server using federation.

docker run -d -p 15672:15672 --name buggy_rabbit rabbitmq:3.7.4-management

docker exec -ti buggy_rabbit bash

# Enable pluggins

rabbitmq-plugins enable rabbitmq_federation

rabbitmq-plugins enable rabbitmq_federation_management

# Create vhost

rabbitmqctl add_vhost api

rabbitmqctl set_permissions -p api guest ".*" ".*" ".*"

# Create Exchanges

rabbitmqadmin -u guest -p guest declare exchange --vhost=api name=test1 type=fanout

rabbitmqadmin -u guest -p guest declare exchange --vhost=api name=test2 type=fanout

# Create Queues

rabbitmqadmin -u guest -p guest declare queue --vhost=api name=testq1

rabbitmqadmin -u guest -p guest declare queue --vhost=api name=testq2

# Bind Queues

rabbitmqadmin -u guest -p guest declare binding --vhost=api source=test1 destination_type="queue" destination=testq1

rabbitmqadmin -u guest -p guest declare binding --vhost=api source=test2 destination_type="queue" destination=testq2

# Create Empty Sets

rabbitmqctl set_parameter -p api federation-upstream-set fed-up-set-test1 '[]'

rabbitmqctl set_parameter -p api federation-upstream-set fed-up-set-test2 '[]'

# Create Policies

rabbitmqctl set_policy -p api --apply-to exchanges fed-pol-test1 "test1" '{"federation-upstream-set":"fed-up-set-test1"}'

rabbitmqctl set_policy -p api --apply-to exchanges fed-pol-test2 "test2" '{"federation-upstream-set":"fed-up-set-test2"}'

#####

# Create client vhost

rabbitmqctl add_vhost client1

rabbitmqctl set_permissions -p client1 guest ".*" ".*" ".*"

# Create Exchanges

rabbitmqadmin -u guest -p guest declare exchange --vhost=client1 name=test1 type=fanout

rabbitmqadmin -u guest -p guest declare exchange --vhost=client1 name=test2 type=fanout

# Create Federated Upstreams

rabbitmqctl set_parameter federation-upstream -p api fed-up-client1-test1 '{"uri":"amqp://guest:gu...@127.0.0.1:5672/client1","ack-mode":"on-confirm","trust-user-id":false,"exchange":"test1","max-hops":1}'

rabbitmqctl set_parameter federation-upstream -p api fed-up-client1-test2 '{"uri":"amqp://guest:gu...@127.0.0.1:5672/client1","ack-mode":"on-confirm","trust-user-id":false,"exchange":"test2","max-hops":1}'

# Add upstreams to sets

rabbitmqctl set_parameter -p api federation-upstream-set fed-up-set-test1 '[{"upstream": "fed-up-client1-test1"}]' # This step doesn't work

rabbitmqctl set_parameter -p api federation-upstream-set fed-up-set-test2 '[{"upstream": "fed-up-client1-test2"}]'

#####

# Create client vhost

rabbitmqctl add_vhost client2

rabbitmqctl set_permissions -p client2 guest ".*" ".*" ".*"

# Create Exchanges

rabbitmqadmin -u guest -p guest declare exchange --vhost=client2 name=test1 type=fanout

rabbitmqadmin -u guest -p guest declare exchange --vhost=client2 name=test2 type=fanout

# Create Federated Upstreams

rabbitmqctl set_parameter federation-upstream -p api fed-up-client2-test1 '{"uri":"amqp://guest:gu...@127.0.0.1:5672/client2","ack-mode":"on-confirm","trust-user-id":false,"exchange":"test1","max-hops":1}'

rabbitmqctl set_parameter federation-upstream -p api fed-up-client2-test2 '{"uri":"amqp://guest:gu...@127.0.0.1:5672/client2","ack-mode":"on-confirm","trust-user-id":false,"exchange":"test2","max-hops":1}'

# Add upstreams to sets

rabbitmqctl set_parameter -p api federation-upstream-set fed-up-set-test1 '[{"upstream": "fed-up-client1-test1"},{"upstream": "fed-up-client2-test1"}]'

rabbitmqctl set_parameter -p api federation-upstream-set fed-up-set-test2 '[{"upstream": "fed-up-client1-test2"},{"upstream": "fed-up-client2-test2"}]'

This whole process works on rabbitmq:3.6.15-management, so it might be something really specific to this way to use this feature. I can't discard that it might be some problem on the docker image or the Erlang version provided (20.2.4).

Also, when I restart the server, the api ends up in a stopped state with the reason "vhost suppervisor is not running".

Hope this helps out.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[d.he...@safecard.cl]

Hi

Tested this again on 3.7.5 and now it's working fine :).

Thank you a lot for the help.

rabbitmqctl set_parameter federation-upstream -p api fed-up-client1-test1 '{"uri":"amqp://guest:guest@127.0.0.1:5672/client1","ack-mode":"on-confirm","trust-user-id":false,"exchange":"test1","max-hops":1}'

rabbitmqctl set_parameter federation-upstream -p api fed-up-client1-test2 '{"uri":"amqp://guest:guest@127.0.0.1:5672/client1","ack-mode":"on-confirm","trust-user-id":false,"exchange":"test2","max-hops":1}'

# Add upstreams to sets

rabbitmqctl set_parameter -p api federation-upstream-set fed-up-set-test1 '[{"upstream": "fed-up-client1-test1"}]' # This step doesn't work

rabbitmqctl set_parameter -p api federation-upstream-set fed-up-set-test2 '[{"upstream": "fed-up-client1-test2"}]'

#####

# Create client vhost

rabbitmqctl add_vhost client2

rabbitmqctl set_permissions -p client2 guest ".*" ".*" ".*"

# Create Exchanges

rabbitmqadmin -u guest -p guest declare exchange --vhost=client2 name=test1 type=fanout

rabbitmqadmin -u guest -p guest declare exchange --vhost=client2 name=test2 type=fanout

# Create Federated Upstreams

rabbitmqctl set_parameter federation-upstream -p api fed-up-client2-test1 '{"uri":"amqp://guest:guest@127.0.0.1:5672/client2","ack-mode":"on-confirm","trust-user-id":false,"exchange":"test1","max-hops":1}'

rabbitmqctl set_parameter federation-upstream -p api fed-up-client2-test2 '{"uri":"amqp://guest:guest@127.0.0.1:5672/client2","ack-mode":"on-confirm","trust-user-id":false,"exchange":"test2","max-hops":1}'

[Michael Klishin]

I hope it will allow you to get over the hump with upgrading to 3.7.x :)

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.