any step by step tutorial on federation exchange?

[Mauricio Porras]

Hi there! I really need some hands  ... I have been trying to make it work like 10 hours so far and is driving me crazy! I followed the documentation but I don't know what I'm doing wrong or what.

What I have tried so far. One rabbitMq instance running on my ubuntu local system

1) create two virtual host (branch, server)
2) create two users ( branch_user, server_user) and added permission for each vhost respectively
3) added Federation Upstreams (web UI) following documentation https://www.rabbitmq.com/federation.html
  it means under my /server vhost (for this I need to allow guest user to have access to /server vhost (if not it shows me an error)
4) added the policy by selection /server vhost and following documentation https://www.rabbitmq.com/federation.html
5) Go to federation status and it shows multiple running links and with "running" status (on green)

then I put my worker to listen to /server vhost..... go to my client app and push some messages to my /branch vhost...the messages are never show in the /server vhost :(  ( If I point my worker to /branch vhost the messages are processed correctly)

what's wrong with my conf ? :( thanks in advance!

[Michael Klishin]

 On 4 September 2014 at 22:25:49, Mauricio Porras (maurici...@gmail.com) wrote:
> 1) create two virtual host (branch, server)
> 2) create two users ( branch_user, server_user) and added permission
> for each vhost respectively
> 3) added Federation Upstreams (web UI) following documentation
> https://www.rabbitmq.com/federation.html
> it means under my /server vhost (for this I need to allow guest
> user to have access to /server vhost (if not it shows me an error)
> 4) added the policy by selection /server vhost and following
> documentation https://www.rabbitmq.com/federation.html
> 5) Go to federation status and it shows multiple running links
> and with "running" status (on green)
>
> then I put my worker to listen to /server vhost..... go to my client
> app and push some messages to my /branch vhost...the messages
> are never show in the /server vhost :( ( If I point my worker to /branch
> vhost the messages are processed correctly)

Are you publishing to an exchange that is federated? Simply seeing links
doesn't mean that every message will be federated.

Are your queues bound to the federated exchange? Federated exchanges need to be
bound just like regular ones.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Mauricio Porras]

Hi Michael! thanks again dude! ...hemm I didn't touch the /branch vhost ...should I do something in there too? sorry if my questions are too basic...I'm quite new with this and the documentation is not very specific :(

[Michael Klishin]

 On 4 September 2014 at 23:31:09, Mauricio Porras (maurici...@gmail.com) wrote:
> I didn't touch the /branch vhost ...should I do something in
> there too

Let's take a step back and check what your expectations are.

You can use federation between vhosts in a single node. That's not common but technically
works just fine.

What vhost should be upstream ("source") and downstream ("target") for federation? It's not
uncommon to see people confuse upstream, downstream and message flow direction.

What's your policy like? What exchange do you publish to in the upstream?

[Mauricio Porras]

Okey ...thanks for you patience !


what I'm doing right now is a conceptual test ..I'm trying to see if this setup will satisfy the requirement, but I don't have multiple machines available or I get lazy to install a vm just for probe this ...and as you said it should work in a single node.

In summary I'm gonna have multiple brokers handling their own queues (in different physical locations) ...and that multiple brokers will publish their data to a main queue where the messages will be processed by multiple workers.

Is something like

branch_broker1     ----------->
branch_broker2     -----------> WAN -----------> Main Broker <---- consumer workers
branch_broker_n   ----------->

For make it work I just created one vhost called /branch_broker1 (the upstream) for instance. And another called /main_broker (dowstream) ... then I went to add Federation Upstreams ...in that page the system ask me in a dropdown on which vhost I want to make this operation... I have selected /main_broker ... it means, add the upstream /branch_broker1 to /main_broker downstream.

So far so good?

[Mauricio Porras]

Oh I forgot to tell you ...I'm using celery to publish and to consume ! apparently that makes some differences when you create the exchanges and queues ..right?

[Michael Klishin]

 On 5 September 2014 at 04:26:30, Mauricio Porras (maurici...@gmail.com) wrote:
> Oh I forgot to tell you ...I'm using celery to publish and to consume
> ! apparently that makes some differences when you create the
> exchanges and queues ..right?

What makes a difference is exchange names and what your policies
match, with Celery or not.

I've asked this before and got not answer:

What does your policy definition look like? What exchange do you publish to in the upstream?

[Michael Klishin]

On 5 September 2014 at 18:43:49, Mauricio Porras (maurici...@gmail.com) wrote:
> I'll show you what I hav

What exchange do you publish messages to?

Only exchanges celery.exchange and celery.pidbox are federated. So whatever is published
to them in the vhost "branch" (which is your upstream) will be republished to the
vhost you have on the screenshot.

Messages flow from upstream to downstream, and only for federated exchanges. 

[Mauricio Porras]

I'm attaching my screenshots again, apparently the website got mad

[Michael Klishin]

On 5 September 2014 at 19:05:22, Mauricio Porras (maurici...@gmail.com) wrote:
> I'm attaching my screenshots again, apparently the website
> got mad

 Mauricio,

I don't have anything to add to
https://groups.google.com/d/msg/rabbitmq-users/RiQNEy8flkY/VpjOf79dXv8J

My best guess is that you expect messages to flow from downstream to upstream,
while the direction is actually the opposite.

I've asked you what exchange do you publish messages to (and what vhost
is supposed to be the source vs. destination) but to no avail.

I'm sorry, I cannot help you any further without this information. You have
links set up from vhost "branch" (upstream) to [some downstream vhost].
Your policy matches two exchanges. You do not provide any other information,
even when asked repeatedly.

[Mauricio Porras]

I thought I have clarified that point since very begining :(

vhost /branch (upstream) here's where I'm publishing messages by using celery from my python code

vhost /server (downstream) I'm consuming messages in here with celeryd

(--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery )

then on /server vhost I added /branch as a federation upstream. I haven't created any exchange or queue because as I have noticed it's are created in background by celery.

[Michael Klishin]

 On 5 September 2014 at 20:13:32, Mauricio Porras (maurici...@gmail.com) wrote:
> vhost /branch (upstream) here's where I'm publishing messages
> by using celery from my python code
>
> vhost /server (downstream) I'm consuming messages in here with
> celeryd
>
> (--- ***** ----- [queues]
> -------------- .> celery exchange=celery(direct) key=celery
> )

If you take a look at
https://lh5.googleusercontent.com/-TMZsWvq4tUA/VAnQpaqOj9I/AAAAAAAAAH4/886gSIhhgJE/s1600/exchanges.png

you'll see that your policy only matches "celery.exchange" and "celery.pidbox"
but not "celery". So "celery" is not federated (there is no "upstream" column
for it). And messages published to non-federated exchanges are not replicated
over federation links.

Either change your policy regexp to be "^celery" (without the \.) or use a different
exchange.

[Mauricio Porras]

Hey! kinda good news lol ...I'm able to receive the messages on my consumer on /server vhost...but looks like some security mechanism is playing me a bad joke...this is what my consumer is showing me every time they get a new message.

[2014-09-05 17:39:37,911: CRITICAL/MainProcess] Can't decode message body: ContentDisallowed('Refusing to deserialize untrusted content of type pickle (application/x-python-serialize)',) [type:u'application/x-python-serialize' encoding:u'binary' headers:{u'x-received-from': [{u'redelivered': False, u'uri': u'amqp://localhost/branch', u'cluster-name': u'rabbit@mporras-lap', u'exchange': u'celery'}]}]

[Michael Klishin]

 On 6 September 2014 at 01:42:54, Mauricio Porras (maurici...@gmail.com) wrote:
> [2014-09-05 17:39:37,911: CRITICAL/MainProcess] Can't
> decode message body: ContentDisallowed('Refusing to deserialize
> untrusted content of type pickle (application/x-python-serialize)',)
> [type:u'application/x-python-serialize' encoding:u'binary'
> headers:{u'x-received-from': [{u'redelivered': False,
> u'uri': u'amqp://localhost/branch', u'cluster-name': u'rabbit@mporras-lap',
> u'exchange': u'celery'}]}]

This is a Celery thing:
http://docs.celeryproject.org/en/latest/userguide/security.html#serializers

You can make Celery use JSON for serialization:

http://tglei.blogspot.ru/2014/02/contentdisallowed-refusing-to.html
http://stackoverflow.com/questions/6628016/how-to-disallow-pickle-serialization-in-celery

[Mauricio Porras]

yay! almost there! now is executing the task, but it's still show me en error

[2014-09-05 17:58:06,434: ERROR/MainProcess] Control command error: TypeError('dispatch() takes at least 2 arguments (1 given)',)
Traceback (most recent call last):
  File "/home/mporras/serverApp_py2env/local/lib/python2.7/site-packages/celery/worker/pidbox.py", line 38, in on_message
    self.node.handle_message(body, message)
  File "/home/mporras/serverApp_py2env/local/lib/python2.7/site-packages/kombu/pidbox.py", line 133, in handle_message
    return self.dispatch(**kwdict(body))
TypeError: dispatch() takes at least 2 arguments (1 given)
[2014-09-05 17:58:07,447: WARNING/Worker-2] My Message

[Michael Klishin]

On 6 September 2014 at 01:58:38, Mauricio Porras (maurici...@gmail.com) wrote:
> error: TypeError('dispatch() takes at least 2 arguments (1
> given)',) 

> File "/home/mporras/serverApp_py2env/local/lib/python2.7/site-packages/celery/worker/pidbox.py",
> line 38, in on_message

You should be able to figure this one out on your own.