[rabbitmq-discuss] Round-robin queue failover
Sebastian
I'm looking into a way how to implement a HA setup for persistent
queues. Since the rabbitmq cluster doesn't replicate queues across
multiple nodes my initial idea was to implement this on the client:
connect publisher and consumers to multiple independent servers,
create identical exchanges and queues, then send the messages round
robin to the nodes. While this has the additional benefit of proper
scalability it requires some effort on the client, I basically have to
override all methods and encapsulate the "replication" logic there.
While this isn't a true HA setup (messages in the broken node can't be
delivered until restart and correct order of messages isn't guaranteed
anymore) it solves 90% of the problem by providing a way to allow
continuous message publishing.
Now I have come across this [1] blog-post by Ilya Grigorik which
says: "Assign two identically named queues to an exchange and the
broker will automatically round-robin the messages". So my
understanding is that I could just create identical queues on every
node, and the broker will take care of the rest: if one node goes
down, messages will just be spread over the other queues/nodes.
Is this supported by rabbitmq? I've looked around in the amqp specs
and documentation, but couldn't find anything related.
Regards
Sebastian
[1] http://www.igvita.com/2009/10/08/advanced-messaging-routing-with-amqp/
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq...@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Matthias Radestock
Sebastian wrote:
> Now I have come across this [1] blog-post by Ilya Grigorik which
> says: "Assign two identically named queues to an exchange and the
> broker will automatically round-robin the messages". So my
> understanding is that I could just create identical queues on every
> node, and the broker will take care of the rest: if one node goes
> down, messages will just be spread over the other queues/nodes.
> Is this supported by rabbitmq? I've looked around in the amqp specs
> and documentation, but couldn't find anything related.
>
> [1] http://www.igvita.com/2009/10/08/advanced-messaging-routing-with-amqp/
That part of the article is wrong. Exchanges route messages to all the
queues bound with a binding that matches the message. There is no
round-robining going on; that only happens for delivering messages to
multiple consumers of the same queue.
Regards,
Matthias.
Kyle Schaffrick
Matthias Radestock <matt...@lshift.net> wrote:
> Sebastian,
>
> Sebastian wrote:
> > Now I have come across this [1] blog-post by Ilya Grigorik which
> > says: "Assign two identically named queues to an exchange and the
> > broker will automatically round-robin the messages". So my
> > understanding is that I could just create identical queues on
> > every node, and the broker will take care of the rest: if one node
> > goes down, messages will just be spread over the other queues/nodes.
> > Is this supported by rabbitmq? I've looked around in the amqp
> > specs and documentation, but couldn't find anything related.
> >
> > [1]
> > http://www.igvita.com/2009/10/08/advanced-messaging-routing-with-amqp/
>
> That part of the article is wrong. Exchanges route messages to all
> the queues bound with a binding that matches the message. There is no
> round-robining going on; that only happens for delivering messages to
> multiple consumers of the same queue.
>
> Regards,
>
> Matthias.
>
The author of the article seems to be confusing "declaring and consuming
from the same queue from two different places" with "creating two
identically-named queues" which (if I understand correctly) is not
possible, since declaration is idempotent. I would guess that's why the
operation is called "declare" and not "create".
If you reword it to remove the misunderstanding, then the statement
seems correct. Attempting to create two identically-named queues will
result in declaring the same queue twice; the two consumers then
consuming from that queue will see load-balancing behavior.
-Kyle
Sebastian
Kyle Schaffrick:
> Attempting to create two identically-named queues will
> result in declaring the same queue twice; the two consumers then
> consuming from that queue will see load-balancing behavior.
Thanks for the clarification. Since this does not work: how do I
achieve reliable persistent message delivery even if the one node
holding the queue dies? As stated before my take would be to replicate
queues client-side on independent brokers, but this seems to be
getting quite complicated as the queues have to be "double-declared",
linked and properly re-initialized on startup of a new empty broker
(basically doing some kind of client-side clustering).
Regards
Sebastian
Alexis Richardson
On Mon, Oct 26, 2009 at 4:04 PM, Sebastian <ma...@sebastian-latza.de> wrote:
>
> Kyle Schaffrick:
>
>> Attempting to create two identically-named queues will
>> result in declaring the same queue twice; the two consumers then
>> consuming from that queue will see load-balancing behavior.
>
> Thanks for the clarification. Since this does not work: how do I
> achieve reliable persistent message delivery even if the one node
> holding the queue dies?
At the risk of being seen as pedantic...
You could achieve it by waiting for the node to recover from disk. In
other words: RabbitMQ provides "eventual delivery" out of the box. If
you want to 'persist more reliably' than on one drive using RabbitMQ
out of the box, then use a RAID or a SAN.
I am guessing that what you mean is *availability* in which a message
can always reach a suitable queue even if the sending client loses its
connection with the node that crashed. The problem with using a
single node is the time it takes to recover, and client reconnection
costs (if any). The time to recover is partly influenced by how much
state has to recover, and partly by the size of the unavailability
window that you can tolerate.
Many applications don't like to wait more than (say) 1ms, 10ms or 50ms
before they can send their message. In this case you will typically
want to send a message *before* the crashed node has been able to
recover. You can achieve this in two ways:
1. You don't care about replicating state and can use any other queue
as a 'suitable queue'. This is easy - just make sure the right
consumers and routes are wired up.
2. You do care about replicating state - which I think is the case you
have in mind. Then you can fail over to a passive server. We can
provide some more info on this if you like -- we have done it for a
few people but it is not yet 'available OOTB'. (You can also do
active/active but it is much more fiddly atm).
> As stated before my take would be to replicate
> queues client-side on independent brokers, but this seems to be
> getting quite complicated as the queues have to be "double-declared",
> linked and properly re-initialized on startup of a new empty broker
> (basically doing some kind of client-side clustering).
Right. I think you probably want case 2 above.
alexis
Sebastian
> 2. You do care about replicating state - which I think is the case you
> have in mind. Then you can fail over to a passive server. We can
> provide some more info on this if you like -- we have done it for a
> few people but it is not yet 'available OOTB'. (You can also do
> active/active but it is much more fiddly atm).
>
I want to make sure that when an event happens, it can be delivered to
the correct queue in any case ( "always writable"). As a very
simplified example I would want to send out welcome mails
asynchronously to newly registered customers. If the message queue for
this function wasn't available at the time I'd have to shut down my
registration which I would want to avoid naturally.
I'd be happy to provide you with testing/feedback and maybe some basic
erlang hacking for anything that might help regarding that problem.
Alexis Richardson
> Alexis Richardson:
>
>> 2. You do care about replicating state - which I think is the case you
>> have in mind. Then you can fail over to a passive server. We can
>> provide some more info on this if you like -- we have done it for a
>> few people but it is not yet 'available OOTB'. (You can also do
>> active/active but it is much more fiddly atm).
>>
>
> I want to make sure that when an event happens, it can be delivered to the
> correct queue in any case ( "always writable"). As a very simplified example
> I would want to send out welcome mails asynchronously to newly registered
> customers. If the message queue for this function wasn't available at the
> time I'd have to shut down my registration which I would want to avoid
> naturally.
Right. So does that mean you need to have an 'unavailability' window
of ZERO seconds, or are willing to tolerate holding the message on the
client for delivery within some interval, eg 1ms, 10ms, 100ms, 1sec,
...
In any case, I suspect that active/passive failover would suffice in
your case. More on that soon...
> I'd be happy to provide you with testing/feedback and maybe some basic
> erlang hacking for anything that might help regarding that problem.
Thank-you - we may take you up on that!
alexis
Jason J. W. Williams
clustering. And both consumers and producers declare queues on a fail
over to a hot standby/passive server. That way messages are never
lost. When the old server comes back up we fail back.
-J
Sent via iPhone
On Oct 26, 2009, at 10:23, Alexis Richardson <alexis.r...@gmail.com