[rabbitmq-discuss] RabbitMQ production setup questions around clustering
Dave Greggory
We are setting up a RabbitMQ cluster as our message broker for servicing our
guaranteed delivery needs. We will have a local data store on our producers for
guaranteeing delivery, but we don't want to resort to that except as a last
ditch effort in cases of catastrophic failures. I would like to better
understand how the cluster setup in RabbitMQ works.
Our setup:
2 rabbitmq nodes clustered together sitting behind a hardware load balancer.
Upon initial release, our message volume will not be that high, but it will grow
real fast once we offload more work to the message broker from our current
non-messaging based infrastructure. Since the initial volume is not very high,
we do not intend to use the load balancer for actual load balancing but to
always send connections to a specific rabbitmq node in the cluster. If RabbitMQ
does not respond or the port is not open, it will automatically switch to the
second node for new connections.
Questions:
1. Are there any case studies for setting up clustering in RabbitMQ?
2. When a queue is declared (by one of our producers), is the queue always
created on the rabbitmq node the producer client has a connection to or is the
queue created on a randomly selected node within the cluster?
3. If the queue is durable and the messages sent to it are marked persistent,
will these messages always be persisted to disk and be available after a restart
of the node that has that queue, regardless of whether the node is a disk node
or RAM node? (This line "Should you do this, and suffer a power
failure to the entire cluster, the entire state of the cluster,
including all messages, will be lost. " in
http://www.rabbitmq.com/clustering.html is confusing)
4. Should I configure both my nodes as disk nodes or will one disk node be
sufficient? In other words, if only 1 disk node was there in the cluster and its
hard drive went bust, what can I recover from the RAM node? If nothing can be
recovered from the RAM node, is it mainly for increasing the number of
connections without taking any hits to disk throughput?
5. Are connections redirects actually supported by the current version? The FAQ
and Clustering documents on site are contradictory of each other. (FAQ says
"Future releases will support live failover using, for instance, a
combination of the "known hosts" field in connection.open-ok and the
connection.redirect message.")
6. If redirects are supported, when all connections are being sent to a specific
rabbitmq node in the cluster by the loadbalancer, will that rabbitmq node still
send a redirect request to the client if it's getting too taxed? If so, then is
it possible to limit redirects to only the disk nodes within the cluster so that
we don't lose any data?
7. Will the connections be redirected solely based on RabbitMQ node's ability to
serve it or is it more round-robin?
8. Is there a way to ensure that the cluster configuration is correct because
'disc/RAM' is not reported correctly by rabbitmqctl (as per
http://old.nabble.com/Rabbitmq-v.-1.8.1---Bug-report---Could-not-start-a-node-as-RAM-node-in-cluster-ts29211668.html)?
Thanks so much,
Dave
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq...@lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Aaron Westendorf
I intend to write up a blog post on our cluster setup which will
hopefully address some of your questions. You're trying to set up
your cluster much the way we originally did, but it's not quite the
way Rabbit or AMQP operate.
Think of it as the postal service, where exchanges are your
street-corner deposit boxes and office tellers, and the postal service
is the cluster itself. Your address is your routing scheme, and that
routing key will deliver a message to your postal box. The box/queue
is the ultimate endpoint, it lives in a discrete location, and you
have to go to your local post office and consume the mail out of it.
Your load balancer will be of little use for consumers, as your
consumers must be connecting to the node on which your queues reside.
Load balancing can be used for your producers though, in cases where
they are easily separated.
I've heard redirect is removed from later specifications. Regardless,
I recommend turning the "insist" flag on and avoiding redirects
altogether, because they're inconsistent with the way AMQP clustering
works.
I hope that gives you a good start.
-Aaron
--
Aaron Westendorf
Senior Software Engineer
Agora Games
359 Broadway
Troy, NY 12180
Phone: 518.268.1000
aa...@agoragames.com
www.agoragames.com
Dave Greggory
Dave,
-Aaron
>?
>
>
> Thanks so much,
> Dave
--
Aaron Westendorf
Senior Software Engineer
Agora Games
359 Broadway
Troy, NY 12180
Phone: 518.268.1000
aa...@agoragames.com
www.agoragames.com
Alexandru Scvortov
Hi Dave,
> Questions:
> 2. When a queue is declared (by one of our producers), is the queue always
> created on the rabbitmq node the producer client has a connection to or is the
> queue created on a randomly selected node within the cluster?
The queue is created only on that one node, but is accessible from all
nodes in the cluster. All nodes in the same cluster are functionally
interchangeable, i.e. it doesn't matter which of the nodes you connect to,
you can still use the queue.
> 3. If the queue is durable and the messages sent to it are marked persistent,
> will these messages always be persisted to disk and be available after a restart
> of the node that has that queue, regardless of whether the node is a disk node
> or RAM node? (This line "Should you do this, and suffer a power
> failure to the entire cluster, the entire state of the cluster,
> including all messages, will be lost. " in
> http://www.rabbitmq.com/clustering.html is confusing)
The distinction between RAM and disk nodes refers to where the Mnesia
tables are stored. The routing configuration (what queues there are,
which exchanges they're bound to, etc.) is stored in Mnesia, so at least one
of the nodes should be a disk node. This way, if both nodes go down
at the same time (power failure), when they're restarted, the cluster
should spring back to life just like before.
Messages are persisted to disk, regardless of whether the node is a
disc node or not. Also, messages are persisted to disk only on the node
on which the queue was declared. If said node is a RAM node and it goes
down, it won't lose the messages; when it reconnects to the cluster, the
queue should just get recreated (assuming it was declared as durable).
If both nodes are RAM nodes, and they both go down, when they're
restarted, they won't know what queues were declared, etc. So, in this
case, the messages will be lost.
> 4. Should I configure both my nodes as disk nodes or will one disk node be
> sufficient? In other words, if only 1 disk node was there in the cluster and its
> hard drive went bust, what can I recover from the RAM node? If nothing can be
> recovered from the RAM node
As mentioned above, the RAM node will not store the routing configuration to disk.
If the disc node goes down, you should still be able to recover the broker
configuration from it by adding a new disc node to the cluster.
If any of the nodes goes down permanently (disk failure, etc.), the
messages stored on it will be lost. If you don't want this to happen,
have a look at the HA Clustering guide:
http://www.rabbitmq.com/pacemaker.html
> is it mainly for increasing the number of
> connections without taking any hits to disk throughput?
Pretty much, yes.
> 5. Are connections redirects actually supported by the current version? The FAQ
> and Clustering documents on site are contradictory of each other. (FAQ says
> "Future releases will support live failover using, for instance, a
> combination of the "known hosts" field in connection.open-ok and the
> connection.redirect message.")
> 6. If redirects are supported, when all connections are being sent to a specific
> rabbitmq node in the cluster by the loadbalancer, will that rabbitmq node still
> send a redirect request to the client if it's getting too taxed? If so, then is
> it possible to limit redirects to only the disk nodes within the cluster so that
> we don't lose any data?
> 7. Will the connections be redirected solely based on RabbitMQ node's ability to
> serve it or is it more round-robin?
There is currently some loadbalancing done via AMQP's connection.redirect
methods.
Both redirects and loadbalancing will soon be removed from RabbitMQ. It
would probably be better if you didn't count on this feature.
> 8. Is there a way to ensure that the cluster configuration is correct because
> 'disc/RAM' is not reported correctly by rabbitmqctl (as per
> http://old.nabble.com/Rabbitmq-v.-1.8.1---Bug-report---Could-not-start-a-node-as-RAM-node-in-cluster-ts29211668.html)?
Not in the current release, no. It's been fixed on the develpment branch
and should make it into the next release.
Hope this clears things up.
Cheers,
Alex
Alexandru Scvortov
On Wed, 21 Jul 2010, Aaron Westendorf wrote:
Aaron,
> I intend to write up a blog post on our cluster setup which will
> hopefully address some of your questions.
That's great! Can't wait to read it.
> Think of it as the postal service, where exchanges are your
> street-corner deposit boxes and office tellers, and the postal service
> is the cluster itself. Your address is your routing scheme, and that
> routing key will deliver a message to your postal box. The box/queue
> is the ultimate endpoint, it lives in a discrete location, and you
> have to go to your local post office and consume the mail out of it.
>
> Your load balancer will be of little use for consumers, as your
> consumers must be connecting to the node on which your queues reside.
That's not quite true. Clients can consume the messages from any of the
nodes. The messages reside only on the node on which their queue was
declared, but they're routed through the cluster to the consumer that
needs them.
> I've heard redirect is removed from later specifications.
Yes, it has.
Cheers,
Alex
Aaron Westendorf
<alex...@rabbitmq.com> wrote:
>
> The queue is created only on that one node, but is accessible from all
> nodes in the cluster. All nodes in the same cluster are functionally
> interchangeable, i.e. it doesn't matter which of the nodes you connect to,
> you can still use the queue.
Interesting, I was not aware of this subtlety which is why I went with
the PO analogy when describing how our cluster works. So I can
connect to any host in the cluster, call basic.consume(), and receive
messages from a queue that resides on another host? What about the
case where multiple clients start at the same time and they each
declare the queue and its bindings (identically)? I swear that we
tried this exact setup, maybe in the 1.5 series, and we kept getting
dropped messages. This has a big effect on how one can go about
clustering and failover, and matches our original plan for a generic
pool of rabbit hosts.
We have very wide pipes between our hosts, but there's still overhead
in handling inter-node traffic. Is there an optimal setup, such as
all consumers of a queue connected to the host on which it resides?
-Aaron
--
Aaron Westendorf
Senior Software Engineer
Agora Games
359 Broadway
Troy, NY 12180
Phone: 518.268.1000
aa...@agoragames.com
www.agoragames.com
Aaron Westendorf
Including the list on the reply.
On Wed, Jul 21, 2010 at 5:44 PM, Dave Greggory <davegr...@yahoo.com> wrote:
>
> Your postal service analogy is good one and that is how I understood it mostly.
> From what I understood, if a client requests a message from a queue that is not
> on the node he has the current connection to, because of clustering, you will
> get the message back indirectly. But your analogy makes it sound like the client
> will actually need to establish a second connection to the node that has the
> queue. I'm fine with that, that's not a big deal.
I may be wrong in this, Alexandru hinted that clustering might be more
of flexible than I described.
> I declare my (durable) queues from the both producer and consumer ends because
> exchanges would just drop the messages if we left it up to consumers and no
> consumers were up at the time a producer sends message. So But I'm trying to
> understand where a queue will be created when it's declared (will it be in the
> node the client has the connection open to or could it end up existing in
> another node)?
We have run in to that exact chicken-and-egg problem. We don't have a
need for persistent messages before a consumer starts for the first
time, so we've been able to allow the consumer to drive the queue
declare and drop unrouteable messages. If that doesn't work for you,
then perhaps you can have something that sits outside your apps and
ensures that all queues are declared and in good health. We've taken
to adding a several monitoring tools because it's so easy and it adds
to our reliability.
The queue will exist on the host to which you are connected when you declare it.
> I may have misused the term load-balancer in this context (just because that's
> what we use that piece of hardware most often). Let's just call it node-switcher
> for this purpose. I intended it to serve two purposes,
> a) Hide the hostnames/ip-addresses of rabbitmq nodes from producers and clients
> b) Serve as switch for determining which rabbitmq nodes consumers/producers
> connect to, so that we can:
> i. Switch all new consumer/producer connections automatically to the other
> node if the first node goes down
>
> ii. Upgrade rabbitmq nodes without affecting producer/consumer nodes
> because we can switch out different rabbitmq nodes
We've solved this problem with DNS aliases. It's not automated by
design but it does allow us to cleanly disconnect nodes and have the
clients reconnect to another. The trick there is that if a queue
existed on a node that is shutdown, the clients have to also declare
it, its bindings and the consumers again. This was best accomplished
for us by simply killing the clients off on a Rabbit disconnect and
letting the "hypervisor" restart them.
That only worked because all consumers of the queue were connected to
the same host on which the queue resided. If Alexandru is correct and
the cluster can be treated as a pool, then the reconnect scenario has
to include those other clients re-initializing their consumer calls to
the new queue, after it is re-declared. So perhaps it is best to
always connect to the node hosting the queue you want to consume from.
cheers,
Aaron
Dave Greggory
Off-topic: Why isn't the reply-address set to the mailing address
On Wed Jul 21st, 2010 5:44 PM EDT Dave Greggory wrote:
>Thanks for the reply, Aaron. I look forward to reading your blog post.
>
>Your postal service analogy is good one and that is how I understood it mostly.
>From what I understood, if a client requests a message from a queue that is not
>on the node he has the current connection to, because of clustering, you will
>get the message back indirectly. But your analogy makes it sound like the client
>will actually need to establish a second connection to the node that has the
>queue. I'm fine with that, that's not a big deal.
>
>
>I declare my (durable) queues from the both producer and consumer ends because
>exchanges would just drop the messages if we left it up to consumers and no
>consumers were up at the time a producer sends message. So But I'm trying to
>understand where a queue will be created when it's declared (will it be in the
>node the client has the connection open to or could it end up existing in
>another node)?
>
>
>I may have misused the term load-balancer in this context (just because that's
>what we use that piece of hardware most often). Let's just call it node-switcher
>for this purpose. I intended it to serve two purposes,
>a) Hide the hostnames/ip-addresses of rabbitmq nodes from producers and clients
>b) Serve as switch for determining which rabbitmq nodes consumers/producers
>connect to, so that we can:
> i. Switch all new consumer/producer connections automatically to the other
>node if the first node goes down
>
> ii. Upgrade rabbitmq nodes without affecting producer/consumer nodes
>because we can switch out different rabbitmq nodes
>
>If rabbitmq goes down on 1 node and the node-switches switches out that node,
>I'm okay with queues on the downed node not being available. But would a new
>queue be automatically be created on the switched-in node without having to be
>declared? I'm hoping that because it was part of the cluster before the first
>node went down, it knew that a certain queue existed on the other node and since
>the first node is no longer available, it will take over receiving messages for
>that queue from producers? If that's not true, how do people handle that
>scenario?
>
>Thanks for the heads up on not using redirects.
>
>Dave
>
>
>----- Original Message ----
>From: Aaron Westendorf <aa...@agoragames.com>
>To: Dave Greggory <davegr...@yahoo.com>
>Cc: rabbitmq...@lists.rabbitmq.com
>Sent: Wed, July 21, 2010 5:02:46 PM
>Subject: Re: [rabbitmq-discuss] RabbitMQ production setup questions around
>clustering
>
>Dave,
>
>I intend to write up a blog post on our cluster setup which will
>hopefully address some of your questions. You're trying to set up
>your cluster much the way we originally did, but it's not quite the
>way Rabbit or AMQP operate.
>
>Think of it as the postal service, where exchanges are your
>street-corner deposit boxes and office tellers, and the postal service
>is the cluster itself. Your address is your routing scheme, and that
>routing key will deliver a message to your postal box. The box/queue
>is the ultimate endpoint, it lives in a discrete location, and you
>have to go to your local post office and consume the mail out of it.
>
>Your load balancer will be of little use for consumers, as your
>consumers must be connecting to the node on which your queues reside.
>> 2. When a queue is declared (by one of our producers), is the queue always
>> created on the rabbitmq node the producer client has a connection to or is the
>> queue created on a randomly selected node within the cluster?
>> 3. If the queue is durable and the messages sent to it are marked persistent,
>> will these messages always be persisted to disk and be available after a
>>restart
>> of the node that has that queue, regardless of whether the node is a disk node
>> or RAM node? (This line "Should you do this, and suffer a power
>> failure to the entire cluster, the entire state of the cluster,
>> including all messages, will be lost. " in
>> http://www.rabbitmq.com/clustering.html is confusing)
>> 4. Should I configure both my nodes as disk nodes or will one disk node be
>> sufficient? In other words, if only 1 disk node was there in the cluster and
>>its
>> hard drive went bust, what can I recover from the RAM node? If nothing can be
>> recovered from the RAM node, is it mainly for increasing the number of
>> connections without taking any hits to disk throughput?
>> 5. Are connections redirects actually supported by the current version? The
>FAQ
>> and Clustering documents on site are contradictory of each other. (FAQ says
>> "Future releases will support live failover using, for instance, a
>> combination of the "known hosts" field in connection.open-ok and the
>> connection.redirect message.")
>> 6. If redirects are supported, when all connections are being sent to a
>>specific
>> rabbitmq node in the cluster by the loadbalancer, will that rabbitmq node
>still
>> send a redirect request to the client if it's getting too taxed? If so, then
>is
>> it possible to limit redirects to only the disk nodes within the cluster so
>>that
>> we don't lose any data?
>>
>> 7. Will the connections be redirected solely based on RabbitMQ node's ability
>>to
>> serve it or is it more round-robin?
>> 8. Is there a way to ensure that the cluster configuration is correct because
>> 'disc/RAM' is not reported correctly by rabbitmqctl (as per
>>http://old.nabble.com/Rabbitmq-v.-1.8.1---Bug-report---Could-not-start-a-node-as-RAM-node-in-cluster-ts29211668.html)?
>>?
>>
>>
>> Thanks so much,
>> Dave
>
>
>
Matthias Radestock
> Off-topic: Why isn't the reply-address set to the mailing address
We stick to mailman's default settings here. It's a religious issue. See
http://www.gnu.org/software/mailman/mailman-admin/node11.html
Matthias.
Dave Greggory
Looks like I will be setting up a 2 node cluster with both nodes as disc nodes (as that's the only way to recover from a permanent failure of a disc node). I will have clients connect through the node-switcher/load-balancer with the insist flag on.
On Thu Jul 22nd, 2010 8:05 AM EDT Alexandru Scvortov wrote:
>
>Hi Dave,
>
>> Questions:
>> 2. When a queue is declared (by one of our producers), is the queue always
>> created on the rabbitmq node the producer client has a connection to or is the
>> queue created on a randomly selected node within the cluster?
>
>The queue is created only on that one node, but is accessible from all
>nodes in the cluster. All nodes in the same cluster are functionally
>interchangeable, i.e. it doesn't matter which of the nodes you connect to,
>you can still use the queue.
>
>> 3. If the queue is durable and the messages sent to it are marked persistent,
>> will these messages always be persisted to disk and be available after a restart
>> of the node that has that queue, regardless of whether the node is a disk node
>> or RAM node? (This line "Should you do this, and suffer a power
>> failure to the entire cluster, the entire state of the cluster,
>> including all messages, will be lost. " in
>> http://www.rabbitmq.com/clustering.html is confusing)
>
>The distinction between RAM and disk nodes refers to where the Mnesia
>tables are stored. The routing configuration (what queues there are,
>which exchanges they're bound to, etc.) is stored in Mnesia, so at least one
>of the nodes should be a disk node. This way, if both nodes go down
>at the same time (power failure), when they're restarted, the cluster
>should spring back to life just like before.
>
>Messages are persisted to disk, regardless of whether the node is a
>disc node or not. Also, messages are persisted to disk only on the node
>on which the queue was declared. If said node is a RAM node and it goes
>down, it won't lose the messages; when it reconnects to the cluster, the
>queue should just get recreated (assuming it was declared as durable).
>
>If both nodes are RAM nodes, and they both go down, when they're
>restarted, they won't know what queues were declared, etc. So, in this
>case, the messages will be lost.
>
>> 4. Should I configure both my nodes as disk nodes or will one disk node be
>> sufficient? In other words, if only 1 disk node was there in the cluster and its
>> hard drive went bust, what can I recover from the RAM node? If nothing can be
>> recovered from the RAM node
>
>As mentioned above, the RAM node will not store the routing configuration to disk.
>If the disc node goes down, you should still be able to recover the broker
>configuration from it by adding a new disc node to the cluster.
>
>If any of the nodes goes down permanently (disk failure, etc.), the
>messages stored on it will be lost. If you don't want this to happen,
>have a look at the HA Clustering guide:
>
> http://www.rabbitmq.com/pacemaker.html
>
>> is it mainly for increasing the number of
>> connections without taking any hits to disk throughput?
>
>Pretty much, yes.
>
>> 5. Are connections redirects actually supported by the current version? The FAQ
>> and Clustering documents on site are contradictory of each other. (FAQ says
>> "Future releases will support live failover using, for instance, a
>> combination of the "known hosts" field in connection.open-ok and the
>> connection.redirect message.")
>> 6. If redirects are supported, when all connections are being sent to a specific
>> rabbitmq node in the cluster by the loadbalancer, will that rabbitmq node still
>> send a redirect request to the client if it's getting too taxed? If so, then is
>> it possible to limit redirects to only the disk nodes within the cluster so that
>> we don't lose any data?
>> 7. Will the connections be redirected solely based on RabbitMQ node's ability to
>> serve it or is it more round-robin?
>
>There is currently some loadbalancing done via AMQP's connection.redirect
>methods.
>
>Both redirects and loadbalancing will soon be removed from RabbitMQ. It
>would probably be better if you didn't count on this feature.
>
>> 8. Is there a way to ensure that the cluster configuration is correct because
>> 'disc/RAM' is not reported correctly by rabbitmqctl (as per
>> http://old.nabble.com/Rabbitmq-v.-1.8.1---Bug-report---Could-not-start-a-node-as-RAM-node-in-cluster-ts29211668.html)?
>
>Not in the current release, no. It's been fixed on the develpment branch
>and should make it into the next release.
>
>
>Hope this clears things up.
>
>Cheers,
>Alex
Matthew Sackman
On Thu, Jul 22, 2010 at 08:33:48AM -0400, Aaron Westendorf wrote:
> So I can
> connect to any host in the cluster, call basic.consume(), and receive
> messages from a queue that resides on another host?
Correct. However, the messages are only stored on the node on which the
queue was created, thus latency will be higher due to the extra hop than
connecting to the queue-local node.
> What about the
> case where multiple clients start at the same time and they each
> declare the queue and its bindings (identically)?
By using distributed transactions in mnesia, we detect this case. Also,
queue declaration is idempotent, so the "winner" of the race will indeed
create a new queue. The "loser" of the race will return the queue that
was created by the winner.
> I swear that we
> tried this exact setup, maybe in the 1.5 series, and we kept getting
> dropped messages. This has a big effect on how one can go about
> clustering and failover, and matches our original plan for a generic
> pool of rabbit hosts.
There were some bugs back then, especially with regards to what happens
when a queue is declared that was previously declared on a now-failed
node. That used to be permitted and caused all sorts of problems when
the failed node came back up (if not before). That case is not expressly
forbidden and will result in a 404 (literally - the node on which this
queue has been created cannot be found).
With regards to fail over, it should be noted that clustering is not a
means for HA because of the fact that when a node goes down it takes its
queues with it, and those queues can't be recovered before the node
comes back up.
> We have very wide pipes between our hosts, but there's still overhead
> in handling inter-node traffic. Is there an optimal setup, such as
> all consumers of a queue connected to the host on which it resides?
Yes. Within a cluster, there is no batching of deliveries at least
nothing we're doing explicitly. I don't think the VM does anything magic
underneath but I could be wrong. Thus you may wish to try to ensure that
clients are directly connected to the correct nodes for their queues.
On the publishing side, we do do explicit batching so if you are
publishing a message which ends up going to several queues all on some
other node, then that will only be sent as one message to the other node
rather than N messages.
Matthew