OK, I realize this is probably a very basic thing, but I've been googling and screwing around with this for hours and getting nowhere.
I'm trying to set up my first rabbitmq cluster. I have it installed fine on two ec2 hosts, and I'm trying to get one to cluster with the other. I've tried the instructions at
Error: unable to connect to node 'rabbit@rabbitmq-small02-dev': nodedown diagnostics: - nodes and their ports on rabbitmq-small02-dev: [{rabbit,57641},
{rabbitmqctl12928,46801}] - current node: 'rabbitmqctl12928@rabbitmq-small02-dev' - current node home dir: /var/lib/rabbitmq - current node cookie hash: 4IHR1JMCSCh55g/tisBydg==
So I found more detailed instructions on skaag.net, which seems to be down, but I can get the google-cached instructions at
So, I install (actually re-install) a fresh, up to date rabbitmq on the second machine, according to the instructions for ec2.
Then I go through the instructions.
I stop the server: [jstoner@rabbitmq-small02-dev]> service rabbitmq-server stop Stopping rabbitmq-server: RabbitMQ is not running rabbitmq-server.
I change the cookie to the same as the other one: [jstoner@rabbitmq-small02-dev]> sudo su root@rabbitmq-small02-dev:/home/jstoner/rclient# echo -n [cookie-text]
node with name "rabbit" already running on "rabbitmq-small02-dev" diagnostics: - nodes and their ports on rabbitmq-small02-dev: [{rabbit,57641},
{rabbitmqctl12869,39028},
{rabbitmqprelaunch12879, 37230}] - current node: 'rabbitmqprelaunch12879@rabbitmq-small02-dev' - current node home dir: /var/lib/rabbitmq - current node cookie hash: 4IHR1JMCSCh55g/tisBydg== [jstoner@rabbitmq-small02-dev]> more /var/log/rabbitmq/startup_err [empty file]
That tells me nothing I can make sense of. So I see what happens if I try the next step: [jstoner@rabbitmq-small02-dev]> sudo rabbitmqctl stop_app Stopping node 'rabbit@rabbitmq-small02-dev' ... Error: unable to connect to node 'rabbit@rabbitmq-small02-dev': nodedown diagnostics: - nodes and their ports on rabbitmq-small02-dev: [{rabbit,57641},
{rabbitmqctl12928,46801}] - current node: 'rabbitmqctl12928@rabbitmq-small02-dev' - current node home dir: /var/lib/rabbitmq - current node cookie hash: [hashtext]
And I get the same kind of problem I got on the first set of instructions. Any ideas would be much appreciated. _______________________________________________ rabbitmq-discuss mailing list rabbitmq-disc...@lists.rabbitmq.com https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
OK, I realize this is probably a very basic thing, but I've been googling and screwing around with this for hours and getting nowhere.
I'm trying to set up my first rabbitmq cluster. I have it installed fine on two ec2 hosts, and I'm trying to get one to cluster with the other. I've tried the instructions at
Error: unable to connect to node 'rabbit@rabbitmq-small02-dev': nodedown diagnostics: - nodes and their ports on rabbitmq-small02-dev: [{rabbit,57641},
{rabbitmqctl12928,46801}] - current node: 'rabbitmqctl12928@rabbitmq-small02-dev' - current node home dir: /var/lib/rabbitmq - current node cookie hash: 4IHR1JMCSCh55g/tisBydg==
So I found more detailed instructions on skaag.net, at
So, I install (actually re-install) a fresh, up to date rabbitmq on the second machine, according to the instructions for ec2.
Then I go through the instructions.
I stop the server: [jstoner@rabbitmq-small02-dev]> service rabbitmq-server stop Stopping rabbitmq-server: RabbitMQ is not running rabbitmq-server.
I change the cookie to the same as the other one: [jstoner@rabbitmq-small02-dev]> sudo su root@rabbitmq-small02-dev:/home/jstoner/rclient# echo -n [cookie-text]
node with name "rabbit" already running on "rabbitmq-small02-dev" diagnostics: - nodes and their ports on rabbitmq-small02-dev: [{rabbit,57641},
{rabbitmqctl12869,39028},
{rabbitmqprelaunch12879, 37230}] - current node: 'rabbitmqprelaunch12879@rabbitmq-small02-dev' - current node home dir: /var/lib/rabbitmq - current node cookie hash: 4IHR1JMCSCh55g/tisBydg== [jstoner@rabbitmq-small02-dev]> more /var/log/rabbitmq/startup_err [empty file]
That tells me nothing I can make sense of. So I see what happens if I try the next step: [jstoner@rabbitmq-small02-dev]> sudo rabbitmqctl stop_app Stopping node 'rabbit@rabbitmq-small02-dev' ... Error: unable to connect to node 'rabbit@rabbitmq-small02-dev': nodedown diagnostics: - nodes and their ports on rabbitmq-small02-dev: [{rabbit,57641},
{rabbitmqctl12928,46801}] - current node: 'rabbitmqctl12928@rabbitmq-small02-dev' - current node home dir: /var/lib/rabbitmq - current node cookie hash: [hashtext]
Which is the same kind of problem I got on the first set of instructions, even though the cookies match.
> Error: unable to connect to node 'rabbit@rabbitmq-small02-dev': > nodedown > diagnostics: > - nodes and their ports on rabbitmq-small02-dev: [{rabbit,57641},
Does the RabbitMQ logfile on rabbitmq-small02-dev contain an entry about a connection attempt from a disallowed node at this time?
> I stop the server: > [jstoner@rabbitmq-small02-dev]> service rabbitmq-server stop > Stopping rabbitmq-server: RabbitMQ is not running > rabbitmq-server.
Subsequent commands indicate that RabbitMQ was still running after this command was issued. Check if you have any processes running starting with "beam". Subsequent steps won't make any difference until you succeed in restarting the broker. Kill the process if necessary.
> {rabbitmqctl12928,46801}] > - current node: 'rabbitmqctl12928@rabbitmq-small02-dev' > - current node home dir: /var/lib/rabbitmq > - current node cookie hash: [hashtext]
What is [hashtext] ? Does it match the other node?
Also check the EC2-specific instructions at http://www.rabbitmq.com/ec2.html paying close attention to the section on hostnames.
> OK, I don't think it's the firewall. 4369 is open to TCP (we tried > adding UDP. Didn't help). Plus when I do a tcpdump, I see packets > received on small01 on 'rabbitmqctl cluster rabbitmq-small01-dev'.
The EPMD port is necessary but not sufficient. From the details provided in your previous email you would have needed to open port 57641 also. This is a random port. Do you see packets on this port in your network trace?
I would suggest temporarily eliminating the firewall entirely for debugging purposes. See the firewall section on the clustering page for configuration details:
On Thu, Mar 15, 2012 at 7:00 PM, John Stoner <johnston...@gmail.com> wrote: > OK, I'm checking about deactivating the firewall with the people working > with me on this.
> Following your link, I see
> Once a distributed Erlang node address has been resolved via epmd, other > nodes will attempt to communicate directly with that address using the > Erlang distributed node protocol. The port range for this communication can > be configured with two parameters for the Erlang kernel application:
> - inet_dist_listen_min > - inet_dist_listen_max
> Firewalls must permit traffic in this range to pass between clustered > nodes (assuming all nodes use the same port range). The default port range > is unrestricted.
> The active configuration can be verified in the startup banner, e.g. the > active configuration file:
> config file(s) : /etc/rabbitmq/rabbitmq.config
> which I'm not seeing. I have an empty /etc/rabbitmq, and when I stop and > restart rabbitmq I don't see any mention of a config file.
> On Thu, Mar 15, 2012 at 5:59 PM, Emile Joubert <em...@rabbitmq.com> wrote:
>> Hi John,
>> On 15/03/12 20:54, John Stoner wrote: >> > OK, I don't think it's the firewall. 4369 is open to TCP (we tried >> > adding UDP. Didn't help). Plus when I do a tcpdump, I see packets >> > received on small01 on 'rabbitmqctl cluster rabbitmq-small01-dev'.
>> The EPMD port is necessary but not sufficient. From the details provided >> in your previous email you would have needed to open port 57641 also. >> This is a random port. Do you see packets on this port in your network >> trace?
>> I would suggest temporarily eliminating the firewall entirely for >> debugging purposes. See the firewall section on the clustering page for >> configuration details:
[Just a bit of context, some of this conversation happened off list--I'm trying to start a rabbitmq cluster on some firewalled EC2 instances, and Emile has been helping me identify all the details about ports and configuration and all that good stuff.]
We're looking to open fewer ports, not more.* *Is there a minimum we could do? Would one work, or would it break something else?
Also, we have these ports open to all TCP. In the spirit of securing our systems, I guess we could open 4369 only to the IPs of the other machines in the cluster. Is that a good idea? Can you think of more firewall restrictions to add?
On Fri, Mar 16, 2012 at 5:24 AM, Emile Joubert <em...@rabbitmq.com> wrote: > Hi John,
> On 16/03/12 05:41, John Stoner wrote: > > OK, I got it to start with a good config file. what's an appropriate > > range for these port numbers?
> You are free to use any unused port range between 1024 and 65536.
On Fri, Mar 16, 2012 at 18:25, John Stoner <johnston...@gmail.com> wrote: > [Just a bit of context, some of this conversation happened off list--I'm > trying to start a rabbitmq cluster on some firewalled EC2 instances, and > Emile has been helping me identify all the details about ports and > configuration and all that good stuff.]
> We're looking to open fewer ports, not more.* *Is there a minimum we > could do? Would one work, or would it break something else?
> Also, we have these ports open to all TCP. In the spirit of securing our > systems, I guess we could open 4369 only to the IPs of the other machines > in the cluster. Is that a good idea? Can you think of more firewall > restrictions to add?
> On Fri, Mar 16, 2012 at 5:24 AM, Emile Joubert <em...@rabbitmq.com> wrote:
> Hi John,
>> On 16/03/12 05:41, John Stoner wrote: >> > OK, I got it to start with a good config file. what's an appropriate >> > range for these port numbers?
>> You are free to use any unused port range between 1024 and 65536.
I assume that you established the reason for the clustering problems encountered earlier was due to firewall configuration.
On 16/03/12 17:25, John Stoner wrote:
> We're looking to open fewer ports, not more./ /Is there a minimum we > could do? Would one work, or would it break something else?
One port is possible (then inet_dist_listen_min = inet_dist_listen_max), but a small number like 5 is more common. Avoid the ephemeral port range when you make your selection.
> Also, we have these ports open to all TCP. In the spirit of securing > our systems, I guess we could open 4369 only to the IPs of the other > machines in the cluster. Is that a good idea? Can you think of more > firewall restrictions to add?
As discussed previously and above, you need to open at least one port in addition to the one used by the port mapper daemon. You are free to add further firewall restrictions, as long as all clusternodes are accessible from all other clusternodes on the relevant ports, as discussed here:
So what is the considerations on having 5 over 10 or 1? Is there a reasonable criteria that can be used to determine what an appropriate number of ports that should be allocated?
On Fri, Mar 16, 2012 at 12:56 PM, Emile Joubert <em...@rabbitmq.com> wrote:
> Hi John,
> I assume that you established the reason for the clustering problems > encountered earlier was due to firewall configuration.
> On 16/03/12 17:25, John Stoner wrote: > > We're looking to open fewer ports, not more./ /Is there a minimum we > > could do? Would one work, or would it break something else?
> One port is possible (then inet_dist_listen_min = inet_dist_listen_max), > but a small number like 5 is more common. Avoid the ephemeral port range > when you make your selection.
> > Also, we have these ports open to all TCP. In the spirit of securing > > our systems, I guess we could open 4369 only to the IPs of the other > > machines in the cluster. Is that a good idea? Can you think of more > > firewall restrictions to add?
> As discussed previously and above, you need to open at least one port in > addition to the one used by the port mapper daemon. You are free to add > further firewall restrictions, as long as all clusternodes are > accessible from all other clusternodes on the relevant ports, as > discussed here:
mcumi...@narrativescience.com> wrote: > So what is the considerations on having 5 over 10 or 1? Is there a > reasonable criteria that can be used to determine what an appropriate > number of ports that should be allocated?
> On Fri, Mar 16, 2012 at 12:56 PM, Emile Joubert <em...@rabbitmq.com>wrote:
>> Hi John,
>> I assume that you established the reason for the clustering problems >> encountered earlier was due to firewall configuration.
>> On 16/03/12 17:25, John Stoner wrote: >> > We're looking to open fewer ports, not more./ /Is there a minimum we >> > could do? Would one work, or would it break something else?
>> One port is possible (then inet_dist_listen_min = inet_dist_listen_max), >> but a small number like 5 is more common. Avoid the ephemeral port range >> when you make your selection.
>> > Also, we have these ports open to all TCP. In the spirit of securing >> > our systems, I guess we could open 4369 only to the IPs of the other >> > machines in the cluster. Is that a good idea? Can you think of more >> > firewall restrictions to add?
>> As discussed previously and above, you need to open at least one port in >> addition to the one used by the port mapper daemon. You are free to add >> further firewall restrictions, as long as all clusternodes are >> accessible from all other clusternodes on the relevant ports, as >> discussed here:
Does the RabbitMQ logfile on rabbitmq-small01-dev contain an entry about a connection attempt from a disallowed node? If it doesn't then internode network traffic is probably being blocked by a firewall.
You can confirm whether that has taken effect by checking the port numbers returned by "epmd -names".
> seeing traffic on 4369, but not the other ports.
If you don't see other internode traffic on the configured port(s) then then the firewall config needs to be updated.
I suggest independently testing the firewall config with another application such as netcat to get some confidence that it is correct. Alternatively temporarily eliminate the firewall entirely for debugging purposes if possible.
> So what is the considerations on having 5 over 10 or 1? Is there a > reasonable criteria that can be used to determine what an appropriate > number of ports that should be allocated?
Convenience only, I think. It is possible to configure one or as many unused ports as you like.
OK, apparently there was a bit of flakiness in EC2's security. We forced a reapply of the rules by making a change and then undoing it, and now it clusters ok.
On Mon, Mar 19, 2012 at 5:20 AM, Emile Joubert <em...@rabbitmq.com> wrote: > Hi Michael,
> On 16/03/12 18:37, Michael Cumings wrote: > > So what is the considerations on having 5 over 10 or 1? Is there a > > reasonable criteria that can be used to determine what an appropriate > > number of ports that should be allocated?
> Convenience only, I think. It is possible to configure one or as many > unused ports as you like.
