rabbitmq-peer-discovery-aws not working as well

[Firxiao zhang]

Hi Guys:

 I am following http://next.rabbitmq.com/cluster-formation.html configure the rabbitmq cluster on aws. I have configured the IAM role and apply it on the instance. but from log. I see the aws call timeout. is there a way to debug why aws call timeout?   It's appreciate for any help.  thank you.  

 here is my configuration:

 /etc/rabbitmq/rabbitmq.conf 

cluster_formation.peer_discovery_backend = rabbit_peer_discovery_aws
cluster_formation.aws.region = cn-north-1
cluster_formation.aws.instance_tags.region = cn-north-1
cluster_formation.aws.instance_tags.service = rabbitmq
cluster_formation.aws.instance_tags.environment = sandbox
log.file.level = debug

here is the logs:

2018-03-08 08:54:05.160 [info] <0.243.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@ip-10-209-18-85 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2018-03-08 08:54:05.160 [info] <0.243.0> Configured peer discovery backend: rabbit_peer_discovery_aws
2018-03-08 08:54:05.160 [debug] <0.243.0> Peer discovery backend supports initialisation.
2018-03-08 08:54:05.160 [debug] <0.243.0> Peer discovery AWS: initialising...
2018-03-08 08:54:05.160 [debug] <0.243.0> HTTP client proxy is not configured
2018-03-08 08:54:05.160 [debug] <0.243.0> Peer discovery backend initialisation succeeded.
2018-03-08 08:54:05.160 [info] <0.243.0> Will try to lock with peer discovery backend rabbit_peer_discovery_aws
2018-03-08 08:54:05.160 [info] <0.243.0> Peer discovery backend rabbit_peer_discovery_aws does not support registration, skipping randomized startup delay.
2018-03-08 08:54:05.160 [debug] <0.243.0> Started rabbitmq_aws
2018-03-08 08:54:05.160 [debug] <0.243.0> Will use AWS access key of 'undefined'
2018-03-08 08:54:05.160 [debug] <0.243.0> Setting AWS region to "cn-north-1"
2018-03-08 08:54:05.161 [debug] <0.313.0> Supervisor inet_gethost_native_sup started undefined at pid <0.314.0>
2018-03-08 08:54:05.161 [debug] <0.56.0> Supervisor kernel_safe_sup started inet_gethost_native:start_link() at pid <0.313.0>
2018-03-08 08:54:10.167 [error] <0.242.0> CRASH REPORT Process <0.242.0> with 0 neighbours exited with reason: {timeout,{gen_server,call,[rabbitmq_aws,{request,"ec2",get,[],"/?Action=DescribeInstances&Filter.1.Name=tag%3Aenvironment&Filter.1.Value.1=sandbox&Filter.2.Name=tag%3Aregion&Filter.2.Value.1=cn-north-1&Filter.3.Name=tag%3Aservice&Filter.3.Value.1=rabbitmq&Version=2015-10-01",[],[],undefined}]}} in application_master:init/4 line 134
2018-03-08 08:54:10.167 [info] <0.31.0> Application rabbit exited with reason: {timeout,{gen_server,call,[rabbitmq_aws,{request,"ec2",get,[],"/?Action=DescribeInstances&Filter.1.Name=tag%3Aenvironment&Filter.1.Value.1=sandbox&Filter.2.Name=tag%3Aregion&Filter.2.Value.1=cn-north-1&Filter.3.Name=tag%3Aservice&Filter.3.Value.1=rabbitmq&Version=2015-10-01",[],[],undefined}]}

[Michael Klishin]

A GET [AWS endpoint]/?Action=DescribeInstances&Filter.1.Name=tag%3Aenvironment&Filter.1.Value.1=sandbox&Filter.2.Name=tag%3Aregion&Filter.2.Value.1=cn-north-1&Filter.3.Name=tag%3Aservice&Filter.3.Value.1=rabbitmq&Version=2015-10-01



request timed out. Please double check your IAM, VPC and firewall settings.


--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Firxiao zhang]

Thanks for your reply. Now I have figured out the root cause. because the endpoint url is different between china(aws.com.cn) and global(aws.com). so far. I have tested adding the DNS point to china's endpoint IP in /etc/hosts. and it works. also I am reading the code of rabbitmq_aws(https://github.com/gmr/rabbitmq-aws) to see how make it support china aws.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

May I ask for more details? What specifically did you add to /etc/hosts? We'd be happy to document any region-specific differences.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Firxiao zhang]

Here is the document from aws. https://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region 

you'll see china region endpoint is

China (Beijing)

cn-north-1

ec2.cn-north-1.amazonaws.com.cn

the global is

US East (Ohio)

us-east-2

ec2.us-east-2.amazonaws.com

and from the rabbitmq_aws. I found the endpoint url is "amazonaws.com"

so I do a test. get the ip of "ec2.cn-north-1.amazonaws.com.cn" and add "ip ec2.cn-north-1.amazonaws.com" in /etc/hosts. and it works. 

if you guys can let rabbimq_aws support china region. It will be perfect. thanks 

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/-LKiY5TMPPI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

[Michael Klishin]

Yes, we definitely want all AWS regions to work with the minimal amount of hassle for the user.
Thanks for the details, it's very helpful!

I have filed an issue:

https://github.com/rabbitmq/rabbitmq-peer-discovery-aws/issues/23

and we will add a note to the docs to raise awareness about the differences.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/-LKiY5TMPPI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Gavin M. Roy]

I'm assuming the issue relates to https://github.com/gmr/rabbitmq-aws/blob/master/src/rabbitmq_aws.erl#L220

I'm not sure off hand if this is a discoverable value. 

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/-LKiY5TMPPI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Hi Gavin,

Since we already ask the region to be configured, we can use that value

to pick the endpoint per [1]. Does it make sense?

1. https://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region

[Gavin M. Roy]

Well I was hoping for a programatic way to figure this out via something like the EC2 metadata API, but couldn't find anything.

botocore has a json file that's distributed with the library that sets the dnssuffix (tld) based upon the region. It appears that this is only done for China. (https://github.com/boto/botocore/blob/7d098b9c2f60bcbb6b1ef72e7e2b83531caaac7d/botocore/data/endpoints.json#L1978)

As inelegant as it is, do we just hardcode an exception for China?

Gavin

[Michael Klishin]

Given that it’s a very rarely changing set of values, I’m happy to add a lookup table similar to botocore, even if it’s just for China at the moment.

Should AWS introduce a lookup endpoint (which I think is unlikely, since there’s a chicken-and-egg problem with endpoint discovery), we can switch to use that.

[Firxiao zhang]

Thank you all your guys work. It is very much appreciated.

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/-LKiY5TMPPI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/-LKiY5TMPPI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

[Michael Klishin]

So we added a special case for the Chinese regions and it will be using .com.cn instead of .com

for API endpoint TLD.

It will ship in 3.7.5.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/-LKiY5TMPPI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/-LKiY5TMPPI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Firxiao zhang]

Cool.  Thanks again

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/-LKiY5TMPPI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/-LKiY5TMPPI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/-LKiY5TMPPI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.