rabbitmq-peer-discovery-aws giving Forbidden and Unauthorised messages

[Mihir Mone]

I am running Ubuntu 18.04 with RabbitMQ 3.7.6.

The peer discovery plugins keeps failing irrespective of what I do. When I do it using autoscaling I get "Forbidden" error and when I do it using instance tags I get "Unauthorised".

I am following the cluster formation guide from https://www.rabbitmq.com/cluster-formation.html. 

Shown below is the log of when setup to detect via autoscaling group:

2018-06-24 04:31:04.560 [info] <0.210.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@sandpit-rabbitmq is empty. Assuming we need to join an existing cluster or initialise from scratch...
2018-06-24 04:31:04.560 [info] <0.210.0> Configured peer discovery backend: rabbit_peer_discovery_aws
2018-06-24 04:31:04.560 [debug] <0.210.0> Peer discovery backend supports initialisation.
2018-06-24 04:31:04.560 [debug] <0.210.0> Peer discovery AWS: initialising...
2018-06-24 04:31:04.560 [debug] <0.210.0> HTTP client proxy is not configured
2018-06-24 04:31:04.560 [debug] <0.210.0> Peer discovery backend initialisation succeeded.
2018-06-24 04:31:04.560 [info] <0.210.0> Will try to lock with peer discovery backend rabbit_peer_discovery_aws
2018-06-24 04:31:04.560 [info] <0.210.0> Peer discovery backend does not support locking, falling back to randomized delay
2018-06-24 04:31:04.560 [info] <0.210.0> Peer discovery backend rabbit_peer_discovery_aws supports registration.
2018-06-24 04:31:04.560 [debug] <0.210.0> Randomized startup delay: configured range is from 5000 to 60000 milliseconds, PRNG pick: 15521...
2018-06-24 04:31:04.560 [info] <0.210.0> Will wait for 15521 milliseconds before proceeding with registration...
2018-06-24 04:31:20.082 [debug] <0.210.0> Started rabbitmq_aws
2018-06-24 04:31:20.082 [debug] <0.210.0> Will use AWS access key of <i have removed this>
2018-06-24 04:31:20.082 [debug] <0.210.0> Setting AWS region to "ap-southeast-2"
2018-06-24 04:31:20.086 [debug] <0.210.0> Setting AWS credentials, access key: <i have removed this>
2018-06-24 04:31:20.086 [debug] <0.210.0> Fetched EC2 instance ID from "http://169.254.169.254/latest/meta-data/instance-id": "i-02d725efc2c6f07cc"
2018-06-24 04:31:20.110 [error] <0.210.0> Error fetching autoscaling group instance list: "Forbidden"
2018-06-24 04:31:20.111 [warning] <0.210.0> Cannot discover any nodes because AWS autoscaling group description API call failed.
2018-06-24 04:31:20.111 [info] <0.210.0> All discovered existing cluster peers:
2018-06-24 04:31:20.111 [info] <0.210.0> Discovered no peer nodes to cluster with
2018-06-24 04:31:20.113 [info] <0.33.0> Application mnesia exited with reason: stopped

Shown below is the log of when setup to detect using instance tags:

2018-06-24 05:11:22.525 [info] <0.210.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@sandpit-rabbitmq is empty. Assuming we need to join an existing cluster or initialise from scratch...
2018-06-24 05:11:22.526 [info] <0.210.0> Configured peer discovery backend: rabbit_peer_discovery_aws
2018-06-24 05:11:22.526 [debug] <0.210.0> Peer discovery backend supports initialisation.
2018-06-24 05:11:22.526 [debug] <0.210.0> Peer discovery AWS: initialising...
2018-06-24 05:11:22.526 [debug] <0.210.0> HTTP client proxy is not configured
2018-06-24 05:11:22.526 [debug] <0.210.0> Peer discovery backend initialisation succeeded.
2018-06-24 05:11:22.526 [info] <0.210.0> Will try to lock with peer discovery backend rabbit_peer_discovery_aws
2018-06-24 05:11:22.526 [info] <0.210.0> Peer discovery backend does not support locking, falling back to randomized delay
2018-06-24 05:11:22.526 [info] <0.210.0> Peer discovery backend rabbit_peer_discovery_aws supports registration.
2018-06-24 05:11:22.526 [debug] <0.210.0> Randomized startup delay: configured range is from 5000 to 60000 milliseconds, PRNG pick: 56029...
2018-06-24 05:11:22.526 [info] <0.210.0> Will wait for 56029 milliseconds before proceeding with registration...
2018-06-24 05:12:18.556 [debug] <0.210.0> Started rabbitmq_aws
2018-06-24 05:12:18.556 [debug] <0.210.0> Will use AWS access key of <i have removed this>
2018-06-24 05:12:18.556 [debug] <0.210.0> Setting AWS region to "ap-southeast-2"
2018-06-24 05:12:18.560 [debug] <0.210.0> Setting AWS credentials, access key: <i have removed this>
2018-06-24 05:12:18.578 [error] <0.210.0> Error fetching node list via EC2 API, request path: /?Action=DescribeInstances&Filter.1.Name=tag%3Aautocluster_name&Filter.1.Value.1=sandpit-rabbitmq-autocluster&Version=2015-10-01, error: "Unauthorized"
2018-06-24 05:12:18.579 [warning] <0.210.0> Cannot discover any nodes because AWS instance description with tags #{"autocluster_name" => "sandpit-rabbitmq-autocluster"} failed
2018-06-24 05:12:18.579 [info] <0.210.0> All discovered existing cluster peers:
2018-06-24 05:12:18.579 [info] <0.210.0> Discovered no peer nodes to cluster with
2018-06-24 05:12:18.580 [info] <0.33.0> Application mnesia exited with reason: stopped

This is my IAM policy:

InstanceRole:
 Type: AWS::IAM::Role
 Properties:
 AssumeRolePolicyDocument:
 Version: "2012-10-17"
 Statement:
 - Effect: Allow
 Principal:
 Service:
 - ec2.amazonaws.com
 - autoscaling.amazonaws.com
 Action:
 - sts:AssumeRole
 Policies:
 - PolicyName: rabbitmq-autocluster
 PolicyDocument:
 Version: "2012-10-17"
 Statement:
 - Effect: Allow
 Action:
 - autoscaling:DescribeAutoScalingGroups
 - autoscaling:DescribeAutoScalingInstances
 - ec2:DescribeInstances
 Resource: "*"

I am doing all this through cloudformation templates not manually through the console.

Can someone please help me resolve this?

[Michael Klishin]

The plugin relies on two EC2 API operations:

 * Listing (describing) instances (covered in http://www.rabbitmq.com/cluster-formation.html#peer-discovery-aws-credentials)

 * Listing autoscaling group members (see https://docs.aws.amazon.com/autoscaling/ec2/userguide/control-access-using-iam.html)

How exactly you set up those permissions for the user/IAM role you use with this plugin is entirely up to you.

Perhaps the second link and one example in it are worth adding to the docs. We'll consider it.

[Michael Klishin]

I now see we already mention the "autoscaling:DescribeAutoScalingInstances" permission

in the docs.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Mihir Mone]

If you see the IAM role policies I have mentioned with the original post, you'll see that I'm already granting

- autoscaling:DescribeAutoScalingInstances
- ec2:DescribeInstances

to all resources for the EC2 instances.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Please start new threads for new questions. The permissions required by AWS peer discovery are documented (and pretty few) [1].

1. https://www.rabbitmq.com/cluster-formation.html

On Mon, Feb 24, 2020 at 7:56 PM Emre gündoğdu <aa.em...@gmail.com> wrote:

Hi Micheal,

Its requried to using aws access key and secret key of aws account ? Instance role which include required policies that have mention in cluster formation document is enough alone to access aws api endpoint ?  

And how often aws peer discovery plugin query aws api to list and check cluster members of current cluster in case of autoscaling group membership ?

24 Haziran 2018 Pazar 16:50:24 UTC+3 tarihinde Mihir Mone yazdı:

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/4fa00591-9c67-4d17-9a7b-f053a7b0c1c0%40googlegroups.com.