RabbitMQ peer discovery aws autoscaling

di...@healthqtech.com

unread,

Apr 5, 2018, 8:38:35 AM4/5/18

to rabbitmq-users

Hi,

Hoping some one can point me in the correct direction as how to continue debugging my issue.

I get this error when trying to use AWS auto cluster via the peer discovery aws plugin: I am trying to cluster from my autoscaling group.

2018-04-05 12:20:51.683 [debug] <0.760.0> AWS request: /?Action=DescribeAutoScalingInstances&Version=2011-01-01
Response: [{"DescribeAutoScalingInstancesResponse",[{"DescribeAutoScalingInstancesResult",[{"AutoScalingInstances",[]}]},{"ResponseMetadata",[{"RequestId","c7cdcd91-38cb
-11e8-89fd-875bbb62dd45"}]}]}]
2018-04-05 12:20:51.683 [warning] <0.760.0> Cannot discover any nodes because no AWS autoscaling group could be found in the instance description. Make sure that this in
stance belongs to an autoscaling group.

However when i query the host via the AWS EC2 cli from the local rabbitmq system I see the following:

aws ec2 describe-instances --instance-ids i-xxxxxxxxxxxx

{

"Value": "asg-rabbitmq-cluster",

"Key": "aws:autoscaling:groupName"

},

I have the following attached to my IAM policy for this host

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"ec2:DescribeInstances"
],
"Resource": [
"*"
]
}
]
}

My rabbit.conf looks like this:

loopback_users = none

cluster_formation.peer_discovery_backend = rabbit_peer_discovery_aws

cluster_formation.aws.region = us-west-2

cluster_formation.aws.access_key_id = xxxxxxxxx

cluster_formation.aws.secret_key =xxxxxxxxx

cluster_formation.aws.use_autoscaling_group = true

cluster_formation.aws.use_private_ip = true

log.file.level = debug

My rabbit-env file looks like this:

RABBITMQ_USE_LONGNAME=true
HOME=/var/lib/rabbitmq
LOG_BASE=/var/log/rabbitmq
MNESIA_BASE=/var/lib/rabbitmq/mnesia

My rabbitmqctl status output

Status of node rab...@ip-x.x.x.x.us-west-2.compute.internal ...

[{pid,30700},

{running_applications,

[{rabbitmq_peer_discovery_aws,

"AWS-based RabbitMQ peer discovery backend","3.7.4"},

{rabbitmq_federation_management,"RabbitMQ Federation Management",

"3.7.4"},

{rabbitmq_shovel_management,

"Management extension for the Shovel plugin","3.7.4"},

{rabbitmq_top,"RabbitMQ Top","3.7.4"},

{rabbitmq_management,"RabbitMQ Management Console","3.7.4"},

{rabbitmq_management_agent,"RabbitMQ Management Agent","3.7.4"},

{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.7.4"},

{rabbitmq_federation,"RabbitMQ Federation","3.7.4"},

{rabbitmq_shovel,"Data Shovel for RabbitMQ","3.7.4"},

{rabbitmq_sharding,"RabbitMQ Sharding Plugin","3.7.4"},

{rabbitmq_amqp1_0,"AMQP 1.0 support for RabbitMQ","3.7.4"},

{rabbitmq_stomp,"RabbitMQ STOMP plugin","3.7.4"},

{amqp_client,"RabbitMQ AMQP Client","3.7.4"},

{rabbitmq_peer_discovery_common,

"Modules shared by various peer discovery backends","3.7.4"},

{rabbit,"RabbitMQ","3.7.4"},

{mnesia,"MNESIA CXC 138 12","4.15.3"},

{rabbitmq_aws,

"A minimalistic AWS API interface used by rabbitmq-autocluster (3.6.x) and other RabbitMQ plugins",

"3.7.4"},

{rabbit_common,

"Modules shared by rabbitmq-server and rabbitmq-erlang-client",

"3.7.4"},

{cowboy,"Small, fast, modern HTTP server.","2.2.2"},

{amqp10_client,"AMQP 1.0 client from the RabbitMQ Project","3.7.4"},

{cowlib,"Support library for manipulating Web protocols.","2.1.0"},

{amqp10_common,

"Modules shared by rabbitmq-amqp1.0 and rabbitmq-amqp1.0-client",

"3.7.4"},

{os_mon,"CPO CXC 138 46","2.4.4"},

{lager,"Erlang logging framework","3.5.1"},

{ranch_proxy_protocol,"Ranch Proxy Protocol Transport","1.4.4"},

{ranch,"Socket acceptor pool for TCP protocols.","1.4.0"},

{ssl,"Erlang/OTP SSL application","8.2.4"},

{public_key,"Public key infrastructure","1.5.2"},

{asn1,"The Erlang ASN1 compiler version 5.0.5","5.0.5"},

{recon,"Diagnostic tools for production use","2.3.2"},

{jsx,"a streaming, evented json parsing toolkit","2.8.2"},

{crypto,"CRYPTO","4.2.1"},

{xmerl,"XML parser","1.3.16"},

{inets,"INETS CXC 138 49","6.5"},

{goldrush,"Erlang event stream processor","0.1.9"},

{compiler,"ERTS CXC 138 10","7.1.5"},

{syntax_tools,"Syntax tools","2.1.4"},

{sasl,"SASL CXC 138 11","3.1.1"},

{stdlib,"ERTS CXC 138 10","3.4.4"},

{kernel,"ERTS CXC 138 10","5.4.3"}]},

{os,{unix,linux}},

{erlang_version,

"Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:64] [hipe] [kernel-poll:true]\n"},

{memory,

[{connection_readers,0},

{connection_writers,0},

{connection_channels,0},

{connection_other,2840},

{queue_procs,0},

{queue_slave_procs,0},

{plugins,3278880},

{other_proc,22165752},

{metrics,198352},

{mgmt_db,163920},

{mnesia,75800},

{other_ets,2578328},

{binary,115536},

{msg_index,30960},

{code,30243182},

{atom,1172689},

{other_system,10055785},

{allocated_unused,16781848},

{reserved_unallocated,8716288},

{strategy,rss},

{total,[{erlang,70082024},{rss,95580160},{allocated,86863872}]}]},

{alarms,[]},

{listeners,

[{clustering,25672,"::"},

{amqp,5672,"::"},

{stomp,61613,"::"},

{http,15672,"::"}]},

{vm_memory_calculation_strategy,rss},

{vm_memory_high_watermark,0.4},

{vm_memory_limit,838308659},

{disk_free_limit,50000000},

{disk_free,48734334976},

{file_descriptors,

[{total_limit,924},{total_used,2},{sockets_limit,829},{sockets_used,0}]},

{processes,[{limit,1048576},{used,413}]},

[{total_limit,924},{total_used,2},{sockets_limit,829},{sockets_used,0}]},
{processes,[{limit,1048576},{used,413}]},
{run_queue,0},
{uptime,967},
{kernel,{net_ticktime,60}}]

I have manually clustered the two machines as a test and that works fine. Has anyone experienced this issue before? If so how did you resolve it?

Thanks so much in advance.

Dino

Michael Klishin

unread,

Apr 5, 2018, 6:25:11 PM4/5/18

to rabbitm...@googlegroups.com

What you need to do is enable debug logging and see the logs,

There's a certain chance it could be https://github.com/rabbitmq/rabbitmq-peer-discovery-aws/issues/20.

3.7.5 should be released hopefully next week (but don't quote me on that).

© 2018 HealthQ Technologies. All Rights Reserved. The information contained in this e-mail is confidential - please do not cross-post. This communication is intended only for the use of the addressee(s) and may be a lawyer-client communication and, as such, it is privileged and confidential. If you are not the intended recipient, you are hereby notified that any review, reliance, disclosure, distribution or copying of this communication may be prohibited by law. If you have received this communication in error, please notify us immediately at ad...@healthqtech.com and delete it and all copies from your system.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

di...@healthqtech.com

unread,

Apr 6, 2018, 4:08:53 AM4/6/18

to rabbitmq-users

Thanks for the response, as you can see from the rabbit config i already have debug logging enabled.

I will take a look further and see if i come across anything more.

thanks

DIno

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Reply all

Reply to author

Forward