Troubleshooting adding a vhost and user on RabbitMQ v3.3.4.

[brendan...@reancloud.com]

Hello,

I am working as a DevOps Engineer on a project so I apologize for this post, but I'm at a loss..

Part of this project is to create an AWS AMI (Amazon Machine Image) from a prebake using Packer and Ansible for the configuration management. During the Packer build process, I can see the Ansible steps being successfully executed to install Erlang, RabbitMQ (v3.3.4), then execute rabbitmqctl commands to add a vhost, add a user, and set permissions for that user on the new vhost. All output from the Packer build is showing these steps are successfully:

  

    aws: TASK: [ems | debug var=rabbitmq_ems_user verbosity=2] *************************

    aws: ok: [127.0.0.1] => {

    aws: "var": {

    aws: "rabbitmq_ems_user": {

    aws: "changed": true,

    aws: "cmd": "/bin/bash /usr/sbin/rabbitmqctl add_vhost testMessaging && /usr/sbin/rabbitmqctl add_user test rabbit && /usr/sbin/rabbitmqctl set_permissions -p testMessaging test '.*' '.*' '.*'",

    aws: "delta": "0:00:00.406674",

    aws: "end": "2016-08-03 15:15:14.019714",

    aws: "invocation": {

    aws: "module_args": "/bin/bash /usr/sbin/rabbitmqctl add_vhost testMessaging && /usr/sbin/rabbitmqctl add_user test rabbit && /usr/sbin/rabbitmqctl set_permissions -p testMessaging test '.*' '.*' '.*'",

    aws: "module_complex_args": {},

    aws: "module_name": "shell"

    aws: },

    aws: "rc": 0,

    aws: "start": "2016-08-03 15:15:13.613040",

    aws: "stderr": "",

    aws: "stdout": "Creating vhost \"testMessaging\" ...\n...done.\nCreating user \"test\" ...\n...done.\nSetting permissions for user \"test\" in vhost \"testMessaging\" ...\n...done.",

    aws: "stdout_lines": [

    aws: "Creating vhost \"testMessaging\" ...",

    aws: "...done.",

    aws: "Creating user \"test\" ...",

    aws: "...done.",

    aws: "Setting permissions for user \"test\" in vhost \"testMessaging\" ...",

    aws: "...done."

    aws: ],

    aws: "warnings": []

    aws: }

    aws: }

    aws: }

Line 6 is the actually command being executed on the RHEL6 box. The extra stuff is from the Ansible debug I've included as part of my troubleshooting & validation.

After the Packer process completes successfully, Packer shutdowns the EC2 instance used and generates an AWS AMI (Amazon Machine Image). During our deploy, we reference the created AMI and launch a new instance from that image, but once I login to the server and attempt to execute rabbitmqctl list_vhsots or rabbitmqctl  list_users neither the testMessaging vhost or test user are found.

Is there a save step I'm missing to ensure these values remain after a RabbitMQ stop/system shutdown?

I'm at my wits end & any help is appreciated.

Thanks,

Brendan

[brendan...@reancloud.com]

Forgot to include output from the launched server:

[root@lpr-634309fd ~]# rabbitmqctl list_users

Listing users ...

guest [administrator]

...done.

[root@lpr-634309fd ~]# rabbitmqctl list_vhosts

Listing vhosts ...

/

...done.

[root@lpr-634309fd ~]#

[Michael Klishin]

Almost certainly it's due to a node name change.

RabbitMQ database directory currently includes a hostname in the node database path but you can make it anything (Cloud Foundry RabbitMQ service does, for example).

See http://rabbitmq.com/configure.html.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Also look for messages about node database initialization from scratch in the logs, modern versions will log when a node starts with a blank data directory.

[brendan...@reancloud.com]

I definitely will have to agree that the node name change is likely the culprit. There's two rabbit logs in /var/log/rabbitmq-server with ip-10-71-3-51.log and the new hostname. 

-rw-r--r--. 1 rabbitmq pivotal 2078 Aug  3 11:17 rabbit@ip-10-71-3-51.log
-rw-r--r--. 1 rabbitmq pivotal 1611 Aug  3 13:03 rabbit@lpr-634309fd.log

I'll confirm this later today or tomorrow & I'll look at standardizing things moving forward.

Thank you for the information!

[brendan...@reancloud.com]

So in my attempt to define a standard NODENAME via /etc/rabbitmq/rabbitmq-env.conf in my prebake to hopefully carry over for deploy time. I noticed a different error when I launched the created AMI.

Prebake:

    aws: TASK: [ems | Create rabbitmq-env.conf file to define RabbitMQ NODE name] ******
    aws: changed: [127.0.0.1]

On the spun up EC2 instance:

[root@ip-10-100-50-119 rabbitmq]# cat /etc/rabbitmq/rabbitmq-env.conf

NODENAME=ems[root@ip-10-100-50-119 rabbitmq]#

[root@ip-10-100-50-119 rabbitmq]# ls -al /var/log/rabbitmq/

total 32

drwxr-x---. 2 rabbitmq pivotal 4096 Aug  4 14:59 .

drwxr-xr-x. 5 root     root    4096 Aug  4 15:06 ..

-rw-r--r--. 1 rabbitmq pivotal 5382 Aug  4 15:07 ems.log

-rw-r--r--. 1 rabbitmq pivotal  825 Aug  4 15:07 ems-sasl.log

-rw-r--r--. 1 root     root       0 Aug  4 14:59 shutdown_err

-rw-r--r--. 1 root     root      60 Aug  4 14:59 shutdown_log

-rw-r--r--. 1 root     root      77 Aug  4 15:07 startup_err

-rw-r--r--. 1 root     root    1675 Aug  4 15:07 startup_log

Startup Log output:

[root@ip-10-100-50-119 rabbitmq]# cat startup_log

              RabbitMQ 3.3.4. Copyright (C) 2007-2014 GoPivotal, Inc.

  ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/

  ##  ##

  ##########  Logs: /var/log/rabbitmq/ems.log

  ######  ##        /var/log/rabbitmq/ems-sasl.log

  ##########

              Starting broker...

BOOT FAILED

===========

Error description:

   {error,{failed_to_cluster_with,['ems@ip-10-71-2-101'],

                                  "Mnesia could not connect to any nodes."}}

Log files (may contain more information):

   /var/log/rabbitmq/ems.log

   /var/log/rabbitmq/ems-sasl.log

Stack trace:

   [{rabbit_mnesia,change_extra_db_nodes,2,[]},

    {rabbit_mnesia,init_db,3,[]},

    {rabbit_mnesia,init_db_and_upgrade,3,[]},

    {rabbit_mnesia,init,0,[]},

    {rabbit,'-run_boot_step/1-lc$^1/1-1-',1,[]},

    {rabbit,run_boot_step,1,[]},

    {rabbit,'-start/2-lc$^0/1-0-',1,[]},

    {rabbit,start,2,[]}]

BOOT FAILED

===========

Error description:

   {could_not_start,rabbit,

       {bad_return,

           {{rabbit,start,[normal,[]]},

            {'EXIT',

                {rabbit,failure_during_boot,

                    {error,

                        {failed_to_cluster_with,

                            ['ems@ip-10-71-2-101'],

                            "Mnesia could not connect to any nodes."}}}}}}}

Log files (may contain more information):

   /var/log/rabbitmq/ems.log

   /var/log/rabbitmq/ems-sasl.log

{"init terminating in do_boot",{rabbit,failure_during_boot,{could_not_start,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{rabbit,failure_during_boot,{error,{failed_to_cluster_with,['ems@ip-10-71-2-101'],"Mnesia could not connect to any nodes."}}}}}}}}}

My assumption is that I need more configuration in order to resolve this, but I'm not 100% sure. 

I don't know if this has any merit to the issue, but as part of our typical boot process we are performing a hostname rename that is a project requirement. AWS also dynamically generates the hostname on spin up.

[Michael Klishin]

This node was clustered with 'ems@ip-10-71-2-101' at the time the image was built
but now 'ems@ip-10-71-2-101' is not reachable.

You probably want to use something like https://github.com/aweber/rabbitmq-autocluster or Chef/Puppet/etc

to form clusters with prebuilt images.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[brendan...@reancloud.com]

I'm not trying to cluster. I'm trying to install RabbitMQ and configure with a base user/password and vhost in a prebake stage. That way when the image is launched the instance already has a preconfigured RabbitMQ with the user/password & vhost.

[Michael Klishin]

Well, the error says that your node was clustered with another node.

See "Load definitions at startup", that sounds like what you want

http://www.rabbitmq.com/management.html.

On Thu, Aug 4, 2016 at 9:14 AM, <brendan...@reancloud.com> wrote:

I'm not trying to cluster. I'm trying to install RabbitMQ and configure with a base user/password and vhost in a prebake stage. That way when the image is launched the instance already has a preconfigured RabbitMQ with the user/password & vhost.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[brendan...@reancloud.com]

Would being installed in a prebake under a different hostname then being launched with a new hostname constitute a "cluster"? I assumed that by setting the NODENAME in /etc/rabbitmq/rabbitmq-env.config defines the necessary configuration & takes the hostnames out of the equation?

[brendan...@reancloud.com]

Here is my Ansible steps:

# install Erlang RPM

- name: Install Erlang RPM via YUM

  yum:

    name="{{staging_dir}}/{{ems_archive_dir}}/{{erlang_rpm_filename}}"

    state=present

# install RabbitMQ RPM

- name: Install RabbitMQ RPM via YUM

  yum:

    name="{{staging_dir}}/{{ems_archive_dir}}/{{rabbiqmq_rpm_filename}}"

    state=present

# update ConfigureEMS script perms

- name: Update ConfigureEMS script perms

  file:

    path="{{staging_dir}}/{{ems_archive_dir}}/{{configure_ems_filename}}"

    mode=0700

- name: Create rabbitmq-env.conf file to define RabbitMQ NODE name

  template:

    src=rabbitmq-env.conf.j2

    dest=/etc/rabbitmq/rabbitmq-env.conf

    mode=0644

# make RabbitMQ auto-start on boot

- name: Ensure RabbitMQ service startup and running

  service:

    name=rabbitmq-server

    enabled=yes

    state=started

# execute ConfigureEMS script

- name: Execute ConfigureEMS script

  shell: "bash {{configure_ems_filename}} {{ems_username}} {{ems_password}} {{ems_vhostname}}"

  args:

    chdir: "{{staging_dir}}/{{ems_archive_dir}}"

• The {{...}} signifies variables in Ansible. 

• I am creating the /etc/rabbitmq/rabbitmq-env.config file after the YUM install of the RPM. It's contents are: 
  

NODENAME={{ems_rabbitmq_node_name}}

Which renders to:

NODENAME=ems

• The ConfigureEMS script is just a Bash wrapper for rabbitmqctl to create the defined vhost, user, pass, and set permissions.
  

ConfigureEMS:

#!/bin/sh

SCRIPT=`basename $0`

USAGE="Usage: ${SCRIPT} username password virtualhost"

if [ $# -ne 3 ]; then

   echo

   echo "${USAGE}" >&2

   echo

   exit 1

fi

if [ `id -u` != 0 ] ; then

   echo

   echo "Only root should run ${SCRIPT}"

   echo

   exit 1

fi

AMQP_USER=

AMQP_PASS=

AMQP_HOST=

while [ $# -gt 0 ]; do

   if [ -z "${AMQP_USER}" ]; then

     AMQP_USER="${1}"

   elif [ -z "${AMQP_PASS}" ]; then

     AMQP_PASS="${1}"

   elif [ -z "${AMQP_HOST}" ]; then

     AMQP_HOST="${1}"

   fi

   shift

done

if [ -z "${AMQP_USER}" ]; then

   echo "Required: username"

   exit 1

fi

if [ -z "${AMQP_PASS}" ]; then

   echo "Required: password"

   exit 1

fi

if [ -z "${AMQP_HOST}" ]; then

   echo "Required: virtualhost"

   exit 1

fi

PATH=/sbin:/usr/sbin:${PATH}

ROOT_SERVICE="service"

AMQP_SERVER="rabbitmq-server"

AMQP_CONTROL="rabbitmqctl"

AMQP_PLUGIN_CONFIG="rabbitmq-plugins"

AMQP_PLUGIN_NAME="rabbitmq_management"

AMQP_ADD_USER="add_user"

AMQP_ADD_ACCESS="set_user_tags"

AMQP_ADD_HOST="add_vhost"

AMQP_ADD_HOST_ACCESS="set_permissions -p"

AMQP_DEFAULT_HOST="/"

AMQP_ADMIN_ACCESS="administrator"

# Ensure the AMQP service is running

echo

echo "Starting the TEST Messaging Service..."

echo

${ROOT_SERVICE} ${AMQP_SERVER} start

EXEC_STATUS=$?

if [ ${EXEC_STATUS} -ne 0 ]; then

   echo

   echo "WARNING: EMS start encountered Error ${EXEC_STATUS}" >&2

   echo

fi

# Enable the Administrative plug-in

echo

echo "Enabling the Admin GUI on port 15672..."

echo

${AMQP_PLUGIN_CONFIG} enable ${AMQP_PLUGIN_NAME}

EXEC_STATUS=$?

if [ ${EXEC_STATUS} -ne 0 ]; then

   echo

   echo "WARNING: EMS GUI enable encountered Error ${EXEC_STATUS}" >&2

   echo

fi

# Restart the service to enable the plug-in

${ROOT_SERVICE} ${AMQP_SERVER} restart

EXEC_STATUS=$?

if [ ${EXEC_STATUS} -ne 0 ]; then

   echo

   echo "WARNING: EMS restart encountered Error ${EXEC_STATUS}" >&2

   echo

fi

# Create the admin user

${AMQP_CONTROL} ${AMQP_ADD_USER} ${AMQP_USER} ${AMQP_PASS}

EXEC_STATUS=$?

if [ ${EXEC_STATUS} -ne 0 ]; then

   echo

   echo "WARNING: EMS user create encountered Error ${EXEC_STATUS}" >&2

   echo

fi

# Make the user an administrator

${AMQP_CONTROL} ${AMQP_ADD_ACCESS} ${AMQP_USER} ${AMQP_ADMIN_ACCESS}

EXEC_STATUS=$?

if [ ${EXEC_STATUS} -ne 0 ]; then

   echo

   echo "WARNING: EMS admin control encountered Error ${EXEC_STATUS}" >&2

   echo

fi

# Create the virtual host

${AMQP_CONTROL} ${AMQP_ADD_HOST} ${AMQP_HOST}

EXEC_STATUS=$?

if [ ${EXEC_STATUS} -ne 0 ]; then

   echo

   echo "WARNING: EMS virtual host creation encountered Error ${EXEC_STATUS}" >&2

   echo

fi

# Grant access to the virtual host to the user

${AMQP_CONTROL} ${AMQP_ADD_HOST_ACCESS} ${AMQP_HOST} ${AMQP_USER} ".*" ".*" ".*"

EXEC_STATUS=$?

if [ ${EXEC_STATUS} -ne 0 ]; then

   echo

   echo "WARNING: EMS virtual host access control encountered Error ${EXEC_STATUS}" >&2

   echo

fi

# Grant access to the default virtual host to the user

${AMQP_CONTROL} ${AMQP_ADD_HOST_ACCESS} ${AMQP_DEFAULT_HOST} ${AMQP_USER} ".*" ".*" ".*"

EXEC_STATUS=$?

if [ ${EXEC_STATUS} -ne 0 ]; then

   echo

   echo "WARNING: EMS virtual host access control encountered Error ${EXEC_STATUS}" >&2

   echo

fi

echo

echo "TEST Messaging Service configuration completed."

echo

[Michael Klishin]

No but hostname changing between starts can confuse the node. Tools such as Chef and Puppet

re-render templates using the effective node name before a node starts, for instance,

effectively making that invisible.

Are you setting both RABBITMQ_NODENAME and RABBITMQ_MNESIA_DIR?

Again, if the goal is to only pre-create some vhosts, users, and possibly entities,

definition loading from a JSON file (exported from an existing node, for example)

on startup should be sufficient. Packer then would enable the management plugin

(which can be done without even starting a node) and override DB directory

to not have a hostname in it.

On Thu, Aug 4, 2016 at 9:28 AM, <brendan...@reancloud.com> wrote:

Would being installed in a prebake under a different hostname then being launched with a new hostname constitute a "cluster"? I assumed that by setting the NODENAME in /etc/rabbitmq/rabbitmq-env.config defines the necessary configuration & takes the hostnames out of the equation?

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[brendan...@reancloud.com]

I'm only setting the NODENAME in rabbitmq-env.config. According to the node log in /var/log/rabbitmq-server/ems.log, the database dir hasn't changed from the initial startup under node : ems@ip-10-71-2-101

=INFO REPORT==== 4-Aug-2016::14:59:41 ===
node           : ems@ip-10-71-2-101
home dir       : /var/lib/rabbitmq
config file(s) : (none)
cookie hash    : XtGvrbshozbMlz3PP+dXQw==
log            : /var/log/rabbitmq/ems.log
sasl log       : /var/log/rabbitmq/ems-sasl.log
database dir   : /var/lib/rabbitmq/mnesia/ems
--

Copyright (C) 2007-2014 GoPivotal, Inc.

Licensed under the MPL.  See http://www.rabbitmq.com/

=INFO REPORT==== 4-Aug-2016::14:59:46 ===
node           : ems@ip-10-71-2-101
home dir       : /var/lib/rabbitmq
config file(s) : (none)
cookie hash    : XtGvrbshozbMlz3PP+dXQw==
log            : /var/log/rabbitmq/ems.log
sasl log       : /var/log/rabbitmq/ems-sasl.log
database dir   : /var/lib/rabbitmq/mnesia/ems
--

Copyright (C) 2007-2014 GoPivotal, Inc.

Licensed under the MPL.  See http://www.rabbitmq.com/

=INFO REPORT==== 4-Aug-2016::15:07:08 ===
node           : ems@ip-10-100-50-119
home dir       : /var/lib/rabbitmq
config file(s) : (none)
cookie hash    : XtGvrbshozbMlz3PP+dXQw==
log            : /var/log/rabbitmq/ems.log
sasl log       : /var/log/rabbitmq/ems-sasl.log
database dir   : /var/lib/rabbitmq/mnesia/ems

However, reading through these info reports, the node is being referred to as 'ems@hostname' where I would've expected NODENAME in rabbitmq-env.config to just produce the node entry as 'ems' without the hostname attached.

[Michael Klishin]

There are two node name types: long (without @domain) and long (FQDNs).

If you know what FQDN will be used for the node at the time it is deployed,

you can use that via a Packer variable. Otherwise you'd most likely need to run

Ansible on the deployed image so that it re-generates all files that include

effective hostname and node name.

Pre-generated hostnames and IPs is a common pain point when deploying

data services via images.

Alternatively you can use definitions import or HTTP API to pre-declare vhosts, users, permissions, and entities.

I recommend overriding RABBITMQ_MNESIA_DIR to point to /var/lib/rabbitmq/db

or something like that, without any traces of node name, just to make it most predictable.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[brendan...@reancloud.com]

Just to wrap this up, I ended up moving the entire RabbitMQ installation and basic configuration piece out of the prebake phase due to the limitations surrounding the nodename, etc. We now have an Ansible playbook being executed during EC2's User Data and list_users and list_vhosts show the expected configuration.

Thank you for the information regarding RabbitMQ -- It really is appreciated.

-Brendan