rabbitmq-delayed-message-exchange crashing docker image

[Julian Cuevas]

Hey everyone,

I was tasked with installing the rabbitmq-delayed-message-exchange plugin into a RabbitMQ Kubernetes cluster, I tried to install it but RabbitMQ started crashing as soon as I added it to the filesystem.

I thought the issue was with the RabbitMQ chart I was using but I was able to replicate the error successfully with the official Docker image (the chart uses this image as its base).

Here are the steps that I'm following:

1) Download the plugin to local storage:

curl -O https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/releases/download/v3.8.0/rabbitmq_delayed_message_exchange-3.8.0.ez

2) Create enabled_plugins file:

echo "[rabbitmq_delayed_message_exchange,]." > enabled_plugins

3) Mount enabled_plugins & plugin folder and run image:

 docker run -v $(pwd)/plugins:/plugins --mount type=bind,source=$(pwd)/enabled/enabled_plugins,target=/etc/rabbitmq/enabled_plugins rabbitmq:3.8.1-alpine

I tried with the plugin zipped and unzipped and I tried 7 different versions of the docker image, all end with the same log lines:

{"init terminating in do_boot",{undef,[{rabbit_nodes_common,make,["rabbit@55846b452211"],[]},{rabbit_prelaunch,start,0,[{file,"src/rabbit_prelaunch.erl"},{line,38}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ({undef,[{rabbit_nodes_common,make,rabbit@55846b452211,[]},{rabbit_prelaunch,start,0,[{_},{_}]},{init,start_em,1,[]},{init,do_boot,3,[]}]})

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

Attached is the crash dump file

I also noticed that just having the plugin in the plugins folder crashes rabbit, without asking it to load it via the enabled_plugins file.

Can anyone give me some pointers regarding this?

[Luke Bakken]

Hi Julian,

Any time you use curl to download a file, be sure that the file you downloaded is valid.

By default, curl does not follow redirects, so when you run this command:

curl -O https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/releases/download/v3.8.0/rabbitmq_delayed_message_exchange-3.8.0.ez

It doesn't actually save the .ez file, but HTML corresponding to a redirect.

The command you want to use is this (note the L argument):

curl -LO https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/releases/download/v3.8.0/rabbitmq_delayed_message_exchange-3.8.0.ez

https://curl.haxx.se/docs/manpage.html#-L

Thanks,

Luke

[Julian Cuevas]

Thanks for the reply Luke, I tried it with wget and unzipped the file to check whether it was the correct one.

I'm still getting the same issue.

[Luke Bakken]

Hello,

I did try to reproduce this using my local 3.8.3 test environment (non-Docker). Enabling the plugin and running RabbitMQ worked correctly.

Can you provide exact details about how you are installing the plugin?

What is the output of rabbitmq-plugins list when it is installed?

Finally, please check the SHA-256 checksum of the file. It should match this:

$ sha256 plugins/rabbitmq_delayed_message_exchange-3.8.0.ez
SHA256 (plugins/rabbitmq_delayed_message_exchange-3.8.0.ez) = 908e4ae1148a26afb4cce3c882082376dbe37b286e0560630da3e131acff3150

Thanks,

Luke

[Julian Cuevas]

Hey Luke,

Thanks for the help, I really appreciate it.

I tried it in a non-docker environment and it worked as well, I tried downloading and enabling the plugin after the docker image has booted and it worked correctly as well (exec'ing into the container and running the steps manually), however, neither of those will work in a Kubernetes cluster.

Here's an sh that I wrote that follows the exact steps I'm following (plus checking the sha256 as you suggested):

#!/bin/sh
mkdir plugins
mkdir enabled
wget https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/releases/download/v3.8.0/rabbitmq_delayed_message_exchange-3.8.0.ez
echo "908e4ae1148a26afb4cce3c882082376dbe37b286e0560630da3e131acff3150 rabbitmq_delayed_message_exchange-3.8.0.ez" | sha256sum -c
mv rabbitmq_delayed_message_exchange-3.8.0.ez plugins/rabbitmq_delayed_message_exchange-3.8.0.ez
echo "[rabbitmq_delayed_message_exchange, rabbitmq_shovel, rabbitmq_shovel_management, rabbitmq_federation, rabbitmq_federation_management]." > enabled/enabled_plugins
docker run -v $(pwd)/plugins:/plugins --mount type=bind,source=$(pwd)/enabled/enabled_plugins,target=/etc/rabbitmq/enabled_plugins rabbitmq:3.8.3-alpine

This will create the folders, download the plugin, check sha256 and boot rabbit from the docker image, it will try to enable the plugin as described in the enabling plugins section of docker hub.

I've noticed that it only crashes if the plugin is present in the plugins folder when rabbit boots and the plugin isn't enabled,  if the plugin is downloaded and enabled while rabbit is running it won't crash no matter how many times the container is stopped and started again.

I'm attaching the crash dump file again in case it helps

Thanks for your awesome help!

Julián Camilo Cuevas A.

DevOps Engineer

Mobile: +573105514204

Skype: akath19

Address: Calle 94A No. 11A - 32 Oficina 503

Bogota, Colombia

www.yellowpepper.com

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/zUlKBn4M_X8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/b3116e7a-5150-42c5-a9f8-7528c5880c11%40googlegroups.com.

[Luke Bakken]

Hi Julian,

I'm not a docker expert, but it looks like you're saving the rabbitmq_delayed_message_exchange-3.8.0.ez file to a local directory called plugins, but then mounting that directory in the container at the path /plugins

RabbitMQ has no idea about /plugins ... you'll have to make sure the .ez file is copied into the correct location in your dockter container. Usually the path is something like /usr/lib/rabbitmq/plugins

Thanks -

Luke

[Julian Cuevas]

Hey Luke,

/plugins is a symlink to /opt/rabbitmq/plugins so it should technically work.

However, the only solution that works is creating a docker image with the plugins included, seems like adding any plugins besides the included ones crashes rabbit, I tried it with rabbitmq_rtopic_exchange and rabbitmq_delayed_message_exchange, both crash with the same issue:

{"init terminating in do_boot",{init terminating in do_boot ({undef,[{rabbit_nodes_common,make,rabbit@9f3dc0e84undef,[{rabbit_nodes_common,make,["rabbit@9f3dc0e845a5"],[]},{rabbit_prelaunch,start,0,[{file,"src/rabbit_prelaunch.erl"},{line,38}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}5a5,[]},{rabbit_prelaunch,start,0,[{_},{_}]},{init,start_em,1,[]},{init,do_boot,3,[]}]})Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

Perhaps a line should be included in Docker Hub to clarify that only bundled plugins can be enabled via the process described there.

Regards,

Julián Camilo Cuevas A.

DevOps Engineer

Mobile: +573105514204

Skype: akath19

Address: Calle 94A No. 11A - 32 Oficina 503

Bogota, Colombia

www.yellowpepper.com

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/zUlKBn4M_X8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/56b51490-f531-41c3-a39f-8245f210fdfa%40googlegroups.com.

[Luke Bakken]

Hi Julian,

Have you tried without a symbolic link?

If you think this is an issue with the docker image, I suggest filing an issue here: https://github.com/docker-library/rabbitmq/issues

The RabbitMQ team does not maintain that image. You have sufficient information for someone to reproduce and investigate further.

Thanks -

Luke

On Tuesday, April 28, 2020 at 9:10:46 AM UTC-7, Julian Cuevas wrote:

Hey Luke,

[Julian Cuevas]

Thanks Luke, I was under the impression that the rabbitmq team maintained the image.

I'll report there.

Thanks again,

Julián Camilo Cuevas A.

DevOps Engineer

Mobile: +573105514204

Skype: akath19

Address: Calle 94A No. 11A - 32 Oficina 503

Bogota, Colombia

www.yellowpepper.com

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/zUlKBn4M_X8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/cd6c73fc-d19b-478c-93b5-95a065503756%40googlegroups.com.

[Luke Bakken]

Hi Julian -

We help out when we can but aren't (yet) responsible for it.

Thanks -

Luke