Upgrade from docker image 3.7.17 to 3.8.5 fails to load definitions on boot

89 views
Skip to first unread message

Iain Earl

unread,
Jul 22, 2020, 7:41:39 AM7/22/20
to rabbitmq-users
I'm upgrading our cluster from rabbitmq:3.7.17-alpine to rabbitmq:3.8.5-alpine and I'm getting the following error:

Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{{error,<<"Validation failed\n\ncomponent federation-upstream not found\n (/workers/federation-upstream/some-upstream

I have tried changing the config to use the new built-in definition loading (old: management.load_definitions: /some/file.json, new: load_definitions: /some/file.json) but I still get the same error. It's like it's loading the definitions before enabling the rabbitmq_federation plugin. I'm enabling those plugins using the enabled_plugins file and it worked fine on the old version.

Rolling back to 3.7.17 works fine again with exactly the same configuration.

Any insight would be much appreciated!

Luke Bakken

unread,
Jul 22, 2020, 9:06:15 AM7/22/20
to rabbitmq-users
Hello,

This is a known bug that will be fixed in version 3.8.6.

For now, you can start RabbitMQ and wait for it to start by using the rabbitmqctl await_startup command. After that returns, import definitions - https://www.rabbitmq.com/definitions.html#import

Thanks,
Luke

Michael Klishin

unread,
Jul 22, 2020, 9:55:57 AM7/22/20
to rabbitmq-users
The issue here is that as of 3.8.4, plugin startup and definition import have a chicken-and-egg problem that was not as visible in earlier versions.

3.8.6 addresses this by moving definition import to the very end of the boot process. You can still import definitions after the node starts
by removing a `load_definitions` configuration key and adding a RUN step that imports the same file using `rabbitmqadmin` or `rabbitmqctl import_definitions` (the latter is available as of 3.8.3 or so)

Iain Earl

unread,
Jul 23, 2020, 6:22:58 AM7/23/20
to rabbitmq-users

Thanks guys!

The upgrade isn't urgent so I can wait for 3.8.6 to come out.

Iain Earl

unread,
Sep 7, 2020, 6:09:43 AM9/7/20
to rabbitmq-users
Hi,

Coming back to this, I tried to upgrade to 3.8.8 today and I get the same error. Is there a bug I can follow for this?

Cheers
Iain

Iain Earl

unread,
Sep 7, 2020, 6:16:48 AM9/7/20
to rabbitmq-users
My bad, reading the comments on the PR I can see I was using the old management.load_definitions setting


Changing to just load_definitions fixes it!
Reply all
Reply to author
Forward
0 new messages