Plugins development and broker boot steps

[noxdafox]

Greetings,

I was investigating an issue in a RabbitMQ plugin I maintain:
https://github.com/noxdafox/rabbitmq-message-deduplication

When restarting the broker, all declared deduplication queues do not behave as such any more.

Deduplication queues implement the `rabbit_backing_queue` behaviour adding the deduplication functionality through the `is_duplicate` callback.
Similarly as for priority queues, the plugin installs a boot-step which overrides the `backing_queue_module` application environment variable at startup.

When the broker is restarted, all existing queues are initialized before the plugin. Therefore, the deduplication queues will adopt the `rabbit_variable_queue` module instead of the `rabbit_message_deduplication_queue` one.

It took me some time to realize plugins are loaded as last components. As a consequence, even if they hook to the correct boot step, they won't be initialized during such. Hence, the erroneous behaviour described above.

Currently, I developed a workaround which restarts all the deduplication queue processes when the plugin is started. In this way, I ensure the correct `backing_queue_module` is adopted.
Nevertheless, I dislike this approach as:
 * It pokes into the broker internal supervisor hierarchy
 * It generates misleading error messages in the logs while restarting the queues
 * It can significantly slow down the broker startup as queues are recovered twice
 * It might cause undesirable side effects

The main question is if there's a better way to fix this particular issue. One approach, for example, would be to notify the queue process of its correct `backing_queue_module` when initializing the plugin.

The second thing I wanted to ask is what is the difference from a plugin development perspective between installing a boot step and the implementation of the Application behaviour.
If plugins initialization are executed last, their boot dependency declaration will be always ignored (the `enables` statement).
For example, the RabbitMQ MGMT application plugin enables the `empty_db_check` boot step, most of the exchange type plugins enable `kernel_ready`. Even if these steps are executed way before the plugins boot steps are executed.
Yet most of the plugins seem to install their boot steps nevertheless.

What is I am missing or misunderstanding?

Kind regards,

Matteo.

[Michael Klishin]

We had to update a few stateful exchange plugins and they recover their state (from bindings) explicitly. The only alternative to what you have in mind would be

to make queues "reload" their backing queue module, or move queue recovery to the very last step of the boot sequence, which can have consequences

for other plugins. I think we are done with moving boot steps around for a while: no matter what we do, some plugins will inevitably be affected.

[noxdafox]

I think the problem is not that boot steps have been shuffled around but rather that the plugins are not able to register them correctly.

As I mentioned in the example, the RMQ Management Plugin place its `rabbit_management_load_definitions` boot step as a dependency of the `empty_db_check` one. Nevertheless, as the plugin module is loaded last, its boot step is registered way after the `empty_db_check` is executed.

https://github.com/rabbitmq/rabbitmq-server/blob/master/deps/rabbitmq_management/src/rabbit_mgmt_app.erl#L25

This breaks the assumption of the developers regarding their plugins being initialized at the right moment.

This is the reason I am questioning the value of adopting boot steps for plugins development. If they are always executed last, what is the difference compared to starting an Erlang application instead?

Regarding your suggestions, I'd definitely avoid moving the queue recovery as we are not sure of the possible implications. The `backing_queue_module` "reloading" would be a less intrusive action.

Nevertheless, it would only partially solve the issue as the `backing_queue_module` specific initialization implementations would still be skipped. For example, if we imagine `rabbit_priority_queue` implemented as a plugin (as it was previously), this would mean the priority queues would not be correctly recovered.