Questions regarding management api - definitions import and export - possibility to have ignored keys?

[Olivier Van Dam]

Hi,

I'm trying to setup an automated process for pushing new configuration to our clusters.

In that context, the approach that seemed more fitting relates to importing new definitions,

but to make it more "solid", I was wondering if there were some keys that would be ignored by the import api, specifically something that would allow us to state if a queue should be recreated if it's arguments have changed, or similar logic.

If this was possible, we could have our reference configuration, have a script processing it beforehand to check the current state vs target state and perform deletes prior to importing the new definitions.

[Michal Kuratczyk]

Import is additive - RabbitMQ doesn't delete things that are not present in the imported file.

If you configure the definitions file in the config file (to be load on each boot), you can skip the import, if the definitions file didn't change

since the previous boot: https://www.rabbitmq.com/definitions.html#import-on-boot-skip-if-unchanged

However, if the definitions file contains a queue definition different from an existing queue with the same name,

the import will succeed, but the queue definition won't change. This is because the import process skips existing

queues to speed up the process. Since some changes would require the queue to be deleted and recreated,

it would introduce a few additional behaviours and configuration options that I don't think we want to introduce

(things like, recreate if empty, recreate even if not empty, etc). Sounds like what you want is a Management API

client that reconciles the definitions, similar to https://github.com/rabbitmq/messaging-topology-operator

(but this one is Kubernetes-specific).

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/6a888da4-e2db-4831-b65e-4ad9322ef15an%40googlegroups.com.

--

Michał

RabbitMQ team

[Olivier Van Dam]

Hi,

I'm planning to implement the logic for reconciliation, but would like to use additional keys in the original definition file,

so that a local dev can test out the definition in a new environment,

while I can use the same definitions file, preprocess it and figure out whether to "skip" some changes or delete some objects that need to be recreated based on said new keys.

I'll do a quick test, but was hoping someone who knew the import process well could tell me if there are prefixed keys that would be automatically ignored, or not.

[Michal Kuratczyk]

I don't fully understand your question but there are no special keys to ignore things. You either import the file or you don't

and existing entities with matching names are skipped.

The import process is here if you want to go through it:

https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/src/rabbit_definitions.erl#L425

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/7909f756-f6fd-4991-b430-fa33833551b0n%40googlegroups.com.

--

Michał

RabbitMQ team

[Vilius Šumskas]

Hi,

 

from what I understood you are trying to use the same definition for both, initial environments configuration and testing. My advice is to not do this. If you need a separate data sample (fixture) for testing then every test should have its own.

 

We use similar approach internally. For initial new environment preconfiguration our application prepopulates queues automatically via Management API from the data stored in Postgres DB. This can be changed into pure definition import if needed, but we chose to use database migrations to manage all data and metadata for environment instead.

For all local testing and integration tests developers write their own fixtures which then get attached to the test suite.

 

--

    Vilius

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/7909f756-f6fd-4991-b430-fa33833551b0n%40googlegroups.com.

[Olivier Van Dam]

Thanks to both.

@Michal: Will check the import process (not versed in Erlang, but one has to start somewhere ;) )

@Vilius: Thanks for your input. I will see to provide a sample script for "cleanup" of the enriched definition and work from there. 

I was originally trying to avoid that as I'd like to promote local tests by Devs and anything that can make it harder (one more component in the flow, one more thing to know compared to the documented api, ...) makes it less likely to happen.

I'll probably still check and test if I can put custom keys in the definition so I learn more of the RabbitMQ specificities, but will probably leave it as general info for me.