Unable to transfer definitions between RMQ instances - function_clause

[HGN]

Hello,

I've created and configured RMQ instance and need to "clone" it - DEV and PROD instances.

So I've transferred definitions in json file and tried to import them during boot up using

load_definitions = /etc/rabbitmq/rabbit_EnerSpot-DEV.json

in rabbitmq.conf of an empty RMQ installation. But the new instance refused to start with following errors (journalctl -xeu rabbitmq-server.service):

--------------------

Dec 07 08:36:29 bus rabbitmq-server[48088]: 2023-12-07 08:36:29.700322+00:00 [notice] <0.44.0> Application syslog exited with reason: stopped
Dec 07 08:36:29 bus rabbitmq-server[48088]: 2023-12-07 08:36:29.703078+00:00 [notice] <0.232.0> Logging: switching to configured handler(s); following message>
Dec 07 08:36:29 bus rabbitmq-server[48088]:   ##  ##      RabbitMQ 3.12.10
Dec 07 08:36:29 bus rabbitmq-server[48088]:   ##  ##
Dec 07 08:36:29 bus rabbitmq-server[48088]:   ##########  Copyright (c) 2007-2023 VMware, Inc. or its affiliates.
Dec 07 08:36:29 bus rabbitmq-server[48088]:   ######  ##
Dec 07 08:36:29 bus rabbitmq-server[48088]:   ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com
Dec 07 08:36:29 bus rabbitmq-server[48088]:   Erlang:      25.2.3 [jit]
Dec 07 08:36:29 bus rabbitmq-server[48088]:   TLS Library: OpenSSL - OpenSSL 3.0.11 19 Sep 2023
Dec 07 08:36:29 bus rabbitmq-server[48088]:   Release series support status: supported
Dec 07 08:36:29 bus rabbitmq-server[48088]:   Doc guides:  https://rabbitmq.com/documentation.html
Dec 07 08:36:29 bus rabbitmq-server[48088]:   Support:     https://rabbitmq.com/contact.html
Dec 07 08:36:29 bus rabbitmq-server[48088]:   Tutorials:   https://rabbitmq.com/getstarted.html
Dec 07 08:36:29 bus rabbitmq-server[48088]:   Monitoring:  https://rabbitmq.com/monitoring.html
Dec 07 08:36:29 bus rabbitmq-server[48088]:   Logs: /var/log/rabbitmq/rab...@bus.log
Dec 07 08:36:29 bus rabbitmq-server[48088]:         <stdout>
Dec 07 08:36:29 bus rabbitmq-server[48088]:   Config file(s): /etc/rabbitmq/rabbitmq.conf
Dec 07 08:36:31 bus rabbitmq-server[48088]:   Starting broker...
Dec 07 08:36:31 bus rabbitmq-server[48088]: BOOT FAILED
Dec 07 08:36:31 bus rabbitmq-server[48088]: ===========
Dec 07 08:36:31 bus rabbitmq-server[48088]: Exception during startup:
Dec 07 08:36:31 bus rabbitmq-server[48088]: error:function_clause
Dec 07 08:36:31 bus rabbitmq-server[48088]:     rabbit_db_rtparams:get_all/2, line 178
Dec 07 08:36:31 bus rabbitmq-server[48088]:         args: [undefined,<<"vhost-limits">>]
Dec 07 08:36:31 bus rabbitmq-server[48088]:     rabbit_runtime_parameters:list/2, line 261
Dec 07 08:36:31 bus rabbitmq-server[48088]:     rabbit_vhost_limit:get_limit/2, line 192
Dec 07 08:36:31 bus rabbitmq-server[48088]:     rabbit_vhost_limit:would_exceed_queue_limit/2, line 111
Dec 07 08:36:31 bus rabbitmq-server[48088]:     rabbit_definitions:validate_vhost_limit/3, line 867
Dec 07 08:36:31 bus rabbitmq-server[48088]:     maps:fold_1/3, line 411
Dec 07 08:36:31 bus rabbitmq-server[48088]:     rabbit_definitions:apply_defs/3, line 433
Dec 07 08:36:31 bus rabbitmq-server[48088]:     rabbit_definitions:maybe_load_definitions/0, line 100
Dec 07 08:36:37 bus rabbitmq-server[48088]: {"init terminating in do_boot",{error,function_clause}}
Dec 07 08:36:37 bus rabbitmq-server[48088]: init terminating in do_boot ({error,function_clause})
Dec 07 08:36:37 bus rabbitmq-server[48088]:
Dec 07 08:36:37 bus rabbitmq-server[48088]: Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done
Dec 07 08:36:37 bus systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=1/FAILURE

--------------------

And logfile rab...@bus.log is attached. I did try with removed directory /var/lib/rabbitmq/mnesia (cookie file untouched) with no success. I did attached even the exported definition file used.

Both RMQs are version v3.12.10 using erlang from Debian Bookworm 1:25.2.3+dfsg-1

What should I try? Thank you.

Best regards

Marek

[HGN]

I did even try to use rabbitmqctl to import definitions with (same) error:

root@bus:/etc/rabbitmq# rabbitmqctl import_definitions /etc/rabbitmq/definitions.json
Importing definitions in JSON from a file at "/etc/rabbitmq/definitions.json" ...
Error:
{:function_clause, [{:rabbit_db_rtparams, :get_all, [:undefined, "vhost-limits"], [file: ~c"rabbit_db_rtparams.erl", line: 178]}, {:rabbit_runtime_parameters, :list, 2, [file: ~c"rabbit_runtime_parameters.erl", line: 261]}, {:rabbit_vhost_limit, :get_limit, 2, [file: ~c"rabbit_vhost_limit.erl", line: 192]}, {:rabbit_vhost_limit, :would_exceed_queue_limit, 2, [file: ~c"rabbit_vhost_limit.erl", line: 111]}, {:rabbit_definitions, :validate_vhost_limit, 3, [file: ~c"rabbit_definitions.erl", line: 867]}, {:maps, :fold_1, 3, [file: ~c"maps.erl", line: 411]}, {:rabbit_definitions, :apply_defs, 3, [file: ~c"rabbit_definitions.erl", line: 433]}, {:rabbit_definitions, :import_parsed, 1, []}]}

Best Regards

Marek

Dne čtvrtek 7. prosince 2023 v 9:53:12 UTC+1 uživatel HGN napsal:

[Luke Bakken]

Thank you for the information.

Are you running RabbitMQ directly on a Debian server?

[Luke Bakken]

Ah, yes you are, based on the fact that you said you ran "journalctl -xeu rabbitmq-server.service" to get logs.

I'll see if I can reproduce this.

[Luke Bakken]

Thank you for the comprehensive report. I can reproduce your issue and have opened the following: https://github.com/rabbitmq/rabbitmq-server/issues/10068

[Michael Klishin]

The queue definitions in this file do not have a virtual host, so you cannot import this file at boot time (you might be able to via the HTTP API endpoint that imports them for a **specific** virtual host.

If you want to "clone" definitions, you must import and then export **all** of them. Most likely you have exported a single virtual host

and are trying to import it at boot time.

Compare this definition file produced by a 3.12.10 node:

``` json
  "queues": [
    {
      "name": "sq.1",
      "vhost": "/",
      "durable": true,
      "auto_delete": false,
      "arguments": {
        "x-queue-type": "stream"
      }
    },
    {
      "name": "cq.1",
      "vhost": "/",
      "durable": true,
      "auto_delete": false,
      "arguments": {}
    },
    {
      "name": "qq.1",
      "vhost": "/",
      "durable": true,
      "auto_delete": false,
      "arguments": {
        "x-queue-type": "quorum"
      }
    }
  ],
```

and one of the queues in in your file:

``` json
      {
          "source": "ote_messages",
          "destination": "ote_measurements",
          "destination_type": "exchange",
          "routing_key": "otemsg.121",
          "arguments": {}
      },
```

This means that in `rabbit_definitions`, the following line produces a map where keys are `undefined`:

``` erl
{ok, VHostMap} = filter_out_existing_queues(Queues0),
```

[Michael Klishin]

I am sorry, I have copied a binding. But in the provided file, the queues do not have a virtual host:

```

{

"name": "tbdconn-requests",

"durable": true,

"auto_delete": false,

"arguments": {

"x-queue-type": "classic",

"x-queue-version": 2

}

},

```

[Michael Klishin]

I have verified that if you export definitions using management UI (the form with a button at the bottom) while you have All virtual host selected, you get back all definitions and all resources have the virtual host field.

If you only export when just one virtual host is selected, the resources do not have a virtual host set.

There are multiple different ways to avoid this confusion. On the export side:

 * Set virtual host even when a single virtual host's definitions are imported (I have no idea why it is not currently the case, could be unintentional)
 * Name resulting files in a way that would hint at the fact that these are definitions of a single virtual hosts

On the importing code path:

 * Validate queues specifically and throw an exception
 * OR use the default virtual host (a setting in RabbitMQ) but I suspect this can end up being more confusing, not less

Or we can always export **all** definitions because export of an individual virtual host is an rarely used feature. This is an option for 3.13.

[Michael Klishin]

The per-virtual host import was introduced in [1], seemingly only to be used with CLI tools. I don't remember any details since that was almost eight years ago.

But this is something easy to change one way or another.

1. https://github.com/rabbitmq/rabbitmq-server/commit/8ba0e077df366273df8dc5fd588ac6a15999fb28

[Michael Klishin]

Judging from a few things in the commit above, maybe the idea was to allow a virtual host-specific import file to be imported into any virtual host (in a different cluster).

I don't know how common this scenario is but adding back the virtual host field into virtual host-specific exports will break it.

[HGN]

Hello everybody,

as you already found out there's a problem with definition export from Web Management page. I successfuly exported DEV definitions once, didn't know how and why it succededed, but I was then able to import them without problem. After changes I wanted to back up definitions of the second instance but wasn't successful again. And again I didn't select 'all' virtual hosts.

I now checked, that when 'all' virtual hosts are selected it works as I expected. I didn't find this behaviour documented anywhere ( https://www.rabbitmq.com/definitions.html) .

Thanks for cooperation.

Marek

Dne pátek 8. prosince 2023 v 3:47:56 UTC+1 uživatel Michael Klishin napsal:

[HGN]

And I can see a hint message in web interface w/o mentioning the fact of distinguishing between VHosts:

"The definitions consist of users, virtual hosts, permissions, parameters, exchanges, queues, policies and bindings. They do not include the contents of queues. Exclusive queues will not be exported."

Marek

Dne pátek 8. prosince 2023 v 8:13:50 UTC+1 uživatel HGN napsal:

[Michael Klishin]

We did not encounter "a problem", we have encountered a confusing feature that works by design: when a single virtual host definitions are imported, they will all (by design) lack the virtual host field.
Therefore they only can be used with the virtual host-specific import feature and not import on boot.

Future RabbitMQ versions will fail with such files. Whether we will change the exporting API endpoint remains to be seen, it's impossible to measure how many environments can be affected by this.

[Michael Klishin]

Those who are interested in definition import enough

can follow https://github.com/rabbitmq/rabbitmq-server/pull/10072 and https://github.com/rabbitmq/rabbitmq-server/issues/10078