[RabbitMQ 3.9.19] Default user and pass no longer work with docker compose setup

[Tosh Y]

Hello,

After wanting to update my RabbitMQ container from 3.9.7-management to the latest 3.9-management image, I've encountered an issue that setting the environment variables `RABBITMQ_DEFAULT_USER` / `RABBITMQ_DEFAULT_PASS` do not longer work.

Error from container logs:

```

Starting broker...2022-10-01 15:29:31.030898+00:00 [info] <0.837.0> accepting AMQP connection <0.837.0> (192.168.80.11:51834 -> 192.168.80.8:5672)
rabbitmq-1  | 2022-10-01T15:29:31.033360772Z 2022-10-01 15:29:31.032492+00:00 [error] <0.837.0> Error on AMQP connection <0.837.0> (192.168.80.11:51834 -> 192.168.80.8:5672, state: starting):
rabbitmq-1  | 2022-10-01T15:29:31.033377193Z 2022-10-01 15:29:31.032492+00:00 [error] <0.837.0> PLAIN login refused: user 'default' - invalid credentials
rabbitmq-1  | 2022-10-01T15:29:31.033380459Z 2022-10-01 15:29:31.032733+00:00 [info] <0.837.0> closing AMQP connection <0.837.0> (192.168.80.11:51834 -> 192.168.80.8:5672)

```

I've narrowed it down to the release 3.9.19 causing the issue (3.9.18 still works), which seems to have a "bug fix" regarding the user/definitions.

My docker compose setup is fairly straightforward:

```yml

version: '3.9'

services:

  rabbitmq:
    image: rabbitmq:${RABBITMQ_IMAGE_VERSION}
      hostname: rabbitmq
      security_opt:
        - no-new-privileges:true
      <<: *restart-always
      environment:
        RABBITMQ_DEFAULT_USER: ${MESSENGER_TRANSPORT_USER}
        RABBITMQ_DEFAULT_PASS: ${MESSENGER_TRANSPORT_SECRET}
      expose:
        - 5672
        - 15672
      volumes:
        - ./docker/rabbitmq/definitions.json:/etc/rabbitmq/definitions.json
        - ./docker/rabbitmq/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
        - rabbitmq_data:/var/lib/rabbitmq
        networks:
          - proxy

volumes:

  rabbitmq_data:

    driver: local

networks:

  proxy:

    external: true

```

My `definitions.json`:

```json

{
"vhosts": [
{
"name": "/"
}
],
"parameters": [],
"global_parameters": [
{
"name": "cluster_name",
"value": "rabbitmq_cs"
}
],
"policies": [],
"queues": [
{
"name": "messages_normal",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {}
},
{
"name": "messages_high",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {}
},
{
"name": "messages_cron",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {}
},
{
"name": "messages_mail",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {}
}
],
"exchanges": [
{
"name": "messages",
"vhost": "/",
"type": "direct",
"durable": true,
"auto_delete": false,
"internal": false,
"arguments": {}
}
],
"bindings": [
{
"source": "messages_normal",
"vhost": "/",
"destination": "messages",
"destination_type": "queue",
"routing_key": "*",
"arguments": {}
},
{
"source": "messages_high",
"vhost": "/",
"destination": "messages",
"destination_type": "queue",
"routing_key": "*",
"arguments": {}
},
{
"source": "messages_cron",
"vhost": "/",
"destination": "messages",
"destination_type": "queue",
"routing_key": "*",
"arguments": {}
},
{
"source": "messages_mail",
"vhost": "/",
"destination": "messages",
"destination_type": "queue",
"routing_key": "*",
"arguments": {}
}
]
}

```

My `rabbitmq.conf`

```

vm_memory_high_watermark.relative = 0.7
vm_memory_high_watermark_paging_ratio = 0.8
log.file = rabbit.log
log.dir = /var/log/rabbitmq
log.default.level = error
log.file.level = debug
log.connection.level = info
# rotate when the file reaches 10 MiB
log.file.rotation.size = 10485760
# keep up to 5 archived log files in addition to the current one
log.file.rotation.count = 5
# definitions
management.load_definitions = /etc/rabbitmq/definitions.json

```

And in the `.env` file

```

RABBITMQ_IMAGE_VERSION="3.9.19-management"

MESSENGER_TRANSPORT_USER="default"
MESSENGER_TRANSPORT_SECRET="password"

```

Additional information:

Ubuntu 20.04

Docker Compose V2.10.2

If more information is required, please let me know.

Regards,

ToshY

[Luke Bakken]

Hello,

Thanks for all of the information. Going forward, it's always preferable to provide a source code repository that can be cloned with instructions how to use the code you've provided. I just spent some time importing your information into this repository, when I could have been investigating your issue instead:

https://github.com/lukebakken/rabbitmq-users-3EXvhb8uLnU

At any rate, there is a problem with what you have provided. Please fork my repository and provide a pull request with a fix.

Here is a transcript from my Arch Linux workstation:

lbakken@shostakovich ~/development/lukebakken/rabbitmq-users-3EXvhb8uLnU (main =)
$ docker compose up
yaml: line 6: mapping values are not allowed in this context

lbakken@shostakovich ~/development/lukebakken/rabbitmq-users-3EXvhb8uLnU (main =)
$ docker compose version
Docker Compose version 2.11.1

lbakken@shostakovich ~/development/lukebakken/rabbitmq-users-3EXvhb8uLnU (main =)
$ docker version
Client:
 Version:           20.10.18
 API version:       1.41
 Go version:        go1.19.1
 Git commit:        b40c2f6b5d
 Built:             Sat Sep 10 11:31:10 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.18
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.19.1
  Git commit:       e42327a6d3
  Built:            Sat Sep 10 11:30:17 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6.m
 runc:
  Version:          1.1.4
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[Luke Bakken]

For what it's worth, running RabbitMQ from the v3.9.x branch (which is pretty much equivalent to version 3.9.23) using this command ...

make RABBITMQ_DEFAULT_USER=foobar RABBITMQ_DEFAULT_PASS=bazbat PLUGINS='rabbitmq_management rabbitmq_top' run-broker

...results in the default user/pass being set correctly:

lbakken@shostakovich ~/development/rabbitmq/rabbitmq-server_v3.9.x (v3.9.x =)
$ ./sbin/rabbitmqctl authenticate_user foobar bazbat
Authenticating user "foobar" ...
Success

...which means something in docker isn't propagating the environment correctly. I'll continue investigating but it would be appreciated to have a fix in the files you provided. If nothing else then we'll be starting from the same point.

Thanks,

Luke

[Tosh Y]

Hey Luke,

Thanks for the reply. I wasn't aware of the way of working/reproducing for RabbitMQ as I came from Github to post it here.

Anyway, I've created a PR on the repository you've linked. Seems that it was mostly indentation issues.

Also added an example python script I got from the RabbitMQ docs, and supplied a DSN.

Regards,

[Luke Bakken]

Hmm yes something is up with the docker image that doesn't exist . Even using rabbitmq:3-management, those env vars aren't working as expected. The following is after running "docker compose up" -

$ docker compose exec rabbitmq rabbitmqctl eval 'os:getenv("RABBITMQ_DEFAULT_USER").'
"default"

$ docker compose exec rabbitmq rabbitmqctl eval 'os:getenv("RABBITMQ_DEFAULT_PASS").'
"password"

$ docker compose exec rabbitmq rabbitmqctl authenticate_user default password
Authenticating user "default" ...
Error:
Error: failed to authenticate user "default"

user 'default' - invalid credentials

$ docker compose exec rabbitmq rabbitmqctl list_users
Listing users ...

Continuing my investigation! Thanks for the PR.

[Luke Bakken]

OK I learned something today...

If you are doing any definitions import, RabbitMQ will disregard any default user or default password settings in the environment or via rabbitmq.conf

https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/src/rabbit.erl#L1040-L1051

So that means you need to ensure that your users are specified in your definitions.json file. This adds a bit of a wrinkle because the password must be hashed in this file:

"name":"default", "password_hash":"l7E36KOLOLoXYvKBXoEt2WizbhbQ9Wv9FZkFofC+wYVT6FR1"

Here are some ways to generate the hash:

https://stackoverflow.com/questions/41306350/how-to-generate-password-hash-for-rabbitmq-management-http-api

Thanks. I'll ask the team about the best way to document this behavior.

Luke

[Tosh Y]

Hey Luke,

Thank you for further investigating the issue. 

Regarding your first reply: 

• The conclusion is the same as I initially stated. I've narrowed it down to breaking in 3.9.19, as releases =<3.9.18 still work. 

Regarding you're second reply:

• If you are doing any definitions import, RabbitMQ will disregard any default user or default password settings in the environment or via rabbitmq.conf

This doesn't really add up when you take into account that it worked prior to 3.9.19. Doing a git blame on the file shows that part of code hasn't been touch in roughly 2 years.

If this is indeed intended to work like you stated, then it probably has been broken for quite a while and fixed in 3.9.19. My best guess would be that it's related to the first point in the bugfixes list of 3.9.19.

 Regarding both generating and using the hash inside the definitions.json:

1. Generating a hash seems a bit cumbersome when looking at the steps provided in the documentation. The SO question you referenced has tons of answer for different languages, making it quite hard to figure out what's good, bad or easy to use. In order to make it easy to use, I personally would create a custom docker image for it, so that you can just call the script in the container with your plain text password as argument, and it will return the hash. Maybe this would be a nice addition to have for RabbitMQ (I'll probably take a shot at it myself).
2. I want my definitions.json free of secrets, so that includes at the very the least the password. So the ability to supply environment variables alongside existing/custom definitions seems to be a sensible thing to have/add.

Anyway if it's indeed intended behavior to either use custom definitions or environment variables, but not both, than this should indeed be stated somewhere in the docs to avoid further confusion.

Regards

[Luke Bakken]

Hello -

 

This doesn't really add up when you take into account that it worked prior to 3.9.19.

It worked as long as you used the default values / env vars with the management.load_definitions setting. There is another way to load definitions that doesn't use the management plugin that would have exhibited the issue.

 

If this is indeed intended to work like you stated, then it probably has been broken for quite a while and fixed in 3.9.19. My best guess would be that it's related to the first point in the bugfixes list of 3.9.19.

This pull request added the management.load_definitions setting to those that override the default values - https://github.com/rabbitmq/rabbitmq-server/pull/4904

 

 Regarding both generating and using the hash inside the definitions.json:

1. Generating a hash seems a bit cumbersome when looking at the steps provided in the documentation. The SO question you referenced has tons of answer for different languages, making it quite hard to figure out what's good, bad or easy to use. In order to make it easy to use, I personally would create a custom docker image for it, so that you can just call the script in the container with your plain text password as argument, and it will return the hash. Maybe this would be a nice addition to have for RabbitMQ (I'll probably take a shot at it myself).

Generating a hashed password should be something that rabbitmqctl and perhaps the HTTP API supports - https://github.com/rabbitmq/rabbitmq-server/issues/5957

1. I want my definitions.json free of secrets, so that includes at the very the least the password. So the ability to supply environment variables alongside existing/custom definitions seems to be a sensible thing to have/add.

I can bring this up with the team.

One workaround that you could use would be to add the user via a custom docker entrypoint -

https://github.com/lukebakken/rabbitmq-users-3EXvhb8uLnU/commit/a9c151d7a00fffa3de67eb2151db3b1f52c79aab

 

Anyway if it's indeed intended behavior to either use custom definitions or environment variables, but not both, than this should indeed be stated somewhere in the docs to avoid further confusion.

Yeah, I'll check the docs tomorrow.

Thanks,

Luke 

[Michal Kuratczyk]

Yes, it was broken for quite some time and got fixed in 3.9.19. The docs (https://rabbitmq.com/definitions.html#import-on-boot) say:

"if a blank (uninitialised) node imports a definition file, it will not create the default virtual host and user."

That was true when load_definitions was used but not true when the legacy management.load_definitions was used.

Starting with 3.9.19, the behaviour is consistent and in-line with the documentation.

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/f3850371-1120-4c07-98fb-a2128a40cf0en%40googlegroups.com.

--

Michał

RabbitMQ team

[Tosh Y]

Hey Luke & Michał,

Thank you both for the explanation and clarification. 

I have enough information to continue now.

Regards