amq.rabbitmq.log exchange not available by default in RabbitMQ version 3.8.3 installation

[Apoorv Verma]

Hello Experts,

I was performing RabbitMQ version upgrade from Version 3.6.10 to Version 3.8.3 and have this observation 

amq.rabbitmq.log exchange is not present in fresh installation of version 3.8.3

wheras the same is present by default in a fresh installation of version 3.6.10

Can you please let me know if this exchange has been removed from latest version?

I googled around this and found that adding in /etc/rabbitmq/rabbitmq.conf the following property

log.exchange = true 

and restarting rabbitMQ server will enable this exchange.

But I tried this and it did not work. 

Please help

[Luke Bakken]

Hello,

The log exchange is correctly set up in my 3.8.3 environment when I use the attached configuration file.

When you say "it did not work" you have to be more specific. Do you not see the exchange in the list of exchanges?

Thanks -

Luke

[Apoorv Verma]

Hi Luke,

Yes I could not see the exchange in the list of exchanges . Attaching my rabbitmq.conf file and screenshots of overview page and exchanges page

Best Regards,

Apoorv

[Luke Bakken]

Hello,

Are you certain that your configuration file is being read and used? RabbitMQ logs the effective configuration file when it starts.

Please attach your complete RabbitMQ log file.

Thanks -

Luke

[Apoorv Verma]

Hi Luke ,

I checked the log file. It reads rabbitmq.config file. Following is the information printed in the log. Also attaching the complete log file.

 node           : rabbit@keng03-dev01-ins01-dmq60-app-1588007285-1

 home dir       : /var/lib/rabbitmq

 config file(s) : /etc/rabbitmq/rabbitmq.config

 cookie hash    : 7bQpZP+1ROmBsTuTSibbeA==

 log(s)         : /data/rabbitmq/logs/rab...@keng03-dev01-ins01-dmq60-app-1588007285-1.log

                : /data/rabbitmq/logs/rabbit@keng03-dev01-ins01-dmq60-app-1588007285-1_upgrade.log

 database dir   : /data/rabbitmq/data/mnesia/rabbit@keng03-dev01-ins01-dmq60-app-1588007285-1

But the change is proposed in rabbitmq.conf. When I made the same change in rabbitmq.config and restarted my docker container (rabbitmq server), the container did not come up .

[Luke Bakken]

Hello,

I don't see evidence in the log of RabbitMQ not coming up. It appears to have started correctly.

In any case, you must use either the /etc/rabbitmq/rabbitmq.conf file, or the /etc/rabbitmq/rabbitmq.config file, but not both.

You will have to figure out why the correct file is not being used in your docker environment.

Thanks -

Luke

On Monday, April 27, 2020 at 10:43:58 AM UTC-7, Apoorv Verma wrote:

Hi Luke ,

I checked the log file. It reads rabbitmq.config file. Following is the information printed in the log. Also attaching the complete log file.

 node           : rabbit@keng03-dev01-ins01-dmq60-app-1588007285-1

 home dir       : /var/lib/rabbitmq

 config file(s) : /etc/rabbitmq/rabbitmq.config

 cookie hash    : 7bQpZP+1ROmBsTuTSibbeA==

 log(s)         : /data/rabbitmq/logs/rabbit@keng03-dev01-ins01-dmq60-app-1588007285-1.log

                : /data/rabbitmq/logs/rabbit@keng03-dev01-ins01-dmq60-app-1588007285-1_upgrade.log

 database dir   : /data/rabbitmq/data/mnesia/rabbit@keng03-dev01-ins01-dmq60-app-1588007285-1

[Apoorv Verma]

Hi Luke,

Thanks for the helpful reponse. 

I removed the rabbitmq.config file from my environment and as such when I deployed a fresh RabbitMQ v3.8.3 I could see the 'amq.rabbitmq.log'  exchange in UI.

But started getting following error in log file - 

2020-04-30 08:07:59.672 [error] <0.151.0> Could not declare exchange 'amq.rabbitmq.log' in vhost '/', reason: exit:{aborted,{no_exists,[rabbit_vhost,<<"/">>]}}

Clearly the log says that exchange cannot be created on default / vhost as it is non existent. But I could see / host present through rabbitmqctl list_host command.

Attaching the log file for reference.

Please help

[Luke Bakken]

Hello,

Yes, you may see that message, but the declaration of the exchange is re-tried during boot. I saw the message in my environment as well. If the exchange declaration fails, it will be re-tried on the next log message that happens at least 5 seconds after the previous log message. If you don't see the exchange in your environment, wait five seconds, and run this command to log a message:

rabbitmqctl eval 'rabbit_log:info("LOG MESSAGE").'

I have attached the configuration file I used, and a screenshot showing the exchange. I am using RabbitMQ 3.8.3 on FreeBSD 12, Erlang 22.3.3

Thanks,

Luke

[Apoorv Verma]

Hi Luke,

Thanks for the quick reply.

The exchange is visible in the UI  but the issue is logs are not getting printed further after this error :

Could not declare exchange 'amq.rabbitmq.log' in vhost '/', reason: throw:{error,{no_such_vhost,<<"/">>}}

I imported a large metadata file after this error and logs are not getting printed like previously on all the 3 nodes.

Attaching the logs of 3 nodes and the crash.log for your reference.

Regards,

Apoorv 

[Luke Bakken]

Hello,

What do you mean by "import a large metadata file" ? Do you mean you imported definitions? Can you share that file?

I need a reliable method to reproduce this, because I can't.

Thanks -

Luke

[Luke Bakken]

Hello,

Can you describe the environment in which your three-node cluster is running? What is the hardware or virtual machine specs like? I'm wondering if your environment is much slower than mine, causing a race condition at startup.

Thanks -

Luke

[Apoorv Verma]

Hi Luke ,

There is an observation over here.

We had replaced the rabbitmq.config with new rabbitmq.conf and advanced.config files . However we failed to remove the rabbitmq.config from the deployment.

So all three files were present and when me made that exchange change in rabbitmq.conf , changes were not picking up since RabbitMQ was reading rabbitmq.config which was also present . And in this case logging was happening properly.

But as you clearly pointed out both config files cannot be present, so I removed rabbitmq.config file and now the exchange was visible but logs were not getting printed properly.

Just now when i placed back rabbitmq.config file again , logging was getting printed properly again.

Attaching the three config files for your reference.

The logging issue is related to config file. Logging is properly working with rabbitmq.config but not with rabbitmq.conf.

Regards,

Apoorv

[Apoorv Verma]

Hi Luke , 

I am using n1-standard-8 , 8 CPUs 30 GB Memory deployment. 

Can you please suggest around this logging not working with new configuration file observation.

I mean logging should work and also new file rabbitmq.conf must be read. 

Regards,

Apoorv

[Luke Bakken]

Apoorv -

Neither the rabbitmq.config nor the rabbitmq.conf file you sent me will enable the log exchange - you don't have the correct settings. I don't know how it is working in your environment, but you must be using a different rabbitmq.config file than what you sent me.

Please re-read this message from me -

https://groups.google.com/d/msg/rabbitmq-users/fNzjkyZIVRU/0qnWNVOoBgAJ

I attached a conf file that will enable the exchange.

Thanks -

Luke

[Apoorv Verma]

Hi Luke,

Initial Issue/Problem 

The log exchange was not visible and as such the following entry was made in rabbitmq.conf file 

log.exchange = true

But again the change was not getting reflected, since rabbitmq.config was also present in the enviornment.

Solution : Remove the rabbitmq.config from the enviornment , and now the log exchange was visible

But this led to a second issue 

Second Issue/Problem 

Logging not getting printed properly

Approaches tried 

1. Remove the log.exchange = true entry made in rabbitmq.conf. This did not work , logging still not getting printed properly

2. Place back the rabbitmq.config back in the the environment . This worked and now logging getting printed properly.

So yes we know

1. log exchange would be visible by this entry log.exchange = true in rabbitmq.conf. But logging should also work which is not true in this case.

2. log exchange would not be visible with rabbitmq.config back in the enviornment , But logging works properly in this case

Ideal solution -

• exchange should be visible
• logging should also work properly
• rabbitmq.config should not be present in the environment

Please suggest necessary changes in rabbitmq.conf and advanced.config files around this.

Note : I had removed the exchange entry from rabbitmq.conf  only to check if logging is working properly without it like previously .

[Luke Bakken]

Please carefully re-read my message -

https://groups.google.com/d/msg/rabbitmq-users/fNzjkyZIVRU/0qnWNVOoBgAJ

* Back up and then delete all of your configuration files from /etc/rabbitmq

* Save the file attached to that message (rabbitmq-log-exchange.conf) as /etc/rabbitmq/rabbitmq.conf

* Ensure there are NO OTHER configuration files in /etc/rabbitmq

* Start RabbitMQ.

If you're testing this on your cluster, all nodes must have the above changes made.