Logging in rabbitmq: cannot see amq.log.exchange

[Hisham Najem]

Hello,

I am planning to consume logging information via logstash plugin and then ship to elasticsearch.

I am using rabbitmq v 3.9.13 and cannot see the amq log exchange when setting log.exchange = true and log.exchange.level = info.

Any other information required I can provide.

Thank you.

[Wes Peng]

How did you configure and setup the architecture?

Have you enabled the plugin?

Thanks 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/1d982cfa-e323-4ac0-9df2-a3c79f92d4c6n%40googlegroups.com.

[Hisham Najem]

Is there a plugin for log exchange?

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/j1AY7lBWI6c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CADwNPEfNCM2xj1gSwGfk30Q-PGWcG0-P5HvAW4%2Bjjt%2BX8na-jA%40mail.gmail.com.

[Wes Peng]

I don’t know but you can give a internet search.

And since you were using logstash there are many solutions for log transfer. For instance,

https://logz.io/learn/complete-guide-elk-stack/amp/

https://logz.io/blog/logstash-plugins/

Thanks 

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CACdsx4FWFNBi8yxe%3D9VG4X9wuvnOOSneYgFCgtGZJLru6qNvkg%40mail.gmail.com.

[Hisham Najem]

The logstash and elk side of things are fine. I'm trying to get the amq.log.exchange to appear in my RMQ.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CADwNPEem5QzVFtJrkmq1AMYs7vhvssDyZQpnn2madLjjSYCL%2Bw%40mail.gmail.com.

[Luke Bakken]

Hello,

You must create the rabbitmq.conf file and have at least this in it:

log.exchange = true

https://github.com/rabbitmq/rabbitmq-server/blob/master/deps/rabbit/docs/rabbitmq.conf.example#L943-L949

The name of the exchange is amq.rabbitmq.log

Funny enough, the last time I answered this question I noticed that it's not documented - https://stackoverflow.com/a/57965847/1466825

https://github.com/rabbitmq/rabbitmq-website/issues/1370

Thanks,

Luke

[Hisham Najem]

Hey Luke,

Thanks for your reply. I do have that in my .conf file! Yet the exchange still does not appear!

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/j1AY7lBWI6c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/8335cc47-8793-41ce-8584-e53ccaac5013n%40googlegroups.com.

[Luke Bakken]

Hello,

I just started RabbitMQ 3.9.13 with the attached configuration file. Since all other logging is disabled, you won't see the message saying that "amq.rabbitmq.log" is ready in the log file. You can see the exchange via rabbitmqctl list_exchanges

rabbitmqctl list_exchanges
Listing exchanges for vhost / ...
name    type
        direct
amq.rabbitmq.log        topic
amq.direct      direct
amq.headers     headers
amq.topic       topic
amq.match       headers
amq.rabbitmq.trace      topic
amq.fanout      fanout

Are you certain that you have your configuration file in the correct place?

Thanks,

Luke

[Hisham Najem]

Hello,

Here is my /etc/rabbitmq/rabbitmq.conf:

#############################################################################
cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = ip
cluster_formation.node_cleanup.interval = 30
cluster_formation.node_cleanup.only_log_warning = false
cluster_partition_handling = autoheal
management.path_prefix = /rabbitmq
management.load_definitions = /etc/rabbitmq/rabbitmq_definitions.json
loopback_users = none
log.console = false
log.file = false
log.exchange = true
log.exchange.level = info

#############################################################################

Here is output of rabbitmqctl list_exchanges

rabbitmqctl list_exchanges
RABBITMQ_ERLANG_COOKIE env variable support is deprecated and will be REMOVED in a future version. Use the $HOME/.erlang.cookie file or the --erlang-cookie switch instead.

Listing exchanges for vhost / ...
name    type

restadapter     direct
pushadapter     direct
workflow-status-listener        direct
restamqpgw      direct
websocketadapter        direct
amq.match       headers
amq.fanout      fanout
filetransfer    direct
amq.rabbitmq.trace      topic
websocketadapter-unallocated    fanout
amq.headers     headers
ProvisionExchange       fanout
amq.rabbitmq.event      topic
onedotfivebridge        direct
restadapter-unrouted    x-consistent-hash
onedotfivebridge-unrouted       x-consistent-hash
filetransfer-unallocated        fanout
pushadapter-unrouted    x-consistent-hash
        direct
restamqpgw-unrouted     x-consistent-hash
notificationbroker-unrouted     x-consistent-hash
amq.topic       topic
notificationbroker      direct
amq.direct      direct


----------------------------------------------------------------------------------------------------------------------------------

I am loading definitions from rabbitmq_definitions.json so perhaps I have to enable the log exchange through that file. Any idea how I'd add that to the json definitions file?

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/j1AY7lBWI6c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/23540fbe-2013-4c26-b122-b08636ff8ce4n%40googlegroups.com.

[Luke Bakken]

I am loading definitions from rabbitmq_definitions.json so perhaps I have to enable the log exchange through that file. Any idea how I'd add that to the json definitions file?

No, that is not necessary.

Please add these settings to your rabbitmq.conf file and restart RabbitMQ:

log.file = true

log.file.level = debug

Once RabbitMQ has started get the log file and attach the complete file to your response. Do not paste the file contents here!

Thank you,

Luke

[Hisham Najem]

Hello, I've added those to the conf file, but do not see a log file under /var/log/rabbitmq

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/j1AY7lBWI6c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/699feeb0-bd5d-417f-9407-0d46c0137532n%40googlegroups.com.

[Hisham Najem]

Rather I see an empty rab...@X.X.X.X_upgrade.log under /var/log/rabbitmq

[Luke Bakken]

All of this indicates that RabbitMQ is not picking up your configuration file, or it's not seeing the changes you make.

Try re-enabling the console log:

log.console = true

log.console.level = debug

When you restart your RabbitMQ container, you should be able  to use k8s / docker commands to see the logs.

It is helpful at this point to share ALL of your configuration... RabbitMQ, kubernetes, anything that could assist.

Thanks,

Luke

[Hisham Najem]

Let me know if you need any other config info.

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/j1AY7lBWI6c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/645502d9-28aa-47a2-882f-62a9f1046fe2n%40googlegroups.com.

[Luke Bakken]

Hello,

The log file is incomplete and does not show the entire startup sequence. Please give it another try.

When I get time I'll start RabbitMQ locally to point out the log messages I'm looking for.

Luke

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CACdsx4F8VAGcfi%3DTqaBUPyhjKnVbJf6zMNc7bNJt%3DFh%3DoN8zpQ%40mail.gmail.com.

[Hisham Najem]

Hello, let me know if this is enough. Thank you for your time. I appreciate it.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CADFEJueJ2%2BhmM6CKbgbEOn7DYbvOCWjOaUSXrO_gkCZKg5HiPg%40mail.gmail.com.

[Luke Bakken]

Thanks, that log is complete.

I don't see a log message like this:

2022-03-16 13:01:42.425000-07:00 [info] <0.238.0> Logging to exchange 'amq.rabbitmq.log' in vhost '/' ready

...so there must be something up. Could you please run the following command on the running node, redirect the output to a file, and attach the file to your response?

rabbitmqctl environment

Then, please remove the management.load_definitions configuration setting, restart k8s / RabbitMQ to ensure the setting is removed, and see if the amq.rabbitmq.log exchange is created. I'm going to use your definitions locally to see if I can reproduce this issue. For now, everything works as expected on my local machine.

Thanks,

Luke

[Hisham Najem]

Hello,

I have attached output of rabbitmqctl environment.

I've also removed management.load_definitions in config and still do not see the exchange.

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/j1AY7lBWI6c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/5862a3ca-a54b-433c-9d39-b5055e7ceac2n%40googlegroups.com.

[Luke Bakken]

Thanks again for providing the requested information.

I must admit, I'm stumped by this right now. I have confirmed that your configuration and definitions import works fine on both Windows and Linux systems running RabbitMQ 3.9.13 and Erlang 24.3.2

So, if you have access to system where you can run RabbitMQ outside of Kubernetes and a container, we can start there. Preferably, you'd do the following on a Linux system:

• Install Erlang
• Un-archive the following generic-unix package somewhere - https://github.com/rabbitmq/rabbitmq-server/releases/download/v3.9.13/rabbitmq-server-generic-unix-3.9.13.tar.xz
• Copy your rabbitmq.conf to the ./etc/rabbitmq sub-directory of the un-archived generic-unix package. Be sure to enable debug console logging as well as log.exchange (of course).
• Copy your enabled_plugins file to the same location.
• Start RabbitMQ with the following command: 

LOG=debug ./sbin/rabbitmq-server

The log output should contain messages like the following:

2022-03-16 14:19:19.454081-07:00 [debug] <0.502.0> Marking RabbitMQ as running
2022-03-16 14:19:19.454110-07:00 [debug] <0.502.0> Change boot state to `ready`
2022-03-16 14:19:19.454154-07:00 [debug] <0.141.0> Boot state/systemd: notifying of state `ready`
2022-03-16 14:19:19.601599-07:00 [debug] <0.9.0> Time to start RabbitMQ: 2563844 µs

2022-03-16 14:19:23.125243-07:00 [debug] <0.242.0> Declared exchange 'amq.rabbitmq.log' in vhost '/'
2022-03-16 14:19:23.125402-07:00 [info] <0.242.0> Logging to exchange 'amq.rabbitmq.log' in vhost '/' ready

If that works, we know the issue has to do with your k8s environment somehow. If it doesn't, copy the entire console output to a file and attach it to your response.

Thanks,

Luke

[Hisham Najem]

Sure, I'll do that and let you know. Just as an added note, this is the docker image I'm using: 3.9.13-management-alpine.

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/j1AY7lBWI6c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/144214af-0d8a-489e-9848-f67b9f53f792n%40googlegroups.com.

[Luke Bakken]

AHA!

Thanks for that information. Yes, I can reproduce this issue when I use a custom docker image built from the Dockerfile here:

https://github.com/lukebakken/rabbitmq-users-j1AY7lBWI6c

I will continue my investigation and will report back. Don't worry about doing what I asked below unless you'd like to see the logging exchange created successfully :-)

Thanks!

Luke

[Hisham Najem]

Thank you very much! 

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/j1AY7lBWI6c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/f2c75ca5-9a0c-4994-98e3-cd078c2c2bcdn%40googlegroups.com.

[Luke Bakken]

More info -

When you run RabbitMQ from an official docker image, the following env variable is set:

RABBITMQ_LOGS=-

This is what is preventing the logging exchange from starting correctly. I can reproduce the behavior outside of Docker if I run RabbitMQ like this:

RABBITMQ_LOGS=- RABBITMQ_ALLOW_INPUT=true RABBITMQ_CONFIG_FILE="/home/lbakken/development/lukebakken/rabbitmq-users-j1AY7lBWI6c/rabbitmq.conf" LOG=debug ./sbin/rabbitmq-server

So now I just need to figure out a fix for this.

As a workaround, you can build an image with a custom entrypoint that unsets the RABBITMQ_LOGS env variable. I have updated my repo with just that workaround and it DOES fix the issue:

https://github.com/lukebakken/rabbitmq-users-j1AY7lBWI6c/blob/main/Dockerfile

As long as you keep log.console = true the Docker logging will still work as you expect.

I will update this discussion when a final fix is in.

Thank you very much for providing all of the information needed to narrow this down.

Luke

[Hisham Najem]

Thank you indeed, this was very helpful. 

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/229f8024-43eb-4cbd-ac37-756188247f49n%40googlegroups.com.

[Luke Bakken]

Hello again,

Unfortunately this issue does not have a simple fix due to how RabbitMQ logging is configured. I have filed an issue so that I can discuss this with another team member - https://github.com/rabbitmq/rabbitmq-server/issues/4303

For now, your best solution is to use your own Dockerfile to unset the RABBITMQ_LOGS environment variable before RabbitMQ starts.

Thanks,

Luke

[Hisham Najem]

Hey, no worries. I just set the environment variable to blank and it fixed the issue.

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/j1AY7lBWI6c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/56d4810e-a8b3-47e5-b10f-00f88858f1fdn%40googlegroups.com.