Changing log levels in rabbitmq

[Gunjan Thapliyal]

Hi

Changing log levels using the config file works well. Only trouble I am having is to restart rabbitmmq every time I make a change.

Changing log level from info to debug without restarting the application is what I am looking for, so that I can collect more ogs when the server is failing.  Is there a way to do so, at least for the log level?

Thanks

Gunjan

[Michael Klishin]

The answer greatly depends on what RabbitMQ version is used and unfortunately that bit is missing from your post.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Gunjan Thapliyal]

Running rabbitmq 3.7.7-alpine docker image.

Making changes to log level via the rabbitmq advanced config file. Changes to log level reflecting only adter a restart

Log configuration looks like this

 {log, [

             {file, [{file, "/var/log/rabbitmq/rabbit.log"}, %% log.file

                     {level, error},        %% log.file.info

                     {date, "$D0"},           %% log.file.rotation.date

                     {size, 1048576},            %% log.file.rotation.size

                     {count, 10}            %% log.file.rotation.count

                     ]}

              ]

[Michael Klishin]

With 3.7.x and its logging framework it is possible to change log level at runtime [1]. With earlier

versions I'm pretty sure all logger settings were set in stone at boot time.

I personally haven't tried it so I'd need to do some experimenting first.

1. https://github.com/erlang-lager/lager#runtime-loglevel-changes

[Gunjan Thapliyal]

Thanks for the info Michael.

While I try it myself, can you comment if modifying lager configuration the way you pointed requires writing some erlang code or plugin. So far, I have not gone beyond editing config files and using rabbitmqctl command to tune rabbitmq.

Thanks

Gunjɐn

[Michael Klishin]

It can be invoked using `rabbitmqctl eval`. We just need to test it before recommending anything :)

[Gunjan Thapliyal]

Thanks. It is okay to suggest based on your expertise. I did try the command but doesn't work. I see this

bash-4.4$ rabbitmqctl eval 'lager:set_loglevel(lager_console_backend, debug).'

{error,bad_module}

I tried to set ERL_COMPILER_OPTIONS and RABBITMQ_CTL_ERL_ARGS based on some searches to 

ERL_COMPILER_OPTIONS='{parse_transform, lager_transform}'

But the command still doesnt work. Is the lager command to be enabled differently to do changes on runtime?

Thanks

Gunjan

[Michael Klishin]

Lager is a library. Instead of setting things randomly found on the Internet (some are compiler flags) just

wait until we have a chance to try it out. The only thing that's needed is

 * A call to the Lager API in the version we use

 * A correct sink name

[Gunjan Thapliyal]

Absolutely. That should make things simple for a user. Hoping to see an easy way to do this in rabbits soon. Thanks for your help.. 

[Luke Bakken]

Hi Gunjan,

You can't change the log level for lager_console_backend because you do not have it active at that point in time. Plus, you won't see the output since RabbitMQ is running as a service.

If you'd like to change the log level to debug for the log file, run this command:

rabbitmqctl eval '{lager_file_backend, F} = proplists:lookup(lager_file_backend, gen_event:which_handlers(lager_event)), lager:set_loglevel(lager_event, lager_file_backend, F, debug), lager:set_loglevel(rabbit_log_lager_event, lager_forwarder_backend, undefined, debug).'

To revert back to info, change all instances of debug to info in the above command and re-run it.

Thanks,

Luke

[Michael Klishin]

FTR RabbitMQ 3.7.10 will have a CLI command that hides all of that jazz from you. Not sure why we haven't done this earlier :)

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Johan Rhodin]

Hi,

I'm unable to set the warning level with rabbitmqctl eval '{lager_file_backend, F} = proplists:lookup(lager_file_backend, gen_event:which_handlers(lager_event)), lager:set_loglevel(lager_event, lager_file_backend, F, warning), lager:set_loglevel(rabbit_log_lager_event, lager_forwarder_backend, undefined, warning).', is there another way to do that? (Without upgrading to 3.7.10+ and use the new CLI command)

/Johan

[Luke Bakken]

Hi Johan,

The new CLI command basically executes the same functions you did.

When you say "unable to set the warning level" what exactly does that mean? Do you see errors? What are you expecting?

Also, it's very important to let us know what version of RabbitMQ and Erlang you're using.

Thanks,

Luke

[Johan Rhodin]

The system I originally experienced this on was 3.7.9/21.2.3. Today I tried reproducing on RabbitMQ 3.7.9 on Erlang 21.2.3, but can't. Sorry about the noise.

Switching to/from info and debug emits a [notice] but I assume it is to be expected that the notice doesn't show for warning/error/critical/none levels?

2019-03-10 18:53:54.272 [notice] <0.105.0> Changed loglevel of /var/log/rabbitmq/rab...@node-01.log to debug
2019-03-10 18:54:01.299 [debug] <0.982.0> User 'admin' authenticated successfully by backend rabbit_auth_backend_internal
2019-03-10 18:54:23.703 [notice] <0.105.0> Changed loglevel of /var/log/rabbitmq/rab...@node-01.log to info

Johan

[Johan Rhodin]

Now I understand my own confusion. I mixed up "lager_file_backend" with the one for the exchange. My question is, what would be the command to change the log level for logs sent to "amq.rabbitmq.log"?

Johan

[Luke Bakken]

Hi Johan,

Could you please provide your complete RabbitMQ configuration? Or, just the complete lager config?

On Monday, March 11, 2019 at 9:12:20 AM UTC-7, Johan Rhodin wrote:

Now I understand my own confusion. I mixed up "lager_file_backend" with the one for the exchange. My question is, what would be the command to change the log level for logs sent to "amq.rabbitmq.log"?

Johan

[Johan Rhodin]

Attaching the lager config and the log part as reported by "rabbitmqctl environment".

[Luke Bakken]

Hi Johan,

I missed the part where you're using the built-in exchange amq.rabbitmq.log for log events. To change that to debug, run this:

rabbitmqctl eval 'lager:set_loglevel(lager_exchange_backend, debug).'

Let me know how that works for you.

Luke

On Monday, March 11, 2019 at 9:12:20 AM UTC-7, Johan Rhodin wrote:

Now I understand my own confusion. I mixed up "lager_file_backend" with the one for the exchange. My question is, what would be the command to change the log level for logs sent to "amq.rabbitmq.log"?

Johan

> On Mar 10, 2019, at 2:39 PM, Johan Rhodin <jo...@cloudamqp.com> wrote:
>
> The system I originally experienced this on was 3.7.9/21.2.3. Today I tried reproducing on RabbitMQ 3.7.9 on Erlang 21.2.3, but can't. Sorry about the noise.
>
> Switching to/from info and debug emits a [notice] but I assume it is to be expected that the notice doesn't show for warning/error/critical/none levels?
>

> 2019-03-10 18:53:54.272 [notice] <0.105.0> Changed loglevel of /var/log/rabbitmq/rabbit@node-01.log to debug

> 2019-03-10 18:54:01.299 [debug] <0.982.0> User 'admin' authenticated successfully by backend rabbit_auth_backend_internal

> 2019-03-10 18:54:23.703 [notice] <0.105.0> Changed loglevel of /var/log/rabbitmq/rabbit@node-01.log to info

>
> Johan
>
>> On Mar 9, 2019, at 10:18 AM, Luke Bakken <lba...@pivotal.io> wrote:
>>
>> Hi Johan,
>>
>> The new CLI command basically executes the same functions you did.
>>
>> When you say "unable to set the warning level" what exactly does that mean? Do you see errors? What are you expecting?
>>
>> Also, it's very important to let us know what version of RabbitMQ and Erlang you're using.
>>
>> Thanks,
>> Luke
>>
>> On Saturday, March 9, 2019 at 6:21:39 AM UTC-8, Johan Rhodin wrote:
>> Hi,
>>
>> I'm unable to set the warning level with rabbitmqctl eval '{lager_file_backend, F} = proplists:lookup(lager_file_backend, gen_event:which_handlers(lager_event)), lager:set_loglevel(lager_event, lager_file_backend, F, warning), lager:set_loglevel(rabbit_log_lager_event, lager_forwarder_backend, undefined, warning).', is there another way to do that? (Without upgrading to 3.7.10+ and use the new CLI command)
>>
>> --
>> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

>> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to rabbitmq-users@googlegroups.com.

[Johan Rhodin]

Yes, that did it!
Thank you very much Luke.

Johan

> On Mar 14, 2019, at 10:20 AM, Luke Bakken <lba...@pivotal.io> wrote:
>
> Hi Johan,
>

> I missed the part where you're using the built-in exchange amq.rabbitmq.log for log events. To change that to debug, run this:
>
> rabbitmqctl eval 'lager:set_loglevel(lager_exchange_backend, debug).'
>
> Let me know how that works for you.
>
> Luke
>
> On Monday, March 11, 2019 at 9:12:20 AM UTC-7, Johan Rhodin wrote:
> Now I understand my own confusion. I mixed up "lager_file_backend" with the one for the exchange. My question is, what would be the command to change the log level for logs sent to "amq.rabbitmq.log"?
>
> Johan
>
> > On Mar 10, 2019, at 2:39 PM, Johan Rhodin <jo...@cloudamqp.com> wrote:
> >
> > The system I originally experienced this on was 3.7.9/21.2.3. Today I tried reproducing on RabbitMQ 3.7.9 on Erlang 21.2.3, but can't. Sorry about the noise.
> >
> > Switching to/from info and debug emits a [notice] but I assume it is to be expected that the notice doesn't show for warning/error/critical/none levels?
> >

> > 2019-03-10 18:53:54.272 [notice] <0.105.0> Changed loglevel of /var/log/rabbitmq/rab...@node-01.log to debug

> > 2019-03-10 18:54:01.299 [debug] <0.982.0> User 'admin' authenticated successfully by backend rabbit_auth_backend_internal

> > 2019-03-10 18:54:23.703 [notice] <0.105.0> Changed loglevel of /var/log/rabbitmq/rab...@node-01.log to info

> >
> > Johan
> >
> >> On Mar 9, 2019, at 10:18 AM, Luke Bakken <lba...@pivotal.io> wrote:
> >>
> >> Hi Johan,
> >>
> >> The new CLI command basically executes the same functions you did.
> >>
> >> When you say "unable to set the warning level" what exactly does that mean? Do you see errors? What are you expecting?
> >>
> >> Also, it's very important to let us know what version of RabbitMQ and Erlang you're using.
> >>
> >> Thanks,
> >> Luke
> >>
> >> On Saturday, March 9, 2019 at 6:21:39 AM UTC-8, Johan Rhodin wrote:
> >> Hi,
> >>
> >> I'm unable to set the warning level with rabbitmqctl eval '{lager_file_backend, F} = proplists:lookup(lager_file_backend, gen_event:which_handlers(lager_event)), lager:set_loglevel(lager_event, lager_file_backend, F, warning), lager:set_loglevel(rabbit_log_lager_event, lager_forwarder_backend, undefined, warning).', is there another way to do that? (Without upgrading to 3.7.10+ and use the new CLI command)
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

> >> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> >> To post to this group, send email to rabbitm...@googlegroups.com.

> >> For more options, visit https://groups.google.com/d/optout.
> >
>
>

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send email to rabbitm...@googlegroups.com.