What is missing in Logger?

[José Valim]

Folks,

When we started the Logger library, we have brought in many features but some of them were postponed to after 1.0. You can see the full list here:

https://github.com/josevalim/logger

The remaining features we have pending today:

1. Tracing support: it mostly includes support to perform certain actions based on metadata

2. File logging support and log rotation: the ability to log to files and rotate logs

I believe tracing support would be useful and feedback on what you would like to see when it comes to tracing is very welcome. However, I am not convinced on the usefulness of file logging support nor log rotations as tools like logrotate do an excellent job and it is easy to pipe the existing log to a file. Thoughts?

Also, is there any other feature you would like to see in Logger?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

[Robert Stockdale]

Custom log levels per module/logger would be a nice enhancement.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAGnRm4Lzgv8F3q9VU4Q%3DzBVW6wqwFwF4yGnU%2BntkS1CgVJmQTQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Alexei Sholik]

I remember there was something in OTP for managing log files and their rotation. Is that tightly couple with error_logger so that we cannot use it with our Logger?

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4Lzgv8F3q9VU4Q%3DzBVW6wqwFwF4yGnU%2BntkS1CgVJmQTQ%40mail.gmail.com.

[José Valim]

There is one but we can't use with our logger. I have also heard of many recommendations for not using it, because it rotates files in a very specific way, with odd naming conventions, which is not what most people would expect.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

--
You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAAPY6eOCJ-W9bnRaZmHwq1B7Qp0xB4e%3Dshr1gMDUNzTETBqdLQ%40mail.gmail.com.

[Booker Bense]

Maybe I'm hopelessly old school, but I'd like logs to go to syslog. 

I would like to see at least a couple choices of backend in the standard

library. File, syslog, stdin to an external process. Getting logging data 

into your monitoring system is a key feature for modern applications. 

While it seems straightforward enough to "just roll your own" for this,

it's something I would expect of any standard logging API. 

Interacting effectively with logrotate also requires a bit of twiddling 

to reset the log based on an external command. It would be nice to

standardize that. As someone that most runs other people's code for

a living I hate it when they create non-standard ways to do logging. 

- Booker C. Bense 

[José Valim]

Agreed syslog is useful. There is a backend end for it too:

https://github.com/smpallen99/syslog

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

--

You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/55782096-fc7e-44b1-b413-c53d092aa5df%40googlegroups.com.

[Peter Hamilton]

+1 for using logrotate rather than have the application handle it.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAB0F1_xi1f38YaDhBcKTZ6PSqC70zsiWPeYpO5tFnOs2%3DRdXNw%40mail.gmail.com.

[miwee]

I run Elixir on my windows machine for development. logrotate, syslog and friends are not available. However, I don't intend to run production code on windows so that is not the issue. All I want is file based logging, so that I can debug some long running applications. There is a backend available, which meets my need fine.
https://github.com/onkel-dirtus/logger_file_backend

I'm mentioning this, just in case if it's considered useful enough to make it part of the default logger library.

thanks
miwee

[David Rouchy]

+1 for Custom log levels per module/logger

that'd be very handy when I need to debug only one part of the system without being flooded by the debug statements

David

[José Valim]

Just to be clear. What does everyone mean by custom log levels per module/logger? Would you like to say: print all log messages that is happening in this module/application as debug?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

--

You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/93b0f89e-7f32-470e-88ce-5e7380abb576%40googlegroups.com.

[Robert Stockdale]

The idea is that you can set the log level for specific modules. For example, your main log level could be set to info, but you could also specify for module Foo.Bar, you want to log at the debug level.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAGnRm4LU%3D4tKXuvmsGu3TnvtGHMXQEtss7ZpqBn_Pm-Bdw%3DOag%40mail.gmail.com.

[Ed W]

On 10/05/2015 15:19, Booker Bense wrote:
> Maybe I'm hopelessly old school, but I'd like logs to go to syslog.
>
> I would like to see at least a couple choices of backend in the standard
> library. File, syslog, stdin to an external process. Getting logging data
> into your monitoring system is a key feature for modern applications.
>

Agreed, but all the cool kids are moving to Kibana for logging (or other
elasticsearch store), and influxdb for event data (which arguably fits a
little into the logger category..?)
https://www.elastic.co/products/kibana

Additionally I see a lot of sense in the trend to log json messages
rather than plain text, eg:
https://journal.paul.querna.org/articles/2011/12/26/log-for-machines-in-json/

I think it's possible to use syslog for both of the above, but I believe
(as in: I'm not using it) that there are other ways to poke the data in
which can be more resistant to losing logs and with potentially lower
overhead. (nginx has resisted supporting syslog logs for some years due
to arguing it has too high an overhead...)

I'm not yet using an elasticsearch store for my logs, but it's one of my
highest priority todo items and also I very much desire structured log
formats to make searching more powerful

Cheers

Ed W

[Booker Bense]

We should not reinvent tools and processes that already exist, but instead

play nice with existing tools. If you're logging to a file or syslog, it's a known

problem to put that log in elasticsearch. (i.e. logstash). 

Now there may be a reason to implement a custom backend for this purpose,

but that should not be the default. And the default shouldn't dictate a particular

technology. The common choice should speak the lingua franca of running a 

service and that is logging to either a file or syslog. 

Getting logging to a file "right" is tricker than it looks, it would be much better

to have a shared solid implementation, rather than every one repeating the 

same mistakes over and over again. There is a lot of engineering in logrotate

and syslog that we should take advantage of. 

I also feel pretty strongly that the application should not depend on logs,  good

design needs to clearly separate logs and application events. Any interface tweaks

that blur that distinction aren't helpful in the long run.

- Booker C. Bense 

[Ed W]

Hi

On 11/05/2015 18:58, Booker Bense wrote:
> We should not reinvent tools and processes that already exist, but
> instead
> play nice with existing tools. If you're logging to a file or syslog,
> it's a known
> problem to put that log in elasticsearch. (i.e. logstash).

I'm 100% behind you! Actually, it obviously came over differently, but
I meant to imply exactly that, ie integrate with some existing
bandwagon, not create my own.

I'm completely a syslog shop at present. Which is to say, I have
several applications that log directly to files and I find this an
annoyance that I don't deal with cleanly. To be specific this is
several web applications written in rails with an nginx frontend. Nginx
recently acquired the ability to log to syslog and it should be possible
to convert the rails apps to syslog (hours in the day...), so those can
go away, but at present my standard is syslog everywhere

However, if we are looking ahead, I would like to put on the table that
syslog can lose messages, it only has a certain level of reliability.
There are several workarounds to that from tcp transports to various
proprietary ack protocols. However, several of the logstash alike log
shippers can do various levels of reliable log shipping and I think that
is a valid reason to break with going via the syslog protocol (in my
opinion anyway). I also like to keep my virtual machines as tiny as
possible, so I prefer not to install extra tools and my dream would be
direct integration from Elixir to whatever log shipper can reliably and
speedily poke logs into an elastic backend...

> I also feel pretty strongly that the application should not depend on
> logs, good
> design needs to clearly separate logs and application events.

Forgive me from jumping on this, but in some jurisdictions we are
legally obliged to keep log files. I guess the lawyers can debate what
that means in terms of reliability and tolerance for dropping the odd
message, also I realise you were likely thinking of different use cases,
but my point would be to keep in mind that logging is often required for
business reasons, not just for the engineering department.

This is actually one of my reasons for investigating moving from syslog
and files (which works plenty fine for my needs...) to an elastic based
setup where we can put fancy front ends on and give business users the
ability to dig and also most easily meet our logging liabilities

Anyway, agree with your points.

Thanks for all the development on Elixir!

Ed W

[Dmitry Aleksandrov]

Would be nice to have application somewhere as __CALLER__.application to be accessible, in the same way, as __MODULE__, that is really, what is lacking in erlang. 

On compiling of a module, it is possible to detect, which module it is, but it is not possible automatic detect application, which will be a very nice feature (for tracing on application/module).

Now, Logger doing: [module: module, function: form_fa(fun), line: line], if we can add :application to it, than it will be possible to have per module/application tracing facilities. And it is really, what all want, I think. With it it will be possible to do Logger.trace(application: my_app, :debug).

I'll personally think, that logger should support backends: for a best case lager-compatible backends(? - do not exactly, if it makes sense).

And, it should be performant and stable at least as lager is become over the time: ( I think, it is known presentation: https://www.youtube.com/watch?t=2614&v=8BNpOHFvg_Q ).

[Dmitry Aleksandrov]

I was needed to find, in what lager_parsetransform generates. And, lager have application attribute as metadata.

info() ->

   case {whereis(lager_event),

         lager_config:get(loglevel, {0, []})}

       of

     {undefined, _} ->

         fun () -> {error, lager_not_running} end();

     {Pidtest10, {Leveltest10, __Tracestest10}}

         when __Leveltest10 band 64 /= 0 orelse

                __Tracestest10 /= [] ->

         lager:do_log(info,

                      [{application, test}, {module, test},

                       {function, info}, {line, 10},

                       {pid, pid_to_list(self())}, {node, node()}

                       | lager:md()],

                      "info:~s", ["test"], 4096, 64, __Leveltest10,

                      Tracestest10, Pidtest10);

     _ -> ok

   end.

Logger doesn't it at the moment.

[José Valim]

Do you have any idea how they find out which application is it? Do they hook into rebar? I would love to support this feature but I don't know how to do it without some sort of Mix + Logger integration (which may be fine).

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/48d21676-5bc3-4216-abb9-964e03546c68%40googlegroups.com.

[Dmitry Aleksandrov]

They do it directly in lager:

https://github.com/basho/lager/blob/master/src/lager_transform.erl#L265

I think, that __CALLER__ meta data information should have something like application field, which can be modified by mix for such use cases. It can be practical in other situation, not only in Logger.

https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/compilers/elixir.ex#L112-L115

Like adding `application:`, which adds this information to enviroments, like module, function? Do you see problems in this approach?

[José Valim]

The way they do it won't work with Mix/Logger because they are assuming there is an .app or .app.src in the output directory at compilation time which is not true for Mix projects.

We would need some sort of configuration, Application.get_env(:logger, :current_application), and Mix could set it before compiling stuff.

Please open up an issue about this particular bit.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

[Roman Chvanikoff]

Could be nice to have ability to use custom, dynamic log-levels. Currently we only have "error", "warn", "info" and "debug" levels and what I want is ability to do something like Logger.log(:my_level, "My log message") to get "21:34:31.786 [my_level]  My log message". All custom "log-levels" might be treated as "debug" level.

воскресенье, 10 мая 2015 г., 11:59:14 UTC+3 пользователь José Valim написал:

[Pedro Medeiros]

It's a good idea. Really liked it. maybe we can do it with macros and instead of calling Logger.log(:my_level "Mu Log message") we can just call Logger.my_level "log message"

--

You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/2817a16a-ecf5-4bcb-878f-2076daf1bd93%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Pedro Henrique de Souza Medeiros
----------------------------------
Cel: +55 (61) 9197-0993
Email: pedr...@gmail.com

Beautiful is better than ugly,
Explicit is better than implicit,
Simple is better than complex,
Complex is better than complicated.

The Zen of Python, by Tim Peters

[José Valim]

We can't dynamically define functions on the Logger module. I am still reluctant to add new levels, wouldn't metadata help solve this?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

[Roman Chvanikoff]

José, first of all I'm thinking about this in context of file backend: with custom levels it will become possible to have different files for different log levels and instead of greping required metadata from a huge "debug.log" it will be possible to just inspect logs you need.

пятница, 3 июля 2015 г., 8:00:24 UTC+3 пользователь José Valim написал:

[José Valim]

Once we support tracing, this should be achievable with metadata too. :)



--

[Roman Chvanikoff]

Ok, I still think syntax format of Logger.log(:mylevel, msg) is better comparing to Logger.debug(msg, level: :mylevel) but in case it's not necessary to achieve routing to different files, I'm ok with it.

пятница, 3 июля 2015 г., 8:19:59 UTC+3 пользователь José Valim написал:

[Devin Torres]

The feature I have always wanted but am too stupid to implement is the ability to adjust log level in config.exs *per application*. As it is today, I cannot enable debug level without all the applications my library uses spamming me.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/27501c0d-4654-4e96-a1f0-0e50a1514907%40googlegroups.com.

[José Valim]

We have an open issue to move the stacktrace information before the state due to such issues. There is also a truncate configuration which you can change.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

On Wed, Jul 8, 2015 at 12:39 PM, Bruno Matos Tavares <bruno.mat...@gmail.com> wrote:

Hi,

When the process state or last message is "big" the stacktrace might not be visible because is truncated. Is it possible to have more granularity on what we truncate? 

Bruno.

On Sunday, May 10, 2015 at 9:59:14 AM UTC+1, José Valim wrote:

Folks,

When we started the Logger library, we have brought in many features but some of them were postponed to after 1.0. You can see the full list here:

https://github.com/josevalim/logger

The remaining features we have pending today:

1. Tracing support: it mostly includes support to perform certain actions based on metadata

2. File logging support and log rotation: the ability to log to files and rotate logs

I believe tracing support would be useful and feedback on what you would like to see when it comes to tracing is very welcome. However, I am not convinced on the usefulness of file logging support nor log rotations as tools like logrotate do an excellent job and it is easy to pipe the existing log to a file. Thoughts?

Also, is there any other feature you would like to see in Logger?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

[Bruno Matos Tavares]

The truncate configuration is for the all message. 

Having a bit more granularity would be great as we can express:

config :logger, :console,

truncate: 8192,

truncate_last_message: 1000, # defaults to `truncate`

truncate_state: 2000 # defaults to `truncate`

Then stackstrace would be whatever space remains to the truncate. 

I can’t find the open issue that you refer to. 

Bruno.

[James Fish]

The order has been changed in master already: https://github.com/elixir-lang/elixir/commit/6cd1478a2ec1a3dad7f02fb5b9291abe3a3a427f.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/BAFFDDFF-6333-4225-B9F8-8BB8381B0C10%40gmail.com.

[Alex Shneyderman]

However, I am not convinced on the usefulness of file logging support nor log rotations as tools like logrotate do an excellent job and it is easy to pipe the existing log to a file.

This train of thought clashes with releases. VM is started with -daemon flag which starts things with no console at all. 

I guess in deciding of what backend should be included in Logger there is a need for a consistent criteria. To me console and file backends seem to satisfy criteria of being built with the tools that are already shipping with elixir. They do not require any thrid-parties. For example, same can not be said of syslog or loggly or other stuff that requires additional services. 

I believe file backend is a must. ATM we are handicapped in a particular deployment scenario (releases as I mentioned earlier). So it seems like file and console  backends are  complementary in that sense.

Sure, we can develop our own backend but in that case what's special about console? I see no reason for it being part of the core distribution ATM.

I think making file backend as simple as  possible is also acceptable as tools like logrotate can do their magic to help with the rest of the needs.

Cheers,

Alex.

[Bruno Matos Tavares]

Alex you have a point there, however, how likely one will use a simple file backend in production?

Just by speaking on having a file backend we had/have a discussion if the backend should rotate the logs. IMO, this kind of indicates that having a file backend will open a pandora box. We could discuss if the file backend or even Logger should allow to swap the Logger.Formatter for a custom formatter because one would like JSON logs entries or log to different files based on the log level.

Speaking by my own experience (I've develop ExSyslog https://hex.pm/packages/exsyslog) I had to allow the features above. So, I'd always have to develop or use a custom backend.

Cheers,

Bruno.