Logger backend to log to disk?

[J David Eisenberg]

Does one exist? I'm not sure I have the Elixir-Fu to be able to write one.

[José Valim]

I don't think one exists right now. I am hoping someone will implement a Logger file backend that supports log rotation. If someone needs inspiration, Lager is an excellent starting point: https://github.com/basho/lager

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

On Tue, Aug 5, 2014 at 5:20 AM, J David Eisenberg <jdavid.e...@gmail.com> wrote:

Does one exist? I'm not sure I have the Elixir-Fu to be able to write one.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Wojciech Kaczmarek]

A note before someone rushes in a hurry to implement this:

While built-in log rotation seems a cool default for experiments and a good selling point, when things come to production, it's crucial to play well with what's already used "in the industry" for rotation, which is atm logrotate and Solaris/SmartOS logadm.

[José Valim]

Honestly, since there are known log rotation tools, I would rather guarantee we play well with them than reinvent the wheel. Btw, can I pipe into those log rotate tools? For example, could I do:

mix run > logrotate dir

?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

[Wojciech Kaczmarek]

Not sure what exactly you'd want to fed as input to logrotate input? :)

Well, logrotate relies on a config file describing a rotation, even if minimal.

You can try with making a conffile like:

"*.log" {

rotate 5

}

and then:  touch foo.log; logrotate -d conffile

will print you debug info on what logrotate would do and why.

Now I have a story which I believe sheds some light on of the meaning of "play well with logrotate". Quite long but the context is IMO important.

Usual problem that logrotate - or any other external rotator - has with some loggers is that they continue to write to a rotated file. That's why "copytruncate" option was invented, now widely used, despite its own problems with possibility of losing some data and obvious need to use additional disk space, not to mention i/o.

Now what "some loggers" means also Java loggers and this had some consequences in the wild.

Tomcat, Weblogic, JBoss & family recommended to use this "copytruncate" and admins stick to this - resulting in recurring problems posted regurarly to serverfault.com and similar sites - you can imagine that on productions logs being rotated are quite big.

In the meantime Java community of developers decided that "external log rotation is wrong" and heavily advertised rotation built into Log4J - which in the real world also isn't so sweet, as it requires admins to reconfig Log4J in order to change the rotation, which is (a) not their way of doing things (b) sometimes not what they are able to do because of some release management process (there are many organizations where OS admins and appserver admins are different teams with different budgets and permissions). 

But that point of view has spread widely beyond Java world.

The outcome we have now: coders usually think that internal log rotation is the best way, while sysadmins think that the external log rotation is the best way.

I think both have valid arguments, but also at many companies logrotate & friends seem to be not going away.

Now what Elixir Logger could do to clear this out is to detect the situation when file underneath was moved.

[Kurt Williams]

I am presently starting a solution that does just this. I'll post a link to the repo and welcome any feedback.

[José Valim]

Thanks for the detailed and excellent explanation Wojciech!

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

[Rob Lally]

I think that a large part of the JVM's insistence on internal log rotation was to ensure that it worked across different platforms because of the strongly divergent behaviour of file systems across windows and *nix. Logrotate doesn't work on Windows (AFAIK) so pushing people in that direction pushes us away from having an inclusive platform. 

R.

[Wojciech Kaczmarek]

Good point, but still there's a chance to save both worlds by taking care of the "moving file" issue and providing internal rotation for those who need it.

[Kurt Williams]

Here's my initial code. Feedback welcome. 

https://github.com/onkel-dirtus/logger_file_backend

[Wojciech Kaczmarek]

Haven't checked it in the wild yet, but the idea looks cool and simple.

My *wild guess* is that even with a flood of messages to log, the cost of calling stat(2) in inode/1 for each message is minimal compared to the fact, that we have to actually write stuff to the log file anyway. But that'd be interesting to measure.