Recommended solution for extensive logging

19 views
Skip to first unread message

Ram Rachum

unread,
Oct 1, 2021, 9:09:27 AM10/1/21
to pyweb-il
Hi everyone!

I'm working on Marley, which is a framework for multi-agent reinforcement learning that I'll use for my research.

I recently added logging, and I'm logging interesting events from various different modules. Right now the logs are simply written to a text file that looks like this. I've got multiple threads and multiple processes writing to the same log file. 

At some point I'm going to have to use a better log architecture. The multiline entries are ugly, it's difficult to filter log entries, etc. I'm guessing that a better architecture would involve the log entries being JSON objects which will be read by a program that's more intelligent than a text editor, and that allows the user to filter and do other fancy things. Of course, I dread having to look at JSON logs in a text editor.

Can you guys recommend a good solution for log management?

I'm hoping for something that isn't a SaaS. I want to use it locally.


Thanks,
Ram.

tal...@gmail.com

unread,
Oct 1, 2021, 9:45:27 AM10/1/21
to pywe...@googlegroups.com
I found filebeat very useful since we already had elastic in use in the project 


On 1 Oct 2021, at 16:09, Ram Rachum <r...@rachum.com> wrote:


--
You received this message because you are subscribed to the Google Groups "PyWeb-IL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyweb-il+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyweb-il/CANXboVZNwyBnkujokj-Rh8CdkDgt9CkffU4soNP%3DNM7H136Fhw%40mail.gmail.com.

Meir Kriheli

unread,
Oct 1, 2021, 11:20:43 AM10/1/21
to pyweb-il
Hi,

Look into bunyan logging, which is taking hold in various languages, a quick look in pypi finds:

The output format is JSONL (json objects separated by newlines instead of commas, which is understandable for the task). Usually tools handling json can handle jsonl as well, e.g.: filtering with jq. For more info about it:

Cheers

--
You received this message because you are subscribed to the Google Groups "PyWeb-IL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyweb-il+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyweb-il/CANXboVZNwyBnkujokj-Rh8CdkDgt9CkffU4soNP%3DNM7H136Fhw%40mail.gmail.com.


--

Ram Rachum

unread,
Oct 2, 2021, 11:40:49 AM10/2/21
to pyweb-il, Meir Kriheli
Thanks Meir. That's pretty much what I imagined for the way log entries are created, but what about how they're read? Can you recommend any tool for reading these JSONL log files?

Ori Hoch

unread,
Oct 3, 2021, 7:58:29 AM10/3/21
to PyWeb-IL
For a self-hosted solution I'm very happy with Loki for log collection and Grafana for UI to browse/filter the logs

https://grafana.com/docs/loki/latest/

But I generally recommend a hosted solution for logs because these services can be hard to setup and maintain long-term

Ram Rachum

unread,
Oct 9, 2021, 6:25:20 AM10/9/21
to Meir Kriheli, pyweb-il
Thank you everybody!

On Mon, Oct 4, 2021 at 11:07 AM Meir Kriheli <mkri...@gmail.com> wrote:
I've mentioned it in my prev comment, for everything related to slicing and dicing json(l), I turn to jq.

Cheers
Reply all
Reply to author
Forward
0 new messages