how to design log package to avoid allocations

[Vasiliy Tolstov]

Hi! I have some logging package and after memory profiling saw that for disabled log levels i'm allocate memory for message.

For example i'm send in logger only Info level. So i want to avoid Debug/Trace stuff.

but memory profiling says that for call log.Tracef("my message %v", msg) i'm allocate memory.

I don't want to guard all calls to check enabled log level to avoid allocation. How can i do zero allocation log in such case?

Thanks!

--

Vasiliy Tolstov,
e-mail: v.to...@selfip.ru

[Axel Wagner]

IMO, there really isn't a super good answer. The simple answer is: You need to delay the actual `fmt.Sprintf` call as long as possible. Which generally means, that the interface you consume (to allow the user to direct and configure logging) will need to reflect formatting and all kinds of things that the user actually doesn't want to deal with. So it's hard to find a good tradeoff between API simplicity and usability and not allocating in this case.

Another, maybe unforseen problem, is that you don't want logging to interfere with production. It's a painful lesson to learn, that synchronous logging can take down a production service with ease. If you want to be prepared for that and reduce the runtime overhead of logging, you will have to make it asynchronous. But that's fundamentally at odds with the above goal of delaying formatting - the user will usually not expect arguments to loggers to be used after the logging call returns. Buffering a formatted string avoids that of course.

Lastly, if you want runtime-configurability of logging and conditionally discard messages, you will likely reach for interfaces and thus limit the usefulness of escape analysis and inlining. So, by trying to allocate less, you might end up allocating more, because the compiler has to put things on the heap to be safe.

FWIW, I've blogged about this a couple of years ago. The opinions expressed there are probably controversial (and I haven't looked at it in a while - I might not even agree with them myself anymore), but it might help open up even more questions or give you some ideas. Also, there are of course many, many logging APIs already existing and I'm sure the people who wrote and use them will have strong opinions about those as well :) Personally, I've kind of given up on the idea of a panacea log API.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CACaajQswqK3DGHESCFd4pCdxndFpOFyKXhO25UDKgEsxYeK3SA%40mail.gmail.com.

[andrey mirtchovski]

to avoid allocations you have to hint at the type of what you're going
to print. for example see/use zerolog: https://github.com/rs/zerolog

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEkBMfF6kRp0V4LUGxfu3qDu4K7mpz%3DdhA76EU_qd1DfveZk-Q%40mail.gmail.com.

[Vasiliy Tolstov]

пн, 9 мар. 2020 г. в 19:41, andrey mirtchovski <mirtc...@gmail.com>:

to avoid allocations you have to hint at the type of what you're going
to print. for example see/use zerolog: https://github.com/rs/zerolog

Tanks, I saw it. But mostly i want to avoid typing hint

[Vasiliy Tolstov]

пн, 9 мар. 2020 г. в 19:36, Axel Wagner <axel.wa...@googlemail.com>:

IMO, there really isn't a super good answer. The simple answer is: You need to delay the actual `fmt.Sprintf` call as long as possible. Which generally means, that the interface you consume (to allow the user to direct and configure logging) will need to reflect formatting and all kinds of things that the user actually doesn't want to deal with. So it's hard to find a good tradeoff between API simplicity and usability and not allocating in this case.

Another, maybe unforseen problem, is that you don't want logging to interfere with production. It's a painful lesson to learn, that synchronous logging can take down a production service with ease. If you want to be prepared for that and reduce the runtime overhead of logging, you will have to make it asynchronous. But that's fundamentally at odds with the above goal of delaying formatting - the user will usually not expect arguments to loggers to be used after the logging call returns. Buffering a formatted string avoids that of course.

Thanks! So i think that most simple case is check each time log level , and output log only if level satisfied.

[Marcin Romaszewicz]

I'm using Uber's zap logger in production systems (https://github.com/uber-go/zap). It is designed to emit structured JSON logs and do the minimal amount of allocations possible. It's a completely different interface from Go's log package, but that is where the efficiency comes from. I'd highly recommend it, and the JSON logs are easy to ingest into services like Datadog or Sumologic.

-- Marcin

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CACaajQshiJaz%2BAJ_Tm5VbTW8FDfP0QyLvzj_KCGDTAM7RpiPrg%40mail.gmail.com.