Logging into single file from multiple modules in python when TimedRotatingFileHandler is used

[Chethan Kumar S]

Hi all,

Need help with below query on python logging module.

I have a main process which makes use of different other modules. And these modules also use other modules. I need to log all the logs into single log file. Due to use of TimedRotatingFileHandler, my log behaves differently after midnight. I got to know why it is so but couldn't get how I can solve it.
Issue was because of serialization in logging when multiple processes are involved.

Below is log_config.py which is used by all other modules to get the logger and log.
import logging
import sys
from logging.handlers import TimedRotatingFileHandler

FORMATTER = logging.Formatter("%(asctime)s — %(name)s — %(message)s")
LOG_FILE = "my_app.log"

def get_file_handler():
file_handler = TimedRotatingFileHandler(LOG_FILE, when='midnight')
file_handler.setFormatter(FORMATTER)
return file_handler

def get_logger(logger_name):
logger = logging.getLogger(logger_name)
logger.setLevel(logging.DEBUG) # better to have too much log than not enough
logger.addHandler(get_file_handler())
#with this pattern, it's rarely necessary to propagate the error up to parent
logger.propagate = False
return logger

All other modules call, 'logging = log_config.get_logger(name)' and use it to log.
I came to know about QueueHandler and QueueListener but not sure how to use them in my code. How can I use these to serialize logs to single file.?

Any help is appreciated.
Thanks,
Chethan

[Lars Liedtke]

--
Lars Liedtke
Software Entwickler


Phone:
Fax: +49 721 98993-
E-mail: l...@solute.de


solute GmbH
Zeppelinstraße 15
76185 Karlsruhe
Germany


Marken der solute GmbH | brands of solute GmbH
billiger.de | Shopping.de


Geschäftsführer | Managing Director: Dr. Thilo Gans, Bernd Vermaaten
Webseite | www.solute.de
Sitz | Registered Office: Karlsruhe
Registergericht | Register Court: Amtsgericht Mannheim
Registernummer | Register No.: HRB 110579
USt-ID | VAT ID: DE234663798


Informationen zum Datenschutz | Information about privacy policy
http://solute.de/ger/datenschutz/grundsaetze-der-datenverarbeitung.php

Am 21.06.22 um 11:04 schrieb Chethan Kumar S:

> I have a main process which makes use of different other modules. And these modules also use other modules. I need to log all the logs into single log file. Due to use of TimedRotatingFileHandler, my log behaves differently after midnight. I got to know why it is so but couldn't get how I can solve it.
> Issue was because of serialization in logging when multiple processes are involved.

Could be unrelated and only a part of a solution, but if you are on a
unixoid system, you could use logrotate, instead of
TimedRotatingFileHandler. logfrotate ensures that the logging service
does not realize, its logs have been rotated. So it can log as if
nothing has happened.

I don't know, if that helps with your multiple processes, or is a
solution at all.

Cheers

Lars

[Barry Scott]

> On 22 Jun 2022, at 11:06, Lars Liedtke <l...@solute.de> wrote:
>
> Could be unrelated and only a part of a solution, but if you are on a unixoid system, you could use logrotate, instead of TimedRotatingFileHandler. logfrotate ensures that the logging service does not realize, its logs have been rotated. So it can log as if nothing has happened.

The process that is writing the file must be told that rotation has happened for it to work.
Other wise all the logs keep being write to the original file via the FD that the process has.

logrotate's config include how to tell the process the log file needs reopening.

Barry

[Lars Liedtke]

> The process that is writing the file must be told that rotation has
> happened for it to work.
> Other wise all the logs keep being write to the original file via the
> FD that the process has.
>
> logrotate's config include how to tell the process the log file needs
> reopening.
>

Thanks for clearing.

[Dieter Maurer]

Chethan Kumar S wrote at 2022-6-21 02:04 -0700:
> ...

>I have a main process which makes use of different other modules. And these modules also use other modules. I need to log all the logs into single log file. Due to use of TimedRotatingFileHandler, my log behaves differently after midnight. I got to know why it is so but couldn't get how I can solve it.
>Issue was because of serialization in logging when multiple processes are involved.
>
>Below is log_config.py which is used by all other modules to get the logger and log.
>import logging
>import sys
>from logging.handlers import TimedRotatingFileHandler
>
>FORMATTER = logging.Formatter("%(asctime)s — %(name)s — %(message)s")

The usual logging usage pattern is:
the individual components decide what to log
but how the logging happens it decided centrally - common
for all components.
This implies that usually the individual components do not handle
handlers or formatters but use the configuration set up centrally.

[Peter J. Holzer]

On 2022-06-21 02:04:52 -0700, Chethan Kumar S wrote:
> I have a main process which makes use of different other modules. And
> these modules also use other modules. I need to log all the logs into
> single log file. Due to use of TimedRotatingFileHandler, my log
> behaves differently after midnight. I got to know why it is so but
> couldn't get how I can solve it.
> Issue was because of serialization in logging when multiple processes

^^^^^^^^^^^^^^^^^^
> are involved.

I think this is crucial point. Not "multiple modules" (as you wrote in
the subject), but "multiple processes". If each of multiple processes
wants to rotate the logfile itself, it is very likely that they will
work at cross-purposes.

Somebody already suggested using logrotate (or similar external tools).
You can then use WatchedFileHandler
(https://docs.python.org/3/library/logging.handlers.html#watchedfilehandler)
to automatically detect when a logile has been rotated.

Alternatively you can use a central logging service (like syslog) which
handles all that stuff.

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | h...@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"