Pulsar logging rotate

[iiva...@firstgaming.com]

Hi.
A have some strange problem.
My app consists of 4 separate workers (so there are 4 separete proceses). Every worker writes logs in one file (like 'mylogs.log').
Pulsar logging is initialized by code:

 pulsar_log.LOGGING_CONFIG['handlers']['file'] = {
        '()': file_handler,
        'formatter': 'very_verbose'
    }

File handler method is:

 def file_handler(**kw):
        return RotatingFileHandler(
            log_path,
            maxBytes='1m',
            backupCount=10,
            **kw
        )        

RotatingFileHandler is class from logging.handlers module.

The problem appears in moment of rotating when all of 4 workers write logs. In this moment file 'mylogs.log' is renamed to 'mylogs.log.1' and new file 'mylogs.log' should be created. That is so, but.
When the projects is under load also files 'mylogs.log.2', 'mylogs.log.3', 'mylogs.log.4' are created and logging continue to write to them. So I can see the situation when logging writes logs in files 'mylogs.log',
'mylogs.log.2', 'mylogs.log.3', 'mylogs.log.4' - 4 files - the same number as 4 workers.
I think this happens because of logging not competely supports multiprocess, but it's only supposing.

Is there somebody, who deal with such problem. How did you solve it?
 

[lsbardel]

Hi,

Yes you are right, the RotatingFileHandler does not work well with the multiprocessing module unfortunately.

To solve the problem you could use log rotation via the OS.

For example, in Ubuntu you could create a config file in /etc/logrotate.d/log-rotate for a weekly log rotation with the text

/var/log/project/*.log {
    weekly
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    copytruncate
}

and write your log files in the /var/log/project/ directory.

Hope that helps