Logging config file examples?

[Samuel Lampa]

Hi,

Is there any examples of a logging file, to pass to a luigi workflow using the --logging-file parameter?

E.g, how do I change the format and handler of the luigi-interface logger, and make that take effect for all the tasks in my workflow?

I have tried submitting a logging config file with the following content, but it does not seem to take effect (except that an empty log file, mmrun.log is created):

[formatters]

keys: default

[handlers]

keys: console, logfile

[loggers]

keys: root, luigi-interface

[formatter_default]

format: %(asctime)s %(name)-15s %(levelname)-8s %(message)s

[handler_console]

class: StreamHandler

args: []

formatter: default

[handler_logfile]

class: FileHandler 

args: ['mmrun.log']

formatter: default

[logger_root]

level: INFO

handlers: console

qualname: root

[logger_luigi-interface]

level: INFO

handlers: logfile

qualname: luigi

 

Any hints of what I'm doing wrong, or any pre-existing examples that work?

Best Regards

// Samuel

[Patrick Russell]

Try adding

propagate=1

To your luigi-interface logger:

[logger_luigi-interface]
level: INFO
handlers: logfile
qualname: luigi

propagate=1

See https://docs.python.org/2/library/logging.config.html#configuration-file-format

"The qualname entry is the hierarchical channel name of the logger,
that is to say the name used by the application to get the logger"


Everything else looks ok I think.

> --
> You received this message because you are subscribed to the Google Groups
> "Luigi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to luigi-user+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Samuel Lampa]

On tis 3 jun 2014 00:32:39, Patrick Russell wrote:
> Try adding
>
> propagate=1
>
> To your luigi-interface logger:
>
> [logger_luigi-interface]
> level: INFO
> handlers: logfile
> qualname: luigi
> propagate=1
>
> See https://docs.python.org/2/library/logging.config.html#configuration-file-format
>
> "The qualname entry is the hierarchical channel name of the logger,
> that is to say the name used by the application to get the logger"

Thanks! The very above phrase is what solved it for me. I changed:
qualname: luigi
to:
qualname: luigi-interface

... and then it worked! :)

Cheers
// Samuel

--
Developer at www.uppmax.uu.se/uppnex / www.farmbio.uu.se / rilpartner.se
G: http://google.com/+samuellampa
B: http://saml.rilspace.org
T: http://twitter.com/smllmp

[Samuel Lampa]

Btw, I then have a follow up question, in case somebody has the answer off the cuff:

In case I want to separate my own logging inside my tasks, by declaring another logger in the config file, where and how do I declare that one?

My initial guess would be to do, in a super-class that my task classes derive from, in the constructor, like this:

class MyMetaTask(luigi.Task):

    def __init__():

        self.log = logger.getLogger('my-workflow-logger')

... but if I recall correctly, it is not adviceable to override the constructor of luigi.Task? If so, the question remains, where to initialize my own logger for use inside my own application code in my tasks?

BR

// Samuel 

>> email to luigi-user+unsubscribe@googlegroups.com.

[jpamp...@gmail.com]

> >> email to luigi-user+...@googlegroups.com.

>
> >> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
>
>
>
> --
>
> Developer at www.uppmax.uu.se/uppnex / www.farmbio.uu.se / rilpartner.se
>
> G: http://google.com/+samuellampa
>
> B: http://saml.rilspace.org
>
> T: http://twitter.com/smllmp

Hi, I am interested on this. Have you found any good solution?

[Elias Freider]

Unless the logger needs to be customized to a specific instance of your task, I'd say create the logger in the module scope. That's a common pattern I've seen in other python projects at least (e.g. log = logging.getLogger(__name__) at the top of every module). Overriding __init__ of luigi tasks is doable if needed, but you need to remember to call super with (*args, **kwargs) so that any arguments passed are assigned to the correct luigi parameters.

/Elias

[Samuel Lampa]

 Many thanks for the info, Elias!

// Samuel