A simpler logging configuration file format?
geoff...@gmail.com
I wonder if there are others out there who like me have tried to use
the logging module's configuration file and found it bloated and over-
complex for simple usage (i.e. a collection of loggers writing to
files)
At the moment, if I want to add a new logger "foo" writing to its own
file "foo" to my config file, I have to repeat the name 7(!) times,
editing 3 different sections in the process:
1) Add "foo" to [loggers] and [handlers]. (These seem completely
pointless and were apparently added in anticipation of new
functionality that never happened).
2) Create the section
[logger_foo]
handler:foo
qualname:foo
level:INFO
3) Create the section
[handler_foo]
class: FileHandler
args:("foo", "a")
Wondering how it got like this, I found this blog from soon after it
was released in 2004:
http://www.mechanicalcat.net/richard/log/Python/Simple_usage_of_Python_s_logging_module
Some of the verdicts are "full of dead chickens and error prone",
"horribly complicated" , "over-engineered". So along comes the
"basicConfig" method in 2005 which is supposed to address thes
concerns. But this can only be applied globally, not even to
individual loggers, so is only helpful for extremely basic usage
(everything to one target) as far as I can see. The config file is
still much as it was then.
I'd really like to see the concept extended to encompass multiple
loggers and the config file. Allowing Logger.basicConfig should be
trivial. Writing a configuration format which took a section and
passed all the options in it to basicConfig would in my view lead to a
much friendlier syntax for normal usage.
Lawrence D'Oliveiro
<30db8a0b-5b19-47e4...@h23g2000vbc.googlegroups.com>,
geoff...@gmail.com wrote:
> At the moment, if I want to add a new logger "foo" writing to its own
> file "foo" to my config file, I have to repeat the name 7(!) times,
> editing 3 different sections in the process:
I suppose there's always the Sendmail solution: stick an m4 wrapper on top
to hide the details of the process. :)
Vinay Sajip
> Hi all,
Some discussion about this has already taken place on the issue
tracker:
http://bugs.python.org/issue6136
and I thought I would summarise my comments there on this thread, for
people who have no need to look at that issue.
> At the moment, if I want to add a new logger "foo" writing to its own
> file "foo" to my config file, I have to repeat the name 7(!) times,
> editing 3 different sections in the process:
Strictly speaking, you don't need to repeat the name "foo" 7 times.
> 1) Add "foo" to [loggers] and [handlers]. (These seem completely
> pointless and were apparently added in anticipation of new
> functionality that never happened).
You don't have to call the logger "foo" and the handler "foo". The
names in the [loggers] and [handlers] section are just to allow the
system to cross-refer different entities (loggers, handlers,
formatters) declared in the config file. You could give the logger a
name of "L1" and the handler a name of "H1", for example.
> 2) Create the section
> [logger_foo]
> handler:foo
> qualname:foo
> level:INFO
Here, if you named your logger L1, the section would be [logger_L1],
not [logger_foo]. If you called your handler H1, then the handler line
would be handlers=H1. The qualname is "foo" because you choose to log
events to the logger named "foo". It could be more specific, e.g.
"myapp.mymodule.myfunctionalarea".
> 3) Create the section
> [handler_foo]
> class: FileHandler
> args:("foo", "a")
Here, if you named your handler H1, the section would be [handler_H1],
not [handler_foo]. You've also called the log file "foo", which more
conventionally you might name "foo.log".
In your simple example where everything is named foo, and you've
chosen to call cross-referencing handles within the file "foo", then
naturally you repeat "foo" a lot of times. In an equivalent file
[loggers]
keys=root,L1
[handlers]
keys=H1
[logger_L1]
level=INFO
handlers=H1
qualname=foo
[handler_H1]
class=FileHandler
args=('foo.log', 'w')
"foo" only occurs twice.
> Wondering how it got like this, I found this blog from soon after it
> was released in 2004:
>
> http://www.mechanicalcat.net/richard/log/Python/Simple_usage_of_Pytho...
>
> Some of the verdicts are "full of dead chickens and error prone",
> "horribly complicated" , "over-engineered". So along comes the
> "basicConfig" method in 2005 which is supposed to address thes
> concerns. But this can only be applied globally, not even to
> individual loggers, so is only helpful for extremely basic usage
> (everything to one target) as far as I can see. The config file is
> still much as it was then.
Some of this is just a consequence of using ConfigParser. The existing
file format is not perfect or even complete (for example - no support
for filters), and was proposed on python-dev for review before the
logging package went into Python. The general consensus was that it
was OK for basic use, and IIRC several people considered it as +0 or
-0 because they envisaged doing their own thing (e.g. having the
logging configuration as just part of an application's overall
configuration). This is the reason configuration was put in a separate
sub-module - if it's not needed, it needn't be loaded. AFAICT since
basicConfig was introduced and later refined, there has not been much
feedback about the configuration file format being a problem - and
while that could be because no-one is using it, that's not borne out
by the evidence. There have been numerous instances on c.l.py where
questions have been raised about how to specify custom handlers in the
configuration file, so at least some people are using it or have used
it (the configuration functionality, I mean).
> I'd really like to see the concept extended to encompass multiple
> loggers and the config file. Allowing Logger.basicConfig should be
> trivial. Writing a configuration format which took a section and
> passed all the options in it to basicConfig would in my view lead to a
> much friendlier syntax for normal usage.
Well, basicConfig is for basic configuration of the logging package as
a whole, not for configuring individual loggers or handlers. It's best
not to conflate loggers and handlers - basicConfig might appear to do
this because its arguments are partly specifying handler configuration
and partly logger configuration. This is because it's the simplest
case - one logger (the root logger) and one handler (according to the
relevant arguments passed in). The handler-related arguments are used
to create a default handler and/or formatter, the logging level is
specified by the level argument, and it's applied to the root logger.
It doesn't make sense to use the same approach if you are going to
configure multiple loggers and handlers; if the existing format is not
palatable, it's relatively easy to create your own format and write
code to parse it and load it, and then call the programmatic API to
configure logging appropriately. You mentioned on the bug tracker that
you are coming from a background of using log4py - in which I believe
loggers provide the functionality which is found in handlers in
Python's logging.
If you believe you have an idea for a much simpler configuration
format (perhaps for simpler use cases, or perhaps for general use) I
would encouraging writing an implementation and putting it up on PyPI
so that people can play around with it and give feedback. ISTM that
there is a general feeling that any sizeable new or changed
functionality intended for the stdlib should be proven in the
community through wide use before being seriously considered for
inclusion. For example, I wrote a hierarchical configuration module,
just one of several configuration-related modules on PyPI:
http://pypi.python.org/pypi?%3Aaction=search&term=config&submit=search
However, I am not proposing it for inclusion in the stdlib, as I don't
believe it has had enough feedback yet from the community to be able
to accurately gauge its quality or utility.