Unification of logging in Python's Standard Library
Matthew Barnes
wondering whether there are plans to update the rest of the Standard
Library to use this module wherever there is logging to be done. I
can think of a few Standard Library modules off the top of my head
that currently "roll their own" logging system:
asyncore.py
BaseHTTPServer.py
cgi.py
doctest.py
imaplib.py
unittest.py
A single, unified logging system across the entire Python Standard
Library would be a worthy goal for some future release (Python 3000,
or maybe even 2.4/2.5?).
Alex Martelli
This seems a good idea, and quite feasible for 2.4 (we do have over
a year before 2.4 is going to be feature-frozen, after all). I would
suggest you post this proposal directly to Python-dev, though --
here on c.l.py it's too likely to get accidentally overlooked!
Alex
Skip Montanaro
Matthew> With the inclusion of the new 'logging' module in Python 2.3,
Matthew> I'm wondering whether there are plans to update the rest of the
Matthew> Standard Library to use this module wherever there is logging
Matthew> to be done.
...
Matthew> A single, unified logging system across the entire Python
Matthew> Standard Library would be a worthy goal for some future release
Matthew> (Python 3000, or maybe even 2.4/2.5?).
Rather than bring it up on python-dev as Alex suggested (I think it might
get lost there as well), I think you should open a bug report on SF which
identifies the problem and which modules you think need to be updated. This
provides a record of how people think this task should be approached. As
time goes on, patches for specific modules can be attached to the report and
incorporated into the modules in question.
Another alternative is to write a PEP, though I don't think this is
necessarily required for this particular task. It also presents a higher
barrier to action than a bug report.
Skip
A.M. Kuchling
Skip Montanaro <sk...@pobox.com> wrote:
> Matthew> A single, unified logging system across the entire Python
> Matthew> Standard Library would be a worthy goal for some future release
> Matthew> (Python 3000, or maybe even 2.4/2.5?).
> necessarily required for this particular task. It also presents a higher
> barrier to action than a bug report.
I suspect a PEP is required, because there are more issues here than just
changing sys.stderr.write() to logging.info(). For example, what logger
should modules use to log messages? If they use the root logger, it will be
difficult to discard messages from asyncore while keeping messages from the
ZODB. If they use a particular logger, how is this name chosen? Is it
dependent on the module name, so asyncore logs to 'asyncore' and so forth?
What if my application has a logger named 'network', and I want it to go to
'network.asyncore', 'database.ZODB', etc.?
If the user specifies a logger, how is this done? Does every public method
have to grow an optional 'log' argument, or is there a module-level global
setting, so you'd call
asyncore.set_logger(logging.getLogger('network.asyncore'))?
None of these options are very difficult to implement, but choosing which
one will take some care; otherwise we'll be saddled with an inconvenient
implementation that we'll be forever working around.
--amk
John Roth
"Matthew Barnes" <mat...@barnes.net> wrote in message
news:3a8e83d2.03081...@posting.google.com...
I'm going to agree with Andrew: write a PEP.
Unittest, as far as I know, doesn't have a logging system, and
I, for one, don't want to see it burdened with a completely unnecessary
logging system. I can't comment on the other examples in your
list because I've never used them, but unittest is part of my standard
project life, and I like it the way it is, thanks. Kent Beck and the
other people involved in the xUnit development put a lot of thought
into how it works, and AFKC, lack of logging isn't one of the problems.
John Roth
Paul Moore
One other point, which is worth mentioning (although don't let it put
you off reporting a bug or writing a PEP), is that something like this
is *far* more likely to happen if you contribute code rather than just
ideas.
If you haven't the expertise or time to contribute code, please still
raise the idea - but don't be too disappointed if it's hard to get
anyone to do anything about it.
If you *can* contribute code, then you'll be welcomed with open arms
of course. And your code will be ripped to shreds, but that's just to
show we all care :-)
Paul.
--
This signature intentionally left blank
Matthew Barnes
> I suspect a PEP is required, because there are more issues here than just
> changing sys.stderr.write() to logging.info(). For example, what logger
> should modules use to log messages? If they use the root logger, it will be
> difficult to discard messages from asyncore while keeping messages from the
> ZODB. If they use a particular logger, how is this name chosen? Is it
> dependent on the module name, so asyncore logs to 'asyncore' and so forth?
> What if my application has a logger named 'network', and I want it to go to
> 'network.asyncore', 'database.ZODB', etc.?
>
> If the user specifies a logger, how is this done? Does every public method
> have to grow an optional 'log' argument, or is there a module-level global
> setting, so you'd call
> asyncore.set_logger(logging.getLogger('network.asyncore'))?
>
> None of these options are very difficult to implement, but choosing which
> one will take some care; otherwise we'll be saddled with an inconvenient
> implementation that we'll be forever working around.
"John Roth" <newsg...@jhrothjr.com> wrote:
> I'm going to agree with Andrew: write a PEP.
>
> Unittest, as far as I know, doesn't have a logging system, and
> I, for one, don't want to see it burdened with a completely unnecessary
> logging system. I can't comment on the other examples in your
> list because I've never used them, but unittest is part of my standard
> project life, and I like it the way it is, thanks. Kent Beck and the
> other people involved in the xUnit development put a lot of thought
> into how it works, and AFKC, lack of logging isn't one of the problems.
Andrew raised some very good points, and a PEP probably is necessary.
I did have some half baked ideas on how to go about this.
Generally speaking, the default logging output of each module that
uses 'logging' should be identical or as identical as possible to what
the module currently produces. This would (hopefully) make the
transition nearly transparent to end-users. Then we provide some
'hook' at either a module-level or (preferably) class-level for
replacing the default logger with a customized one.
Here's a few more module-specific thoughts:
asyncore.py
-----------
Add a 'logger' attribute to the 'dispatcher' class, which defaults to
a logger named 'asyncore' (i.e. the module's __name__) configured to
use a StreamHandler. This would allow each instance of
asyncore.dispatcher to potentially have a unique logger.
BaseHTTPServer.py
-----------------
Add a 'logger' attribute to the 'BaseHTTPRequestHandler' class, which
defaults to a logger named 'BaseHTTPServer' (i.e. the module's
__name__) configured to use a StreamHandler and a Formatter that
produces the RFC931-style messages that are currently produced (note:
haven't looked into whether a logging.Formatter can do that). This
would allow each instance of BaseHTTPServer.BaseHTTPRequestHandler to
potentially have a unique logger.
cgi.py
------
Not quite sure about what to do with this one; Andrew's questions pop
up here. Perhaps provide a logger named 'cgi' (i.e. the module's
__name__) configured to use a StreamHandler. The functions should use
a module-level 'logger' variable (initialized to the 'cgi' logger)
rather than hard-coding the logger name into the function bodies. The
'logger' variable could then be assigned a different logger instance
and the functions would pick up on it.
I need to kill some more brain cells on the other modules in my list.
I will say that unittest is what motivated me to suggest this in the
first place, since I'm working on a testing framework that is
attempting to take advantage of both the unittest *and* logging
modules. Perhaps a new TestRunner class that uses logging
(LoggingTestRunner?) might coexist with the default TextTestRunner.
Matt
John Roth
> I will say that unittest is what motivated me to suggest this in the
> first place, since I'm working on a testing framework that is
> attempting to take advantage of both the unittest *and* logging
> modules. Perhaps a new TestRunner class that uses logging
> (LoggingTestRunner?) might coexist with the default TextTestRunner.
Perhaps you could tell us a bit more about this? My usage of
unittest is (relatively) pure XP/TDD - I expect all of the tests
to pass except the last one I worked on, and when they don't,
I swat it immediately. Consequently, I don't see any need for
logging anything.
John Roth
>
> Matt
Robin Becker
<newsg...@jhrothjr.com> writes
dispatch function. We test everything before going live, nevertheless we
still get errors in production and the logging is then invaluable. A
typical case is that of inconsistencies accidentally introduced into the
back end data base which the end users control; this is not exactly our
error, but it causes failures in our code.
On a different point we have different logging for testing in our office
and on the client's staging, testing and production machines. Typically
we use files for simple testing, but on the client production machines
we have a single database which collects all the errors from several web
facing servers. This requires that our code recognise its location (we
use presence of files called LIVE etc) and I wonder how other people
handle this terrain recognition problem?
--
Robin Becker
Duncan Booth
news:vk5kc9g...@news.supernews.com:
>> I will say that unittest is what motivated me to suggest this in the
>> first place, since I'm working on a testing framework that is
>> attempting to take advantage of both the unittest *and* logging
>> modules. Perhaps a new TestRunner class that uses logging
>> (LoggingTestRunner?) might coexist with the default TextTestRunner.
>
> Perhaps you could tell us a bit more about this? My usage of
> unittest is (relatively) pure XP/TDD - I expect all of the tests
> to pass except the last one I worked on, and when they don't,
> I swat it immediately. Consequently, I don't see any need for
> logging anything.
>
I would find it interesting, and possibly even useful, if unittest could be
made to record some historical information in a database. e.g. The date and
time when each unit test first failed, first passed, and most recently
failed/passed, and the number of times each test has passed/failed. A
record of the #lines of code each time the tests all pass would also be
useful, and you probably want to record the stats separately against each
developer.
On a project with several developers (or one undisciplined one like myself)
you could potentially use these figures to flag problems, such as someone
whose tests usually pass first time is probably doing test last rather than
test first coding, or if a lot of new code appears without new tests it was
either a major refactor or someone isn't writing the tests they need.
Someone who is doing it properly will show a pattern of at least three test
runs (one fail, two passes) for each new test. Finally, should you survive
lynching by the developers, the customer will be really pleased to see
complicated graphs showing how much testing you are doing.
Of course, this is completely unrelated to the sort of logging the logging
module could do for you.
--
Duncan Booth dun...@rcp.co.uk
int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3"
"\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure?