logging?

[Kevin Howerton]

Realized today that by default django more or less suppresses errors
sent to stderr when debug is off. Well I guess it uses email sent to
admins by default. Though, I distinctly remember stderr being sent
into my apache logs for some reason. Maybe this was when I was using
mod_python rather than mod_wsgi?

I think that comprehensive logging might be beneficial in the
long-run. Not sure if django currently has any logging that occurs at
various levels?

For starters though, we should patch the default behavior so that it
logs errors to the python logging system when it handles an exception.
I can do this, and even write some documentation on how to setup and
use a simple python logger.

Where would be the most appropriate place for this patch? Right now I
just created a custom server_error view that takes care of it, but
perhaps it would be more appropriate for it to be in
core/handlers/base.handle_uncaught_exception ?

http://code.djangoproject.com/browser/django/trunk/django/core/handlers/base.py#L148

or here?

http://code.djangoproject.com/browser/django/trunk/django/views/defaults.py#L16

Thanks guys,

-k

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

[Alex Gaynor]

I suggest you review the previous discussions about logging during the
Django 1.2 feature development time frame. There's a lot of prior
discussion here.

Alex

--
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me

[Vinay Sajip]

On May 5, 4:47 am, Kevin Howerton <kevin.hower...@gmail.com> wrote:

I think the plan was to look at integrating logging after the 1.2
release was done. The original discussion was here:

http://groups.google.com/group/django-developers/browse_frm/thread/8551ecdb7412ab22/

With Django's configuration style, it would be beneficial to use
dictionary-based configuration as introduced in PEP 391 and
implemented in Python trunk (2.7/3.2). For older Python versions,
Django could use my standalone implementation of PEP-391 functionality
available at

http://bitbucket.org/vinay.sajip/dictconfig/

I have a Django branch which shows how logging can be easily
integrated during startup, and it's at

https://code.launchpad.net/~vinay-sajip/django/logging

It's a proof-of-concept, last merged with trunk in Jan 2010, but the
changes are simple so it shouldn't be a problem to bring up to date.
I'll look at doing this once 1.2 is out.

Regards,


Vinay Sajip

[Kevin Howerton]

Both yours and simon wilsons solutions look like great solutions.
Though possibly over-arching for an initial implementation of logging,
but if an implementation like that makes it into 1.3 in one fell swoop
I would be as happy as a clam.

Making use of python's built-in logging facility should be a very high
priority for any python project that considers itself worthy of being
used in production environments.

You guys are definitely taking the right direction .... having
comprehensive logging coverage in my opinion is more important than
having comprehensive test coverage.

What I was suggesting in my post though, was that we alter the default
CRITICAL error handling behavior ... as it stands (by default) all of
my ERRORs are being suppressed in production. This is pretty
confusing for new users (there were 3 others in IRC with the same
complaint yesterday), and pretty un-pythonic IMO. Unless I missed
something?

At minimum the documentation should inform the user of this
short-coming, and provide a link to a wiki with the available
work-arounds.

Leaving out logging of critical errors because we don't currently have
the perfect all encompassing logging solution, is not the decision I
would have made. Doing things in an iterative cycle might work
better. Step 1: do what you need to have done. Step 2: do what you
want to have done.

Leaving out critical stuff because you don't know what you want, when
you know what you need that is critical is no bueno. It's like not
setting a broken arm because you can't decide on what color you want
the fiber glass to be.

-k

ps. I don't want to get into a flamewar (although it is certainly one
of my specialities... and prefacing this paragraph with this sentence
probably won't help the cause), though after reading that thread I
felt like this needed to be said.

No one here (me included) is allowed to call a *built-in* Python
library un-pythonic.... especially the logger.

So it shares some similarities with Java? Some stuff in Java is
awesome. Some stuff in Ruby is awesome. Some of the stuff in Rails
is awesome. Some stuff in C# is awesome. Some stuff in Perl is
awesome. I've luckily been able to avoid PHP for my entire career,
but I'm sure there is something really nice about PHP too. The very
reason python is such a great language is *because* it takes some
(possibly even all of it) of the awesome stuff from a variety of
technologies, not because it wildly diverges and does things uniquely.

I see a lot of this pretentiousness in the programming community in
general, and really it is pretty detrimental to who-ever ends up
spouting it off. It's very reductionist. Ignoring prior art just
because it was written in ruby, or targets rails, or runs on the JVM,
or uses CamelCase for function names is stupid.

[Kevin Howerton]

[Kevin Howerton]

[Jacob Kaplan-Moss]

On Wed, May 5, 2010 at 11:48 AM, Kevin Howerton
<kevin.h...@gmail.com> wrote:
> ps.  I don't want to get into a flamewar

Then next time leave out the rambling, ad hominems, and sweeping
generalizations. Focus on technical suggestions -- working code is
even better. If you have problems, suggest fixes concretely.

Jacob

[Kevin Howerton]

I did.

[Karen Tracey]

On Wed, May 5, 2010 at 12:46 PM, Kevin Howerton <kevin.h...@gmail.com> wrote:

What I was suggesting in my post though, was that we alter the default
CRITICAL error handling behavior ... as it stands (by default) all of
my ERRORs are being suppressed in production.  This is pretty
confusing for new users (there were 3 others in IRC with the same
complaint yesterday), and pretty un-pythonic IMO.  Unless I missed
something?

I think you are missing something. From your earlier note you seem to be saying that it is information your own code is writing to stderr that is getting lost somewhere. Django does nothing to suppress anything sent do stderr. I've got views that write to stderr running under mod_wsgi, and the stderr output appears in the configured Apache error log. You do need to either include a newline in the data or explicitly flush stderr for the data to be written, and that may be a difference from mod_python (I don't remember enough about mod_python to say one way or the other). I think the problem you are having is more related to mod_wsgi, and possible differences between it and mod_python, than Django.
 

At minimum the documentation should inform the user of this
short-coming, and provide a link to a wiki with the available
work-arounds.

It isn't clear to me what shortcoming you think should be documented. If it is the behavior of mod_wsgi with respect to stderr possibly the deployment docs for mod_wsgi could mention something, but one danger of Django putting information like this in the docs is that it gets out of date as mod_wsgi evolves. In general it seems better to provide a minimum amount of information in Django doc, stuff that is specific to Django, and point the user to the appropriate source for authoritative documentation on the 3rd-party package.

For the record, Django's default handling of critical errors when DEBUG is False is fully documented:

http://docs.djangoproject.com/en/dev/howto/error-reporting/

Karen

[Kevin Howerton]

Yeah... not sure, though thank you for actually reading and responding
to my post.

"I think the problem you are having is more related to mod_wsgi, and
possible differences between it and mod_python, than Django."

Absolutely. Though, I'm not sure if it's the official recommendation
of django-project.com now or not, but most people in the community are
using mod_wsgi in Daemon mode. It is definitely a result of this,
also I doubt mod_wsgi is going to be changing the nature of their
exception handling code faster than you guys can keep up. It seems
like a pretty sweet setup right now so hopefully they don't change it
;)

Where django pulls a 500 it should call these two lines, which will
dump the exception raised to stderr.

type, value, traceback = sys.exc_info()
sys.excepthook(type, value, traceback)

If you have a logger setup, which I assume most sane people in
production do (right?!?). You can pass the exception info to the
logger with and have the same result as one that passes straight to
stderr only you can have more explicit control over how it behaves,
especially if django uses named loggers:

logger.error("This can't be good", exc_info=True)

We don't really need both, but either one should exist in django if
it's going to support wsgi ;) Perhaps, I'm missing some config
setting available to mod_wsgi that will replicate mod_python's
behavior in this regard, but i "feel" django should work with a
default setup.

Also, I've tested this on both Fedora 8, and mod_wsgi 2.x ... and the
latest 2.6.5 python + wsgi 3.2.

"http://docs.djangoproject.com/en/dev/howto/error-reporting/"

I can't use email reporting because of performance, logistics, and
sheer volume in production.

-k

[Mike Axiak]

Why does this facility need to exist within Django? At least one
person [1] has written a middleware to do what you're saying. If it's
that important, people will start to use that app. There are certainly
many potential features people want out of logging, so it'd definitely
be better suited for an external app to bake.

I'd say that the main issue here isn't with lack of logging, but with
lack of a good one-stop Django app repository (DjangoZen [2] is a
start, but missing a lot of functionality). Unfortunately I don't have
time to maintain something like that.

Cheers,
Mike

1: http://github.com/djangrrl/django-logging-middleware
2: http://djangozen.com/

[Jacob Kaplan-Moss]

On Wed, May 5, 2010 at 1:53 PM, Kevin Howerton <kevin.h...@gmail.com> wrote:
> Where django pulls a 500 it should call these two lines, which will
> dump the exception raised to stderr.
>
>    type, value, traceback = sys.exc_info()
>    sys.excepthook(type, value, traceback)

No, really it shouldn't.

I made a quick scan of sites I maintain (or help maintain) in
production now. Among them:

- 3 use the built-in errors over email (and this works fine).
- 1 logs exceptions to a local syslog daemon.
- 2 log to a remote syslogd server (one via TCP, the other via UDP).
- 2 log to stderr (and hence to the apache error log)
- 1 logs to stdout (<-- that's probably a bug :)

This all works *right now* without any changes to Django. Further,
your suggested change would break at least a couple of them.

Look, Django's got a *LOT* of ways to customize error handling and logging.

- Use the logging package directly where needed.
- Write a piece exception middleware.
- Subclass the handler (either ModPythonHandler or WSGIHandler) and
override exception handling.
- Write a WSGI exception middleware.

We're not going to add yet-another-extension-point unless you can
point to a very good reason for it, and you haven't yet.

Jacob

[Kevin Howerton]

I don't think you understand what the code I just posted is doing or
what the issue I described pertains to.

All I'm asking for is sane defaults, and consistency between
deployment platforms. This isn't important?

I think I'm done here.

Thanks for the professionalism.

-k

[Jacob Kaplan-Moss]

On Wed, May 5, 2010 at 3:31 PM, Kevin Howerton <kevin.h...@gmail.com> wrote:
> I don't think you understand what the code I just posted is doing or
> what the issue I described pertains to.

You want Django to globally catch all exceptions and call
sys.excepthook. I'm telling you that's a bad idea -- I've tried it,
and it breaks a number of sites I have in production.

> All I'm asking for is sane defaults, and consistency between
> deployment platforms.  This isn't important?

From where I stand the defaults are perfectly sane, and deployment is
demonstrably consistent -- Django does the same thing regardless of
where it's deployed. You're actually asking for the opposite: a
special case for mod_wsgi.

Can you explain to me why none of the exception hooks already put into
Django is sufficient? I've still not been able to glean that from your
posts.

[Zain Memon]

Jogging also implements a lot of the ideas from Simon's logging proposal, and can do what you're saying: 

http://github.com/zain/jogging/

[Vinay Sajip]

On May 5, 8:33 pm, Mike Axiak <mcax...@gmail.com> wrote:
> Why does this facility need to exist within Django? At least one
> person [1] has written a middleware to do what you're saying. If it's
> that important, people will start to use that app. There are certainly
> many potential features people want out of logging, so it'd definitely
> be better suited for an external app to bake.

I don't think his isn't just about logging for applications: I believe
it's also about logging events within Django itself, which cannot be
done using a middleware or app.

Logging events within Django would be useful for diagnosing certain
types of problems, e.g. those which happen during start-up, request
workflow internal to Django etc.

You don't need special middleware (or anything else) for logging
support in Django applications; the middleware or app is generally
used to do things such as presenting logging output as part of a page,
or to provide some Django-specific handlers (e.g. emailing via Django
configured mailer, rather than using logging's SMTPHandler) and
conventions for logging.

Regards,

Vinay Sajip

[Vinay Sajip]

On May 5, 5:56 pm, Jacob Kaplan-Moss <ja...@jacobian.org> wrote:
> Then next time leave out the rambling, ad hominems, and sweeping
> generalizations. Focus on technical suggestions -- working code is
> even better. If you have problems, suggest fixes concretely.

To be fair, I'm not sure I saw any ad hominem attacks in Kevin's post
(do correct me if you think I'm wrong). He did disparage a point of
view which says that something must be suspect because of "perceived"
Java (or Ruby, or whatever...) heritage, without ascribing it to any
specific person or group.

Perhaps his comment was in response to some content in the original
thread which I referred to, which mentioned the "Java heritage" FUD.
This stems, it seems, from my acknowledging a debt to some good ideas
(not code) from log4j. I make no apologies for this; good ideas can
come from anywhere, not just Python or Django people. The logging PEP
and package were reviewed by numerous committers on Python-dev when
first proposed, and given the nod - so I don't understand how people
can still have doubts about it's "Pythonicity".

As for performance, there's a page on the Python Wiki (http://
wiki.python.org/moin/LoggingPackage) which provides some rough
performance metrics. I have yet to see any substantive criticisms
based on actual measurements.

Just sayin',

Vinay Sajip