[web2py] Application logging

[Keith Edmunds]

What are others doing for application logging? I'm not referring to the
HTTP logs, but program-generated logs.

I've recently been looking at the Python 'logging' module, which I've not
used before. I initiate logging from a file in the 'modules' directory
which does this:

--------------------------------------------------------------------------------
import logging
from logging.handlers import SysLogHandler

logger = logging.getLogger("MyApp")
logger.setLevel(logging.DEBUG)
hdlr = SysLogHandler(address='/dev/log')
formatter = logging.Formatter('tigerpy[%(process)d]: %(levelname)s:
%(filename)s at line %(lineno)d: %(message)s') hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
--------------------------------------------------------------------------------

Then each file that I want to log from does this:

--------------------------------------------------------------------------------
import logging

logger = logging.getLogger("MyApp")
.
.
logger.info("Something interesting happened")
--------------------------------------------------------------------------------

This logs to syslog: the only problem is that each logging message is
written to syslog many (20-40) times(!). This isn't just in one place in
my code, but throughout. I don't think the code is being executed multiple
times, so I'm not sure why I'm getting so many log entries.

So, I'm interested in what others do to log application events.


--
Subscription settings: http://groups.google.com/group/web2py/subscribe?hl=en

[knitatoms]

I'd be interested in a step by step guide to setting up this kind of
logging. I'm new to web2py and python and I think it would help my
learning.

I already get some good info by monitoring the http log file with the
free Kiwi log viewer (on Windows) which tails the file.

Could someone explain step by step how to set up the Python logging
module to write to a file from a web2py application. Thanks!

[omicron]

In 'models' folder, i create the file named '0.py' like this :

def _init_log():
import os, logging, logging.handlers
logger = logging.getLogger(request.application)
logger.setLevel(logging.DEBUG)
handler =
logging.handlers.RotatingFileHandler(os.path.join(request.folder,'static','applog.txt'),'a',
1024*1024,1)
handler.setLevel(logging.DEBUG)
handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %
(filename)s:%(lineno)d %(funcName)s: %(message)s"))
logger.addHandler(handler)
return logger

app_logging = cache.ram('app_wide_log', lambda:_init_log(),
time_expire=None)

and now in my application, i can write this :

app_logging.info("My message")
or
app_logging.debug("x = %d" % myvar)

Et voilà !

[Keith Edmunds]

On Tue, 27 Apr 2010 09:31:50 -0700 (PDT), mink...@gmail.com said:

> I'd be interested in a step by step guide to setting up this kind of
> logging.

I found some older posts that helped in the end. I now have a
modules/logging.py file that looks like:

def _init_log():
"""
From http://article.gmane.org/gmane.comp.python.web2py/11091

"""

import logging
from logging.handlers import SysLogHandler

logger = logging.getLogger(request.application)
logger.setLevel(logging.DEBUG)
handler = SysLogHandler(address='/dev/log')
handler.setLevel(logging.DEBUG)
handler.setFormatter(logging.Formatter(
'%s' % request.application + '[%(process)d]: %(levelname)s:
%(filename)s at line %(lineno)d: %(message)s'))
logger.addHandler(handler) return logger

logging=cache.ram('once',lambda:_init_log(),time_expire=99999999)

Then, when I want to log something, I do:

logging.debug("blah")

".debug" can be the other standard values including ".exception",
".error", ".info", etc

I don't claim originality for the above: I adapted it slightly from the
URL quoted.

Given the unexpected challenges in implementing this in web2py, I think it
would make sense to include it in the scaffolding (but I'm probably wrong).

[Iceberg]

I happened to be the creator of that code snippet. :-) Glad to know
that it helps you.

But I would recommend you use a later version, quoted already in
Omicron's post. It uses RotatingFileHandler to avoid unstoppable log
file size, and it serves the log by http://yourhost/yourapp/static/applog.txt.

And the only downside (of both version) is that it does not work on
GAE (yet?). A simple workaround might be turn it off when "if
running_on_gae==True".

Besides, I once suggested Massimo to include it inside the scaffolding
app, but no agreement is reached.

Regards,
iceberg

On Apr28, 2:02am, Keith Edmunds <k...@midnighthax.com> wrote:

> On Tue, 27 Apr 2010 09:31:50 -0700 (PDT), minkto...@gmail.com said:
>
> > I'd be interested in a step by step guide to setting up this kind of
> > logging.
>
> I found some older posts that helped in the end. I now have a
> modules/logging.py file that looks like:
>
> def _init_log():
>     """

>     Fromhttp://article.gmane.org/gmane.comp.python.web2py/11091

[mdipierro]

Can it be moved to web2pyslices?

On Apr 28, 1:57 pm, Iceberg <iceb...@21cn.com> wrote:
> I happened to be the creator of that code snippet. :-) Glad to know
> that it helps you.
>
> But I would recommend you use a later version, quoted already in
> Omicron's post. It uses RotatingFileHandler to avoid unstoppable log

> file size, and it serves the log byhttp://yourhost/yourapp/static/applog.txt.

[Keith Edmunds]

Hi Iceberg

> But I would recommend you use a later version, quoted already in
> Omicron's post. It uses RotatingFileHandler to avoid unstoppable log
> file size

Thanks for providing it. I can see the sense of your suggestion; however,
I want the logging to be to the standard syslog files (which are
logrotated independently).

Massimo: I would have thought that application logging was an integral
part of an enterprise application, much as authentication is, and should
be in the scaffolding. It really isn't very big.

Keith

[mdipierro]

It may go in tools. What do other people think?

[Jonathan Lundell]

On Apr 29, 2010, at 7:22 AM, mdipierro wrote:

> It may go in tools. What do other people think?

Logging is important enough for that, yes.

WRT Keith's comment below: Python logging offers a rich collection of logging mechanisms, one of which is syslog, which in turn offers a rich collection of logging mechanisms. For many applications (including mine) we need unified logging with other processes, not necessarily Python. So syslog becomes the unifying mechanism for all our logging.

I can see an argument for *not* using syslog, especially for self-contained apps that may not have access to syslog, or where you don't want to bother setting up a syslog environment. (What's the logging convention for GAE, btw?)

[szimszon]

I think too that logging is important.

And I vote for syslogging :)

[Mathieu Clabaut]

The use of python logging modules inside web2py will ideally allow people to use syslog or any other logging back end provided by this module.

It would be very nice if web2py integration would allow logging for GAE (into database or whatever mechanism is provided by GAE..). 

Perhaps we should have a look at http://code.google.com/p/gae-logger/

-Mathieu

[Thadeus Burgess]

+1 for a enterprise logging system that you can configure the backend :)

--
Thadeus

[pbreit]

What is current recommendation for app logging?

[Jonathan Lundell]

On Feb 12, 2011, at 9:44 PM, pbreit wrote:
> What is current recommendation for app logging?

You can start by copying logging.example.conf to logging.conf, and then editing it to suit your requirements.

Either log with web2py.app (logger_app) or create a new logger specific to your own application.

[Michael McGinnis]

I can't find documentation in the book for this, either by searching
for logging.conf, logger.app or web2py.app. Maybe I'm lazy or
ignorant. But I love the usability consulting part of my job: anytime
something isn't extremely easy for me to find on a website, I can
claim that it's not my fault, that the site should be made more usable
for lazy, ignorant people.

[Jonathan Lundell]

On Feb 13, 2011, at 5:41 AM, Michael McGinnis wrote:
>
> I can't find documentation in the book for this, either by searching
> for logging.conf, logger.app or web2py.app. Maybe I'm lazy or
> ignorant. But I love the usability consulting part of my job: anytime
> something isn't extremely easy for me to find on a website, I can
> claim that it's not my fault, that the site should be made more usable
> for lazy, ignorant people.

It's a feature that was added since the book was written.

It's a standard configuration file for standard Python logging. The example configuration file is intended as a starter for those of us who aren't familiar with Python logging to begin with.

You're right; a book section would be nice.