SQLite Logging

[Yarin]

I'd like to elicit some help in putting together a SQLite-based
logging solution for web2py.

Logging to a file was covered earlier in this forum (See "Global
logging to file"):
http://groups.google.com/group/web2py/browse_thread/thread/e20d0bd2e542aa14/e248314770225225
log.py:
https://sites.google.com/site/ykessler/main/log.py

I've written a SQLite logging handler:
https://sites.google.com/site/ykessler/main/sqlitehandler.py

However, although the SQLite handler works great in normal Python
environments, it errors out with the global logging solution because
the logger emits on multiple threads, and SQLite objects are
restricted to the thread they're created on.

Any ideas on how to sync these two solutions?

[Iceberg]

Just a quick thought. Since web2py itself already handles SQLite db well under multi-thread situation, so a quick tweak to your sqlitehandler.py may be putting the web2py db instance, rather than a filename, to initialize your SQLiteHandler() class.

Best regards,
Iceberg, 2010-Jul-22, 10:28(AM), Thu

----------------------- Original Message -----------------------
From: Yarin <ykes...@gmail.com>
To: web2py-users <web...@googlegroups.com>
Cc: johann.sch...@easytouch-edv.com, mdip...@cs.depaul.edu, ice...@21cn.com
Date: Wed, 21 Jul 2010 19:15:55 -0700 (PDT)
Subject: SQLite Logging
-------------------

[Massimo Di Pierro]

This is useful. I suggest making a web2py slice about this.

Massimo

[Yarin]

Added a slice: Application Logging
http://web2pyslices.com/main/slices/take_slice/91

It covers logging to both a file and SQLite. The code is well
documented- use this instead of the previous code. Would love to get
your comments.

@Iceberg, btw apparently the SQLite cross-thread issues weren't solved
by using the default DB after all. I added a thread-safe work around,
but suggestions are welcome.

On Jul 22, 8:20 am, Massimo Di Pierro <mdip...@cs.depaul.edu> wrote:
> This is useful. I suggest making a web2py slice about this.
>
> Massimo
>
> On Jul 21, 2010, at 9:37 PM, Iceberg wrote:
>
>
>
> > Just a quick thought. Since web2py itself already handles SQLite db  
> > well under multi-thread situation, so a quick tweak to your  
> > sqlitehandler.py may be putting the web2py db instance, rather than  
> > a filename, to initialize your SQLiteHandler() class.
>
> > Best regards,
> >                            Iceberg, 2010-Jul-22, 10:28(AM), Thu
>
> > ----------------------- Original Message -----------------------
> > From:    Yarin <ykes...@gmail.com>
> > To:      web2py-users <web...@googlegroups.com>
> > Cc:      johann.sch...@easytouch-edv.com, mdip...@cs.depaul.edu
> > , ice...@21cn.com
> > Date:    Wed, 21 Jul 2010 19:15:55 -0700 (PDT)
> > Subject: SQLite Logging
> > -------------------
>
> >> I'd like to elicit some help in putting together a SQLite-based
> >> logging solution for web2py.
>
> >> Logging to a file was covered earlier in this forum (See "Global
> >> logging to file"):

> >>http://groups.google.com/group/web2py/browse_thread/thread/e20d0bd2e5...

[mdipierro]

+1

notice that you can use expire_time=None to prevent cache from
expiring instead of 99999999

On Jul 23, 8:24 am, Yarin <ykess...@gmail.com> wrote:
> Added a slice: Application Logginghttp://web2pyslices.com/main/slices/take_slice/91

>
> It covers logging to both a file and SQLite.  The code is well
> documented- use this instead of the previous code. Would love to get
> your comments.
>
> @Iceberg, btw apparently the SQLite cross-thread issues weren't solved
> by using the default DB after all.  I added a thread-safe work around,
> but suggestions are welcome.
>

[Massimo Di Pierro]

Very good work. We should think about a way to include this in web2py
core.

I am not so keen to include the sqlite part because it is too specific
but:

I see the log.py file could go in contrib with minimal changes (it it
were implemented as a function that takes the request and cache objects)

we would need a way to deal with GAE/

[Yarin Kessler]

Glad you like- I'd love to see this as part of the core.  Let me know if there's anything else I can do.

>I am not so keen to include the sqlite part because it is too specific

Is this because it requires the extra module?  I kept it separate because it's a Python, not web2py, specific class, but we could easily bake it into log.py.  

Personally I'm a big fan of SQLite for logging- what would it take to make it less specific and ready for inclusion?  

>we would need a way to deal with GAE/

Hans/Iceberg had a GAE solution in their orig file, but I haven't tested it:

if request.env.web2py_runtime_gae: # if running on Google App Engine 

        handler=logging.handlers.HTTPHandler(request.env.http_host,URL(r=request,f='log')) # assuming there is an optional log action 

    else: 

        handler=logging.handlers.RotatingFileHandler(os.path.join(request.folder,filename),maxBytes=maxBytes,backupCount=backupCount)

I'll re-introduce it into the code though.

[Iceberg]

Well done Yarin! Some comments.

1. Change those time_expire=99999999 into time_expire=None please. The clumsy 99999999 trick was my early "invention" before I know the time_expire=None approach.

2. My another later improvement is to write the log file at {web2py path}/applications/{your app}/static/app.log
so that developer can easily access the log file by http://your_host/your_app/static/app.log
Well, you can refer to the attachment for my latest implementation. Some more trick inside.

3. @Massimo, I suggested to include the applog.py into welcome/model/applog.py since day1 I invented this little component. Only this way people will have an official place to always have the latest code, rather than dig some not-the-latest post from different places. So please do consider it this time.

4. @Yarin, when last time I said "putting the web2py db instance, rather than a filename, to initialize your SQLiteHandler() class", I mean something like:
def _init_sqlite_log(level=logging.DEBUG):
handler = SQLiteHandler( db ) # db is defined by: db = DAL('sqlite://storage.sqlite')
# or
handler = SQLiteHandler( DAL('sqlite://log.sqlite') )
There is some more implementation to do, but you get the idea.

5. With respect to GAE, this was my "concept vehicle":

if request.env.web2py_runtime_gae: # if running on Google App Engine
handler=logging.handlers.HTTPHandler(request.env.http_host,URL(r=request,f='log'))
# assuming there is an optional log action

but I never test it. I think a practical solution would be implement some log handler which wraps GAE's own log facility.

Best regards,
Iceberg, 2010-Jul-24, 02:35(AM), Sat

----------------------- Original Message -----------------------
From: Yarin Kessler <ykes...@gmail.com>
To: Massimo Di Pierro <mdip...@cs.depaul.edu>
Cc: web2py-users <web...@googlegroups.com>, Iceberg <ice...@21cn.com>, Hans <johann.sch...@easytouch-edv.com>, Richard <rich...@gmail.com>, MikeEllis <michael...@gmail.com>, cjparsons <cjpar...@yahoo.co.uk>
Date: Fri, 23 Jul 2010 09:56:17 -0400
Subject: Re: SQLite Logging
-------------------

[mdipierro]

about #3. I would prefer if this were written as a
gluon.contrib.logging module that users can import if needed.

On Jul 23, 2:02 pm, Iceberg <iceb...@21cn.com> wrote:
> Well done Yarin! Some comments.
>
> 1. Change those time_expire=99999999 into time_expire=None please. The clumsy 99999999 trick was my early "invention" before I know the time_expire=None approach.
>
> 2. My another later improvement is to write the log file at {web2py path}/applications/{your app}/static/app.log

> so that developer can easily access the log file byhttp://your_host/your_app/static/app.log

> Well, you can refer to the attachment for my latest implementation. Some more trick inside.
>
> 3. @Massimo, I suggested to include the applog.py into welcome/model/applog.py since day1 I invented this little component. Only this way people will have an official place to always have the latest code, rather than dig some not-the-latest post from different places. So please do consider it this time.
>
> 4. @Yarin, when last time I said "putting the web2py db instance, rather than a filename, to initialize your SQLiteHandler() class", I mean something like:
>     def _init_sqlite_log(level=logging.DEBUG):
>         handler = SQLiteHandler( db ) # db is defined by: db = DAL('sqlite://storage.sqlite')
>         # or
>         handler = SQLiteHandler( DAL('sqlite://log.sqlite') )
> There is some more implementation to do, but you get the idea.
>
> 5. With respect to GAE, this was my "concept  vehicle":
>
>     if request.env.web2py_runtime_gae: # if running on Google App Engine
>         handler=logging.handlers.HTTPHandler(request.env.http_host,URL(r=request,f='log'))
>         # assuming there is an optional log action
>
> but I never test it. I think a practical solution would be implement some log handler which wraps GAE's own log facility.
>
> Best regards,
>                             Iceberg, 2010-Jul-24, 02:35(AM), Sat
>
> ----------------------- Original Message -----------------------

> From:    Yarin Kessler <ykess...@gmail.com>
> To:      Massimo Di Pierro <mdipie...@cs.depaul.edu>
>
> Cc:      web2py-users <web...@googlegroups.com>, Iceberg <iceb...@21cn.com>, Hans <johann.scheibelho...@easytouch-edv.com>, Richard <richar...@gmail.com>, MikeEllis <michael.f.el...@gmail.com>, cjparsons <cjparso...@yahoo.co.uk>

> Date:    Fri, 23 Jul 2010 09:56:17 -0400
> Subject: Re: SQLite Logging
> -------------------
>
> > Glad you like- I'd love to see this as part of the core.  Let me know if
> > there's anything else I can do.
>
> > >I am not so keen to include the sqlite part because it is too specific
>
> > Is this because it requires the extra module?  I kept it separate because
> > it's a Python, not web2py, specific class, but we could easily bake it into
> > log.py.
> > Personally I'm a big fan of SQLite for logging- what would it take to make
> > it less specific and ready for inclusion?
>
> > >we would need a way to deal with GAE/
>
> > Hans/Iceberg had a GAE solution in their orig file, but I haven't tested it:
>
> > if request.env.web2py_runtime_gae: # if running on Google App Engine
>
> >  handler=logging.handlers.HTTPHandler(request.env.http_host,URL(r=request,f='log'))
> > # assuming there is an optional log action
> >     else:
>
> >  handler=logging.handlers.RotatingFileHandler(os.path.join(request.folder,filename),maxBytes=maxBytes,backupCount=backupCount)
>
> > I'll re-introduce it into the code though.
>
> > On Fri, Jul 23, 2010 at 9:38 AM, Massimo Di Pierro

> > <mdipie...@cs.depaul.edu>wrote:

>
> > > Very good work. We should think about a way to include this in web2py core.
>
> > > I am not so keen to include the sqlite part because it is too specific but:
>
> > > I see the log.py file could go in contrib with minimal changes (it it were
> > > implemented as a function that takes the request and cache objects)
>
> > > we would need a way to deal with GAE/
>
> > > On Jul 23, 2010, at 8:24 AM, Yarin wrote:
>
> > >  Added a slice: Application Logging
> > >>http://web2pyslices.com/main/slices/take_slice/91
>
> > >> It covers logging to both a file and SQLite.  The code is well
> > >> documented- use this instead of the previous code. Would love to get
> > >> your comments.
>
> > >> @Iceberg, btw apparently the SQLite cross-thread issues weren't solved
> > >> by using the default DB after all.  I added a thread-safe work around,
> > >> but suggestions are welcome.
>
>
>

>  applog.py
> 1KViewDownload

[Yarin Kessler]

@Iceberg- Thanks for all the input- I'll review your latest and update the code later this weekend.  With respect to log file location though, don't think it makes sense to put it in static since it's a file that will be written to (i.e. non static).  It may need its own location, a var/ or log/...  Massimo, thoughts on this?

[Iceberg]

Sorry I disagree. No need to take "static" literally. In web2py,
"static" just means the content inside such directory will be served
directly by web2py core, unlike other "dynamic" content served by
app's controller. So it is nothing wrong to serve log file inside
"static", no to mention the byproduct, a good one, that you log files
can be access via HTTP natively.

> > From:    Yarin Kessler <ykess...@gmail.com>
> > To:      Massimo Di Pierro <mdipie...@cs.depaul.edu>
> > Cc:      web2py-users <web...@googlegroups.com>, Iceberg <iceb...@21cn.com>,
> > Hans <johann.scheibelho...@easytouch-edv.com>, Richard <
> > richar...@gmail.com>, MikeEllis <michael.f.el...@gmail.com>, cjparsons <
> > cjparso...@yahoo.co.uk>

> > > <mdipie...@cs.depaul.edu>wrote:

[mdipierro]

I think the place for this stuff is "private". From a security point
of view log files should not by default be exposed to the public.

[Iceberg]

That way the developer need to implement a dedicate action, perhaps
protected by authentication, to access log on WEB. It is reasonable
for public site, but an overkill for small enterprise intranet app.

But there is no need to argue on this, man. We can have both. If Yarin
gonna implement a module in web2py/gluon/contrib, please do design a
log_path parameter, which can have a default value as "private". So
everybody will be happy. :-)

[Yarin]

+1 Think this is the right way to go. I'm working on the module now,
should have something to show within next couple days..