@web.background gotcha

[Berwyn Hoyt]

We've hit a bad gotcha in the @web.background function decorator. We
are using the database in both the main request handler and also in the
background task.

In our case, the web user can click a button that starts a stack of
emails being sent out. We don't want the user to have to wait minutes
for the response to his request, so we're decorating our email()
function with @web.background so that the main request handler can get
back to the user immediately.

The trouble is that our background email() function also uses the
database to fetch a list of users to email. Unfortunately, we think
this means that both the foreground and background tasks are using the
same database cursor. Anyway, bad things happen in a randomy
time-related sort of way. The database gets corrupted, invalid data
gets returned, and (strangely) apache sometimes even crashes.

We *think* we have identified the cause and the solution, but we'd like
it verified by some others in the know. Simply make the background task
call web.db.connect before it does anything. We haven't had any
problems *yet* since we implemented this.

Below is a patch to web.background(). Does it make sense?

Thanks,
Berwyn

Index: http.py
===================================================================
--- http.py (revision 168)
+++ http.py (working copy)
@@ -16,7 +16,7 @@
import sys, os, threading, urllib, urlparse
try: import datetime
except ImportError: pass
-import net, utils, webapi as web
+import net, utils, webapi as web, db as webdb

def prefixurl(base=''):
"""

@@ -159,13 +159,15 @@

def newfunc():
web._context[threading.currentThread()] = tmpctx
+ if config.get('db_parameters'):
+ webdb.connect(**web.config.db_parameters)
func(*a, **kw)
myctx = web._context[threading.currentThread()]
for k in myctx.keys():
if k not in ['status', 'headers', 'output']:
try: del myctx[k]
except KeyError: pass
-
+
t = threading.Thread(target=newfunc)
background.threaddb[id(t)] = t
t.start()

[Aaron Swartz]

> Below is a patch to web.background(). Does it make sense?

Yep, that makes sense to me.

[Berwyn Hoyt]

I don't know if anyone used my patch below for the @web.background decorator, but I'm afraid I only thought it was working.  Here is the correction:

-            if config.get('db_parameters'):
+            if web.config.get('db_parameters'):

Cheers,
Berwyn