Using Durus with Pylons
Mike Orr
how to do it in my applications. Durus is not thread safe so I need a
separate connection for each thread. I just want to make sure my
approach is thread correct.
quickwiki.lib.database has a function to open a connection:
def get_connection(config):
"""'config' is PasteDeploy's 'app_config' object.
This function is thread agnostic.
"""
try:
host = config["durus.host"]
port = int(config["durus.port"])
except KeyError:
raise keyError("config keys 'durus.host' and/or
'durus.port' are missing")
storage = ClientStorage(host, port) # Imported from
durus.client_storage
return Connection(storage) # Imported from durus.connection
quickwiki.lib.app_globals contains:
class Globals(object):
def __init__(self, global_conf, app_conf, **extra):
self.db_connection = get_connection(app_conf)
quickwiki.websetup contains:
def setup_config(command, filename, section, vars):
app_conf = appconfig("config:"+filename) # Imported from paste.deploy
conn = get_connection(app_conf)
If the Globals object is thread-specific, I should be safe using
g.db_connection, right?
I had to pass the configuration dict into get_connection rather than
reading the config there because I was getting "TypeError: No
configuration has been registered for this process or thread".
Apparently the environment is different in the two circumstances so
you have to read the config a different way in each.
There appears to be a bug in the quickwiki.lib.app_globals.Globals
docstring. It says "class attributes" inside the .__init__ method,
where you're really creating instance attributes.
--
Mike Orr <slugg...@gmail.com>
Shannon -jj Behrens
Hey Mike! :)
The globals object is *not* thread-specific! I suggest you instead
create the connection in lib/base.py within the __call__.
Happy Hacking!
-jj
Mike Orr
Then I'll be creating a connection for every request. Is there no
better place to do it once per thread?
--
Mike Orr <slugg...@gmail.com>
climbus
http://pylonshq.com/project/pylonshq/wiki/ZODBMiddleware.
I did't find windows version of durus, so i cant make a durus
middlware.
Climbus
James Gardner
>> The globals object is *not* thread-specific! I suggest you instead
>> create the connection in lib/base.py within the __call__.
>
> Then I'll be creating a connection for every request. Is there no
> better place to do it once per thread?
Well, the thing is Pylons is thread-safe but doesn't actually manage any
of the creating of the threads itself. Thread creation is handled by
whichever server is running the Pylons app. This means if you want one
connection *per thread* rather than *per request* you would probably
have to set it up in the server part.
For example. If you were using a paste.httpserver you could write your
own thread handling that added one connection to the environ when each
thread was created. You would then be able to access the connection in
your Pylons app via request.environ.
Hope that helps,
James
Shannon -jj Behrens
I'm surprised by this question. Doesn't every request get a new
thread? I didn't think the threads were getting reused. If so, and
if you're worried about the cost of creating new connections, you
should be using a connection pool; sorry, I know that's a *duh* kind
of statement ;)
Mike Orr
>
> On 12/11/06, Mike Orr <slugg...@gmail.com> wrote:
> > > > quickwiki.lib.app_globals contains:
> > > >
> > > > class Globals(object):
> > > > def __init__(self, global_conf, app_conf, **extra):
> > > > self.db_connection = get_connection(app_conf)
> > > >
> > > > g.db_connection, right?
> > > >
> > > create the connection in lib/base.py within the __call__.
> >
> > Then I'll be creating a connection for every request. Is there no
> > better place to do it once per thread?
>
> I'm surprised by this question. Doesn't every request get a new
> thread? I didn't think the threads were getting reused.
Each thread processes several requests from a shared queue, see
paste.httpserver.ThreadPool.worker_thread_callback(). The object in
the queue is a lambda defined in
paste.httpserver.ThreadPoolMixin.process_request(). The lambda does
all the request processing including sending the response to the
client. If the queue element is a special SHUTDOWN value, the thread
exits.
So is the globals object really thread-unsafe? I thought it was a
StackedObjectProxy (in pylons.__init__) precisely for this purpose, so
you can put something like a thread-specific database connection
there. The docstring for 'paste.registry' says this if I'm reading it
correctly:
[StackedObjectProxy is a ...]
"""Registry for handling request-local module globals sanely
Dealing with module globals in a thread-safe way is good if your
application is the sole responder in a thread, however that approach fails
to properly account for various scenarios that occur with WSGI applications
and middleware.
What is actually needed in the case where a module global is desired that
is always set properly depending on the current request, is a stacked
thread-local object. Such an object is popped or pushed during the request
cycle so that it properly represents the object that should be active for
the current request.
To make it easy to deal with such variables, this module provides a special
StackedObjectProxy class which you can instantiate and attach to your
module where you'd like others to access it. The object you'd like this to
actually "be" during the request is then registered with the
RegistryManager middleware, which ensures that for the scope of the
current WSGI application everything will work properly."""
> If so, and
> if you're worried about the cost of creating new connections, you
> should be using a connection pool; sorry, I know that's a *duh* kind
> of statement ;)
Durus doesn't have a connection pool but I could write one a la
SQLAlchemy if it becomes necessary. But a thread-local variable would
be a simpler equivalent. I'm trying to avoid creating my own
StackedObject proxy and registering it and creating a middleware if I
don't have to.
I finally ended up putting the connection in
MyController.__before__(). That way it's created only in controllers
that actually use the database,and it's one less module to modify.
But I still want to see if it's possible to use one connection for
several requests.
--
Mike Orr <slugg...@gmail.com>
Damjan
> MyController.__before__(). That way it's created only in controllers
> that actually use the database,and it's one less module to modify.
> But I still want to see if it's possible to use one connection for
> several requests.
The usual way is to use a global dictionary that holds the connections
indexed by the thread id.
Something like this should work:
id = thread.get_ident()
global conn_cache
conn = conn_cache.get(id, newconnection())
Something like this is implemented in the threading.local() class which
is new in Python 2.4
Mike Orr
connections in Pylons. It works in my test, using a TCP socket to the
database server and doing multiple requests until a thread is reused.
Are there any flaws in the coding?
Is CONFIG["global_conf"]{"__file__"] the best way to get the config
file's path? Will it always be defined?
To test if the socket is still alive, I do a makeshift "version"
command to the server. I think Durus ClientStorage needs a dedicated
method for this. (.ping() or .is_connected()) (Note: this catches
IOError if the server has been killed or restarted.)
Is there a way to do the equivalent of thread.get_ident() from the
threading module or Thread objects? This seems to be the only way to
get a thread ID.
Is there a way to get an application instance ID to distinguish
between multiple instances of the same application? This is not the
same as id(the_controller), which would change every request.
--
Mike Orr <slugg...@gmail.com>
Damjan
and cache them:
from durus.client_storage import ClientStorage
from durus.connection import Connection
import threading
def connect(address):
tls = threading.local()
try:
connection = tls.connection
except:
connection = Connection(ClientStorage(address=address))
tls.connection = connection
return connection
I'm using it here with Unix domain sockets, so I just need one
'address' parameter.
(BTW I think Durus should unify the address like it's with the Python
sockets, but that's a whole another story).
Damjan
if 'tls' would've been a global variable it would made some sense