Doing threading 'properly'
Ben Sizer
background threads as well as the main thread. I use MySQL for the
database and use sqlobject.util.threadinglocal in the background thread
as shown in the "Creating a background batch processing thread" topic
on this group. (If it made it into the docs, I don't see it.)
This appears to work. However, threaded applications are notorious for
appearing to work, and then sporadically not working. I want to ensure
everything is safe, at least as far as doing the right thing with each
library is concerned. Are there any special considerations I need to
take into account? Do I need to use the woefully underdocumented
sqlobject.util.threadinglocal in my main thread too? Is a user's
Identity thread-safe? Is there anything SQLObject/MySQL/Turbogears
specific I need to know?
--
Ben Sizer
Jorge Godoy
Back to when I started with TurboGears and when I needed something like that I
coded the tasks as URLs and scheduled a job to access them and process their
output.
This looked to me at the time -- and today I have the same opinion -- that it
was the simpler approach I could have and ensure that all behaved the same.
This allowed to have several tasks running simultaneously without having to
worry about how the app would see changes done outside of its usual
environment since everything was the app itself.
For some long running tasks -- I still haven't coded them -- I plan on having
a redirect and refreshing the page with new status until I get the desired
information (probably using some META trick since text browsers aren't
JS-friendly).
Maybe this might be your case as well. This also has the advantage of making
tests simpler...
--
Jorge Godoy <jgo...@gmail.com>
Ben Sizer
> Back to when I started with TurboGears and when I needed something like that I
> coded the tasks as URLs and scheduled a job to access them and process their
> output.
>
> This looked to me at the time -- and today I have the same opinion -- that it
> was the simpler approach I could have and ensure that all behaved the same.
Yeah... I suppose that if the app handles requests in a thread-safe
manner then you can just make 2 requests and assume that they will be
handled correctly.
However, I still wonder if things like identity are handled properly
there - eg. if I'm logged in, make a request in one window to log out,
and in the other window I submit a form which repeated accesses
identity.current in the controller, what will happen? Exception raised
in the form's controller? One request or the other gets blocked until
the other completes? What about SQLObject - does it have any caching,
and if you modify the same object in different threads what happens to
that cache?
I don't think my problem is well suited to the additional request
method, as what I have is a task that needs to be started when a
particular page is submitted, but which needs to continue in the
background as it takes too long, modifying the database when it's
finished. I'd rather not have to pass everything off to an external
queue if possible, especially when running a function via the threading
module is so clean and simple. But I'd like some assurance that it's
safe to do so.
--
Ben Sizer
Jorge Godoy
> However, I still wonder if things like identity are handled properly
> there - eg. if I'm logged in, make a request in one window to log out,
> and in the other window I submit a form which repeated accesses
> identity.current in the controller, what will happen? Exception raised
> in the form's controller? One request or the other gets blocked until
> the other completes? What about SQLObject - does it have any caching,
> and if you modify the same object in different threads what happens to
> that cache?
Using tabs it is handled properly: both share the same session and when I
logout it expires, so when the other tab tries accessing the app it reaches an
expired id and is redirected to the login screen.
With regards to SO's cache, I believe you'll get inconsistencies and this is
one of the problems we faced in the beginning of TG. The solution was to
embed all exposed methods inside their own transactions to force SO to read
data again when it entered a new transaction.
> I don't think my problem is well suited to the additional request
> method, as what I have is a task that needs to be started when a
> particular page is submitted, but which needs to continue in the
> background as it takes too long, modifying the database when it's
> finished. I'd rather not have to pass everything off to an external
> queue if possible, especially when running a function via the threading
> module is so clean and simple. But I'd like some assurance that it's
> safe to do so.
On the TG side I believe that as long as they're all inside transactions it
will be safe that you'll get results right from the database. On your
external code side you'd better adopt a similar approach starting and
commiting transactions explicitly. SO will honor your request to commit data
and when the transaction on the TG app reads from the DB it will get this
commited data from it.
--
Jorge Godoy <jgo...@gmail.com>
Ben Sizer
> "Ben Sizer" <kylo...@gmail.com> writes:
> > However, I still wonder if things like identity are handled properly
> > there - eg. if I'm logged in, make a request in one window to log out,
> > and in the other window I submit a form which repeated accesses
> > identity.current in the controller, what will happen? Exception raised
> > in the form's controller? One request or the other gets blocked until
> > the other completes? What about SQLObject - does it have any caching,
> > and if you modify the same object in different threads what happens to
> > that cache?
> Using tabs it is handled properly: both share the same session and when I
> logout it expires, so when the other tab tries accessing the app it reaches an
> expired id and is redirected to the login screen.
So that means that each request is handled sequentially... is that
guaranteed by TurboGears?
> On the TG side I believe that as long as they're all inside transactions it
> will be safe that you'll get results right from the database. On your
> external code side you'd better adopt a similar approach starting and
> commiting transactions explicitly. SO will honor your request to commit data
> and when the transaction on the TG app reads from the DB it will get this
> commited data from it.
Ok... do you know how that ties in with the use of
'sqlobject.util.threadinglocal'?
--
Ben Sizer
Jorge Godoy
> So that means that each request is handled sequentially... is that
> guaranteed by TurboGears?
It isn't guaranteed by TG. I believe that if I started two transactions on
the DB, the second one starting before the first expires the visit. Thinking
about PostgreSQL that uses MVCC -- so this becomes database dependent -- then
I'd have two snapshots one with the visit expired and the second with the
visit still OK.
For this specific second request it'd still be able to retrieve data from the
database, but a third transaction started right after the expiration has been
commited by the first transaction wouldn't be able to do it since in this new
snapshot the visit would have already expired.
Adding to this there's also the isolation level from the database (it is
possible to serialize all transactions, it is possible to read uncommited
data, etc. -- it all depends on your database implementation of these
isolation levels).
I dunno if I was clear here. If not, ask for more :-)
> Ok... do you know how that ties in with the use of
> 'sqlobject.util.threadinglocal'?
No. I haven't used threads with SO. And I'm holding myself on it, thinking
about doing new apps with SA...
--
Jorge Godoy <jgo...@gmail.com>
Ben Sizer
> "Ben Sizer" <kyl...@gmail.com> writes:
>
> > So that means that each request is handled sequentially... is that
> > guaranteed by TurboGears?
>
> It isn't guaranteed by TG. I believe that if I started two transactions on
> the DB, the second one starting before the first expires the visit. Thinking
> about PostgreSQL that uses MVCC -- so this becomes database dependent -- then
> I'd have two snapshots one with the visit expired and the second with the
> visit still OK.
>
> For this specific second request it'd still be able to retrieve data from the
> database, but a third transaction started right after the expiration has been
> commited by the first transaction wouldn't be able to do it since in this new
> snapshot the visit would have already expired.
>
> Adding to this there's also the isolation level from the database (it is
> possible to serialize all transactions, it is possible to read uncommited
> data, etc. -- it all depends on your database implementation of these
> isolation levels).
>
> I dunno if I was clear here. If not, ask for more :-)
You were probably clear, but I don't know much about the detail of
database transactions and so on. But even if I drop down to the
database level and make the update directly, can I be sure that the
correct values propagated up to SQLObject in the other threads?
> > Ok... do you know how that ties in with the use of
> > 'sqlobject.util.threadinglocal'?
>
> No. I haven't used threads with SO. And I'm holding myself on it, thinking
> about doing new apps with SA...
This all worries me... threads are surely a pretty fundamental part of
many long-running process models, yet nobody except yourself is able to
tell me what is safe and what is not. I at least need to know what sort
of exceptions I need to be able to catch and which circumstances they
may arise in, and which objects in the Turbogears system I can pass
between Python threads and those which I can't.
--
Ben Sizer
Jorge Godoy
> You were probably clear, but I don't know much about the detail of
> database transactions and so on. But even if I drop down to the
> database level and make the update directly, can I be sure that the
> correct values propagated up to SQLObject in the other threads?
In a standard TG application, the way it is designed in 1.0+ you should be
safe on every exposed method. It starts a transaction that reads data from
the DB.
I believe that the only way you'll guarantee that it works for you is writing
a small test case. One thread modifies data and the other reads it.
> This all worries me... threads are surely a pretty fundamental part of
> many long-running process models, yet nobody except yourself is able to
> tell me what is safe and what is not. I at least need to know what sort
> of exceptions I need to be able to catch and which circumstances they
> may arise in, and which objects in the Turbogears system I can pass
> between Python threads and those which I can't.
Hmmm... Due to the nature of web applications -- several concurrent accesses
-- much of TG is thread safe. In fact there's even some code to prevent that
a developer do some unsafe operations.
It isn't hard testing under TG so you can check complex things as you need.
--
Jorge Godoy <jgo...@gmail.com>
Ben Sizer
> "Ben Sizer" <kyl...@gmail.com> writes:
>
> > You were probably clear, but I don't know much about the detail of
> > database transactions and so on. But even if I drop down to the
> > database level and make the update directly, can I be sure that the
> > correct values propagated up to SQLObject in the other threads?
>
> In a standard TG application, the way it is designed in 1.0+ you should be
> safe on every exposed method. It starts a transaction that reads data from
> the DB.
>
> I believe that the only way you'll guarantee that it works for you is writing
> a small test case. One thread modifies data and the other reads it.
It's not really possible to use test cases for synchronisation problems
though. They might work 10000 times in testing and then fail the
10001st time when the instruction interleaving differs slightly, due to
the system load being higher, or some cache being more or less full,
etc.
If you know which methods are synchronised and which ones aren't, then
it's possible to prove whether it will always work or not.
Unfortunately I don't have this sort of information and I was hoping
someone here would.
> > This all worries me... threads are surely a pretty fundamental part of
> > many long-running process models, yet nobody except yourself is able to
> > tell me what is safe and what is not. I at least need to know what sort
> > of exceptions I need to be able to catch and which circumstances they
> > may arise in, and which objects in the Turbogears system I can pass
> > between Python threads and those which I can't.
>
> Hmmm... Due to the nature of web applications -- several concurrent accesses
> -- much of TG is thread safe. In fact there's even some code to prevent that
> a developer do some unsafe operations.
But I need to know what it does to prevent it; does it use mutual
exclusion to ensure that certain operations happen atomically? If so,
which ones? If not, does it throw exceptions when an operation will
fail and back out or roll back; if so, from where, and which
exceptions? Someone posted a recipe involving an sqlobject thread-local
method and nobody seems to know what it does or how important it is.
It may be that I'm worrying about nothing, but it's also not possible
for me to exhaustively test every possible interleaving of Python
bytecode to see if that is the case.
--
Ben Sizer
Bob Ippolito
>
> Jorge Godoy wrote:
> > "Ben Sizer" <kyl...@gmail.com> writes:
> >
> > > You were probably clear, but I don't know much about the detail of
> > > database transactions and so on. But even if I drop down to the
> > > database level and make the update directly, can I be sure that the
> > > correct values propagated up to SQLObject in the other threads?
> >
> > In a standard TG application, the way it is designed in 1.0+ you should be
> > safe on every exposed method. It starts a transaction that reads data from
> > the DB.
> >
> > I believe that the only way you'll guarantee that it works for you is writing
> > a small test case. One thread modifies data and the other reads it.
>
> It's not really possible to use test cases for synchronisation problems
> though. They might work 10000 times in testing and then fail the
> 10001st time when the instruction interleaving differs slightly, due to
> the system load being higher, or some cache being more or less full,
> etc.
>
> If you know which methods are synchronised and which ones aren't, then
> it's possible to prove whether it will always work or not.
> Unfortunately I don't have this sort of information and I was hoping
> someone here would.
Audit the source then. See what kind of locking it tries to do.
The safest thing to do of course is to only send immutable objects
across threads, or deep copies of mutable objects (message passing).
Pretty hard to screw that up.
-bob
ben.ha...@gmail.com
because the sql is simple it has a hacked version of threading in it
and it should show how I did this. It is not scientific per se but it
does work. Also it is weird and spooky that google now has a
customizeable search feature that looks much like it except for a
search interval variable. As I recall mysqldb was pretty fast and
postgres would have been better.
It uses borrowed code for the threading if I recall and the borrowed
code was actually not right! But it works I believe and its not too
slow really it just needs page rank or something and postgres. I
experimented with locking, postgres has this too I think its maybe
different and not as safe I'm not sure. Locking truely is a very bad
thing I found but was necessary at least at one point.
If you use postgres the api might not be as good for threading maybe
but it might even be better.
Inserts went like lightning make sure to use the simplest sql you can
get by with if you want performance this is a complicated subject
though.
Ben Sizer
> On 11/5/06, Ben Sizer <kyl...@gmail.com> wrote:
> > If you know which methods are synchronised and which ones aren't, then
> > it's possible to prove whether it will always work or not.
> > Unfortunately I don't have this sort of information and I was hoping
> > someone here would.
>
> Audit the source then. See what kind of locking it tries to do.
I'm not exactly well-qualified to understand all that code though. And
between the various subprojects, there is certainly a lot of code.
> The safest thing to do of course is to only send immutable objects
> across threads, or deep copies of mutable objects (message passing).
> Pretty hard to screw that up.
Sure, but obviously when you use a framework like TurboGears, a lot
takes place under the hood. And I think it's important to document
these things.
For example, I was concerned that if 2 threads each created an
SQLObject-derived instance that referred to the same underlying data,
would there be some sort of clash? I found in the SQLObject
documentation that it ensures you get the same object in both threads,
which is at least some help, though it is still a little vague. ("If
you ask for a person by a particular ID more than once, you'll get back
the same instance. This way you can be sure of a certain amount of
consistency if you have multiple threads accessing the same data
(though of course across processes there can be no sharing of an
instance). This isn't true if you're using transactions, which are
necessarily isolated.)
I expect that the above guarantee will be enough for my limited use of
threaded database access, but others may have more complex questions
which I hope someone can answer later.
--
Ben Sizer