[cogen] What do to with this MySQL query that is taking up to several seconds. It blocks.

[TK]

I am using Cogen in a Pylons web application. The Pylons controllers
wait for something to come into "myQueue" and then return the result.
I run another separate coro to poll the database, which put a result
into "myQueue". This setup works, but my problem is with blocking
MySQL queries (I use MySQLdb).

Most queries take just a millisecond or so. However, sometimes a more
complicated query takes longer. I want to allow one user to make this
query without blocking the entire server.

@coro
def process(self):
c = self.dbConn.cursor()
res = c.execute("SELECT SLEEP(5)") # We simulate a long query
c.close()
yield myQueue.publish("Got %d rows" % res)

This would have worked well if it wasn't for the long-lasting query.
What is the best solution to handle this? I consider the following:

1) Forking it, but how is that going to yield back the result?
2) Using a thread, but again I'm not sure how to use this with cogen
efficiently
3) Using a part of the cogen framework that handles this, I am not
sure what...

Any tips are appreciated!

Thanks

--
You received this message because you are subscribed to the Google Groups "cogen" group.
To post to this group, send email to co...@googlegroups.com.
To unsubscribe from this group, send email to cogen+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cogen?hl=en.

[Ionel Maries Cristian]

Basically what you need is a thread (or a pool or thread workers) that runs along cogen's scheduler and communicates with it.

Unfortunately this is a missing feature in cogen. It could be implemented as set of operations that interface to a thread pool or a simple "run in thread" operation.  I'm thinking that the later is better for your case (since you do not want a worker thread pool). 

-- ionel

[TK]

Yes, a "run in thread" operation is what I need. Can you please help
me along with your in-depth knowledge of Cogen. I assume I would
subclass Operation and start that as a new thread, but keeping a
reference to the scheduler?

> > cogen+un...@googlegroups.com <cogen%2Bunsu...@googlegroups.com>.

> > For more options, visit this group at
> >http://groups.google.com/group/cogen?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "cogen" group.
> To post to this group, send email to co...@googlegroups.com.
> To unsubscribe from this group, send email to cogen+un...@googlegroups.com.

> For more options, visit this group athttp://groups.google.com/group/cogen?hl=en.

[Ionel Maries Cristian]

Here's an example of what i'm thinking:

http://code.google.com/p/cogen/source/browse/trunk/examples/inthread.py

Mind you it's not really tested/benchmarked/polished and it has a problem with schedulers that don't have other stuff to run (the sched  's main loop will just terminate).

-- ionel