Re: Best way to architect background tasks

[more10]

Hi

I'm pretty new to tornado as well. But I am doing something like you want, but I have only one thread. I am also using websockets.

Think of tornado as the gui thread in any other environment. You can schedule execution on the tornado thread with tornado.ioloop.IOLoop.instance().add_callback(), this is how you call the gui from your thread.

Communication from the gui thread to your user thread could be done by queue.Queue, one per user thread. They are treadsafe.

Best Regards

Mårten

Den tisdagen den 14:e augusti 2012 kl. 18:58:56 UTC+2 skrev Michael Williamson:

Hello Tornado experts, I'm a tornado newbie, so I figured I'd ask here before I started hacking away.

Here's my use case:

Basically, I'm using tornado because I found its websocket library compelling, and I also like the fact that it can handle thousands of websocket connections concurrently without much trouble.

What I need, however, is some way to integrate this with imaplib.  i.e. I was thinking I'd have a background thread per user which would continually look for new mail.  Yes, yes, we could get in a big argument here of non-blocking versus threads, etc.  I couldn't find a non-blocking imap library out there, let alone one that's been written on top of IOLoop.

So, assuming that I was going to use a thread per user model, what would be the best way to go about it.  Would it be too ridiculous to just spawn a thread from within my tornado websocket handler?  Maybe I would need to go all out and use a message queue with a background service that dealt with imaplib....it's just that that seems very heavyweight for what I'm trying to accomplish.

Thanks for any advice,

Mike

[Nick Jennings]

Hi Michael,

I'm also fairly new to tornado, however I have several background
tasks which need to be handled and sent up to their respective client
when something eventful happens.

What I use is Redis, for messaging between listeners (tornado) and
workers (normal python threads).. I have a master class which spawns
the listener(s) and workers, and use a Redis list to queue up the jobs
(only one worker gets an element, to ensure two workers not working on
the same thing). Then, each listener thread subscribes to a Redis
subscription channel. When the worker is done, it broadcasts it's
results to the channel applicable to the client it was working for,
and immediately goes back to the queue to get another job.

For your scenario, you could have a subscription channel per client,
and everytime a new client comes into the mix, you pop on a job for
them on the worker queue. When the listener gets a message on their
subscription channel (worker is completed) it sends the results back
to the client and re-adds the client job to the worker queue. This way
you could have any number of workers (I don't think it's a good idea
to have one per-client, that could get really messy) and they
continually are checking mail for the clients as soon as they are
freed up.

Redis is really fast (stays in memory) and there are async client
libraries so you don't even have to wait for the job to be posted.

Cheers
nick