Gevent vs Python's multiprocessing?

[Juan Luis Cano]

Hello everyone,

I have an interest in asynchronous programming for the web, I have been coding in Python for a short time now and recently I discovered gevent and get amazed with its simplicity :)

My first question is: I was looking at gevent.pool and gevent.queue, and I read they both have a similar API than its relative modules in the Python standard library, part of the multiprocessing package. I was wondering how do they compare in terms of usage, performance, etc for implementing a basic pool of workers. Has anyone done any informal benchmarks on them? Why would you recommend me to use gevent (or why not)? Are there any cases in which one might be better than the other?

Thank you in advance!

[Leif K-Brooks]

Hey Juan,

Basically, gevent and multiprocessing are for different use cases. If
your application needs to do heavy IO, with a large number of
concurrently-open sockets, gevent should work much better for you than
mutliprocessing. Greenlets are very lightweight, so you can have very
many of them active at once without any trouble.

On the other hand, if your program needs to do a lot of CPU-heavy work
in parallel, you'll probably want to use multiprocessing, because
greenlets can't run on multiple CPU cores.

Leif

[Juan Luis Cano]

Hi Leif,

Thanks for your answer. In case I had to do CPU-intensive computations, what do you think about putting gevent behind a reverse proxy, like Nginx with uWSGI? Could I take advantage of more than one core then? I am a total noob about this, and I don't know if this would help to improve the situation in that case. 

[Roberto De Ioris]

Il giorno 26/giu/2011, alle ore 16.35, Juan Luis Cano ha scritto:

> Hi Leif,
>
> Thanks for your answer. In case I had to do CPU-intensive computations, what do you think about putting gevent behind a reverse proxy, like Nginx with uWSGI? Could I take advantage of more than one core then? I am a total noob about this, and I don't know if this would help to improve the situation in that case.

You can put gevent on top of uWSGI without particular problems (you only need to load the greenlet plugin with --plugins greenlet).

Spawning multiple workers in uWSGI will allows you to have cpu concurrency too (remember to use processes and not threads, as they do not works in
async mode).

It is a fair new setup (in the past uWSGI supported only its personal implementation of green threads) so feel free to ask for help
in the uWSGI list or irc channel

--
Roberto De Ioris
http://unbit.it

[AnilG]

> your application needs to do heavy IO, with a large number of
> concurrently-open sockets, gevent should work much better for you than
> mutliprocessing.
>

> if your program needs to do a lot of CPU-heavy work
> in parallel, you'll probably want to use multiprocessing,

I'm sorry if I'm even more of a noob, but a few recent posts like this
have been bothering me.

I'm planning a massively scaling web app server using gevent and
kqueue.
I guess the primary load is going to be handling the sockets for
multiple http connections.
Hence gevent.

But of course there's processing of the incoming data to generate the
outgoing content.
This doesn't make it 'CPU intensive' all of a suddent, does it?
It's not math processing or anything, but there'll be authentication /
session identification,
pulling together the data and confirming authorisation for the data
requested, etc.

Is this still inside the recommended use-case for gevent?
Thanks for your consideration.
Anil