Magnum

[Arek Bochinski]

This was just posted on google code . It looks like an another python async server worth inspecting . I haven't had the time to browse it's source , but I'm sure there might be a few points of interest and learning opportunities.

http://code.google.com/p/magnum-py/

-Arek

[Douglas Stanley]

Anyone else get a chance to take a peak at this? I skimmed the code,
and it seems to be sort of a multiprocess tornado. Basically it spawns
multiple processes that each use epoll.

It's interesting to say the least. I wonder how it performs compared
to tornado? It's not really an apples to apples comparison though, as
they seem to scratch different itches.

But I was wondering, would it be possible to take the ioloop code from
tornado, and run it in several processes? I have to be honest, I
haven't really peeked at the ioloop guts yet, so if the answer is
obvious, I appologize.

Doug

[Bret Taylor]

We have been discussing doing something similar with forking so that multiple subprocesses can listen on the same socket so that you don't need to run with nginx. I think something like that will happen for the next major Tornado release, in addition to things like gzip encoding and a few other features you would need if you aren't using a reverse proxy like nginx.

Bret

[Douglas Stanley]

Even just being able to have multiple procs so that things can be
deferred in the background
to another proc that would normally block would be cool.

Would something like that require many changes to the core? Or is it
something that can
be done on top of the current tornado code?

Thanks,
Doug

[Andrew Gwozdziewycz]

On Tue, Oct 20, 2009 at 1:14 PM, Bret Taylor <bta...@gmail.com> wrote:

> We have been discussing doing something similar with forking so that
> multiple subprocesses can listen on the same socket so that you don't need
> to run with nginx. I think something like that will happen for the next
> major Tornado release, in addition to things like gzip encoding and a few
> other features you would need if you aren't using a reverse proxy like
> nginx.

I like this idea a lot, and look forward to seeing it. And the good
thing is that
I'm sure it'll be configurable enough that you can turn off features in tornado
if you expect to run behind nginx to take advantage of an nginx module,
maybe rate limiting.

Bret, is there some sort of public document regarding features of upcoming
releases, or a schedule? I'm guessing not, but am curious the same.

Thanks

> On Tue, Oct 20, 2009 at 8:48 AM, Douglas Stanley
> <douglas....@gmail.com> wrote:
>>
>> Anyone else get a chance to take a peak at this? I skimmed the code,
>> and it seems to be sort of a multiprocess tornado. Basically it spawns
>> multiple processes that each use epoll.
>>
>> It's interesting to say the least. I wonder how it performs compared
>> to tornado? It's not really an apples to apples comparison though, as
>> they seem to scratch different itches.
>>
>> But I was wondering, would it be possible to take the ioloop code from
>> tornado, and run it in several processes? I have to be honest, I
>> haven't really peeked at the ioloop guts yet, so if the answer is
>> obvious, I appologize.
>>
>> Doug
>>
>> On Tue, Oct 20, 2009 at 12:42 AM, Arek Bochinski <zeeero...@gmail.com>
>> wrote:
>> > This was just posted on google code . It looks like an another python
>> > async
>> > server worth inspecting . I haven't had the time to browse it's source ,
>> > but
>> > I'm sure there might be a few points of interest and learning
>> > opportunities.
>> >
>> > http://code.google.com/p/magnum-py/
>> > -Arek
>
>

--
http://www.apgwoz.com

[Bret Taylor]

No, but we should do that :)  I just want to make sure it is in a format that won't go stale, but I will make sure we put up some reasonable public roadmap for people to comment on in the relatively near term future.

Bret

[gattis]

Hi, I'm the developer who started magnum-py. I wrote up a blog
article about it here:

http://mattgattis.com/blog/2009/10/18/introducing-magnum/

I mention some of the differences between Magnum and Tornado there...
I actually started coding this before Tornado was released, and was
going to kill it once Tornado touched down and I read about it. I
think we have similar goals in mind but I don't like the fact that I
have to run Tornado behind nginx to get multiple processes (because of
the unnecessary socket connections), and the fact that I can't have a
blocking web app like WSGI and still get epoll.

I briefly looked into forking Tornado but I think it has a
fundamentally different socket handling core architecture, and you
guys are only using half the potential of epoll. Tornado doesn't have
to sit around polling or blocking for clients to connect, but it does
sit around and wait for the web app to finish handling the request.
Magnum just puts the request on a shared memory queue for worker
subprocesses to chew on, goes about accepting more connections, and
then the workers modify the epoll socket saying they are ready to
write whenever they are done. That way everything is completely event-
driven, and your performance degrades more gracefully when your app
becomes the bottleneck. Plus, Magnum is using edge-triggered epoll
which slightly outperforms the level-trigger epoll that Tornado is
using.

I've done benchmarks comparing Magnum to apache for a simple dynamic
python app, and it beats the hell out of apache much like Tornado.
I'll probably run benchmarks against Tornado w/ a few nginx reverse
proxies and post the results soon, but I can't really see where
Tornado would have the edge.


On Oct 20, 11:48 am, Douglas Stanley <douglas.m.stan...@gmail.com>
wrote:

[Christian Kreutzer]

nice work! how does it compare to spawning (http://pypi.python.org/
pypi/Spawning/)?

[Bret Taylor]

Looks interesting, I will check it out. Great name, by the way!