[sage-devel] Is there any particular reason why Notebook uses Twisted.Web2 instead of Twisted.Web?

[Tim Dumol]

The Notebook currently uses Twisted.Web2 as its server. As stated <a
href="http://twistedmatrix.com/trac/wiki/TwistedWebPlan">here</a>,
Twisted.Web2 is being phased out and its useful features being merged
back into Twisted.Web. There doesn't seem to be any support for
Twisted.Web2 -- I cannot find proper documentation for Twisted.Web2,
aside from http://twistedmatrix.com/trac/browser/trunk/doc/web2. API
listings for Twisted.Web2 are gone in the latest version of API
documentation (http://twistedmatrix.com/documents/8.2.0/api/
twisted.html).

Would it cause any problems if the Notebook were gradually migrated to
Twisted.Web?

[William Stein]

When we wrote the second version of the notebook, twisted.web2 was *the* way to go, according to various people.  However, it turned out that using twisted.web2 was not the way to go, as the project was discontinued as an independent project. 

I'm not at all convinced that using twisted in any way (web or web2) is a good idea for the Sage notebook.   I plan to revisit this in late September.    The first thing I plan to do is consider switching from twisted to Django, as is done in codenode -- see http://codenode.org/ -- hopefully, even sharing code with that project.   Of course, twisted could still get used at a certain level behind the scenes, but the Sage notebook would then no longer explicitly use it.

So if you want to help as you describe above, perhaps you could accelerate this.  This involves:

  (1) getting familiar with Django, if you don't already know it.

  (2) reading through the current codenode codebase

  (3) then formulating a plan to replace server/notebook/twist.py with something based either directly on Django, or possibly using codenode in some way.

The above is what I would do, but if you do it first that would be spectacular.

William

[Thierry Dumont]

William Stein a écrit :

>
> The first thing I plan to do is consider switching from
> twisted to Django, as is done in codenode -- see http://codenode.org/ --
> hopefully, even sharing code with that project.

Django is used (generally) with mod_python under Apache. Does it means
that the whole Sage will be served by Apache and mod_python? This would
(may be) improve performances, no ?

t.d.

[William Stein]

I doubt it would impact performance in practice.  Improving performance is going to require improvement to the rest of the server code.

For Sage, the default would be that the notebook is still served by some Python library -- most likely Twisted of course -- but that at least the option of replacing use of Twisted by Apache would exist. 

Codenode guys -- since I speak from 0 experience (!) -- any comments on the relative performance of Apache versus Twisted?

William
 

t.d.

--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

[Dr. David Kirkby]

William Stein wrote:

> When we wrote the second version of the notebook, twisted.web2 was *the*
> way to go, according to various people. However, it turned out that
> using twisted.web2 was not the way to go, as the project was
> discontinued as an independent project.

Perhaps a silly questions, but is Apache the way to go?

Dave

[Robert Bradshaw]

Whatever we choose, it should still work well as a stand-alone server
(unless we wanted to ship apache with Sage :) but it would be nice,
especially for large public servers, to use a platform that supported
some mod_* under apache as well.

- Robert

[Jason Grout]

If we use Django, I don't think we need to restrict ourselves to Apache:

http://code.djangoproject.com/wiki/ServerArrangements

Jason

--
Jason Grout

[Tim Dumol]

Codenode doesn't work with Sage currently, although Dorian Raymer is
working on the backend, and it seems it should be ready within a week.

It seems that Codenode can replace the Notebook eventually, assuming
there are ways to plug in a few templates (to include the Help, etc.).
It's well-modularized, so if there isn't a plug-in framework yet, it
should be possible (Django supports using middleware to preprocess
stuff, afaik).

In the meantime, we can stick with the Notebook and make changes to
make a transition to Codenode as smooth as possible. This probably
entails decoupling the components of the Notebook, and finding a way
to generate the documentation (live) that can be plugged into Codenode
easily.

How does that sound?

On Sep 1, 12:17 am, Robert Bradshaw <rober...@math.washington.edu>
wrote:

As far as I can tell, Codenode doesn't support Apache yet either
( http://groups.google.com/group/codenode-devel/browse_thread/thread/176925e2b2e42279?hl=en
). Regarding the performance of Codenode, it seems to respond a bit
slower (at least for Python) right now, although I haven't actually
done any real measurements. I'm guessing it can be attributed to the
communication time between the server and engine.

[Dr. David Kirkby]

I would expect apache to come with any relatively recent Unix or
unix-like box. Perhaps it's not installed by default though. (It is on
Solaris).

I just checked and see the latest Apache is 6.6 MB, so not small, but
not particularly huge.

Apache really is a standard. I don't see it has any serious competition.

[William Stein]

There is no way that the Sage notebook will ever *depend* on Apache.  That doesn't even make sense to suggest.

William

[William Stein]

On Mon, Aug 31, 2009 at 9:40 AM, Tim Dumol <timd...@gmail.com> wrote:

Codenode doesn't work with Sage currently, although Dorian Raymer is
working on the backend, and it seems it should be ready within a week.

It seems that Codenode can replace the Notebook eventually, assuming
there are ways to plug in a few templates (to include the Help, etc.).
It's well-modularized, so if there isn't a plug-in framework yet, it
should be possible (Django supports using middleware to preprocess
stuff, afaik).

In the meantime, we can stick with the Notebook and make changes to
make a transition to Codenode as smooth as possible. This probably
entails decoupling the components of the Notebook, and finding a way
to generate the documentation (live) that can be plugged into Codenode
easily.

How does that sound?

I definitely do not endorse this, at least until I know a lot more about Codenode and other options.  A month from now I will.   Regarding speed, there are a lot of parameters to consider, but in addition to "snappiness" for a single user, it's also important to consider how the notebook handles say N simultaneous users for various values of N.   It's also important that speed testing be do-able using precise tools rather than anecdote.

William
 

On Sep 1, 12:17 am, Robert Bradshaw <rober...@math.washington.edu>
wrote:

> On Aug 31, 2009, at 9:07 AM, Dr. David Kirkby wrote:
>
> > William Stein wrote:
>
> >> When we wrote the second version of the notebook, twisted.web2 was  
> >> *the*
> >> way to go, according to various people.  However, it turned out that
> >> using twisted.web2 was not the way to go, as the project was
> >> discontinued as an independent project.
>
> > Perhaps a silly questions, but is Apache the way to go?
>
> Whatever we choose, it should still work well as a stand-alone server  
> (unless we wanted to ship apache with Sage :) but it would be nice,  
> especially for large public servers, to use a platform that supported  
> some mod_* under apache as well.
>
> - Robert

As far as I can tell, Codenode doesn't support Apache yet either
( http://groups.google.com/group/codenode-devel/browse_thread/thread/176925e2b2e42279?hl=en
). Regarding the performance of Codenode, it seems to respond a bit
slower (at least for Python) right now, although I haven't actually
done any real measurements. I'm guessing it can be attributed to the
communication time between the server and engine.

 

[Brian Granger]

I would expect apache to come with any relatively recent Unix or
unix-like box. Perhaps it's not installed by default though. (It is on
Solaris).

I just checked and see the latest Apache is 6.6 MB, so not small, but
not particularly huge.

Apache really is a standard. I don't see it has any serious competition.

For deploying the type of web application that the Sage notebook is, there are definitely alternatives to apache.  As William has mentioned, there are many different aspect of the Sage notebook performance and scalability.  Then there is deployability - where apache is not that great even on linux (proper apache config is non-trivial).

I think the best solution is to abstract the Sage notebook web application away from the web server using something like WSGI.  Ideally, the Sage notebook could be deployed using any of a number of different web servers depending on a users needs.  Welding the notebook to a particular web server implementation is a second rate choice in many respects.

 

There is no way that the Sage notebook will ever *depend* on Apache.  That doesn't even make sense to suggest.

Whew!

Cheers,

Brian

 

William

[Alex Clemesha]

On Mon, Aug 31, 2009 at 8:59 AM, William Stein<wst...@gmail.com> wrote:
>
>
> On Mon, Aug 31, 2009 at 8:45 AM, Thierry Dumont <tdu...@math.univ-lyon1.fr>
> wrote:
>>
>> William Stein a écrit :
>> >
>> >   The first thing I plan to do is consider switching from
>> > twisted to Django, as is done in codenode -- see http://codenode.org/ --
>> > hopefully, even sharing code with that project.
>>
>> Django is used (generally) with mod_python under Apache. Does it means
>> that the whole Sage will be served by Apache and mod_python? This would
>> (may be) improve performances, no ?
>>
>
> I doubt it would impact performance in practice.  Improving performance is
> going to require improvement to the rest of the server code.
>
> For Sage, the default would be that the notebook is still served by some
> Python library -- most likely Twisted of course -- but that at least the
> option of replacing use of Twisted by Apache would exist.
>
> Codenode guys -- since I speak from 0 experience (!) -- any comments on the
> relative performance of Apache versus Twisted?

For what we are trying to do, i.e. write an "online programming
notebook" (Sage notebook / Codenode),
the bottleneck is definitely going to be the processes that are
actually executing
the code, the communications from these processes back to the "Frontend",
and the additional data operations like saving user state.

Consider a "very small" load on a regular website, maybe 100 users at once doing
various things like reading static articles or checking out photos. In
the case of
the Sage Notebook or Codenode, that's *100 python/sage processes* -
Apache vs Twisted doesn't make much of a difference here.

A Twisted web server can easily handle 100 users, and so can Apache, it's
a moot point for what we are trying to accomplish, imho. There are other
optimizations that are far more important - like optimizing file
system / database calls.

To summarize, we have many more pertinent problems to solve than trying
to decided how a web server might improve performance.

Also, as William has mentioned, talking about all this in an anecdotal
way is mostly useless. We need well thought out benchmarks - which
is a challenging science in its self (there are lies, damn lies, and
benchmarks).


--

Also, Django is a web framework, meant to be served by a WSGI server.

With codenode, the default is to use Twisted, which *has* WSGI serving
functionality,
and it's very convenient to use Twisted because for most situations users
don't have to know anything about the fact that Twisted is doing the WSGI.
Surely this is desirable for the Sage notebook, as with codenode.

You can go to the extra effort to use Apache/Nginx to do the WSGI, but at
this moment with codenode it hasn't been needed for our usage because
Twisted works great. That said, as soon as possible, I want to test and
document all configurations so people can choose what they prefer.


-Alex

>
> William
>
>>
>> t.d.
>>
>>
>>
>
>
>
> --
> William Stein
> Associate Professor of Mathematics
> University of Washington
> http://wstein.org
>
> >
>

--
Alex Clemesha
clemesha.org

[Jason Grout]

Dr. David Kirkby wrote:

>
> Apache really is a standard. I don't see it has any serious competition.

It seems like lighttpd (http://www.lighttpd.net/) is preferable to
apache in some instances.

According to wikipedia: "Lighttpd is used by some of the biggest
websites, including sites such as meebo. Wikimedia runs Lighttpd servers
[4][5][6][7] as does SourceForge.[7] Three of the most famous torrent
listing websites, The Pirate Bay, Mininova and isoHunt, which have more
than 1,000 hits per second, also use Lighttpd.[8] Lighttpd currently
holds fifth place on the Netcraft "Web Server Survey" (November
2008).[9] Lighttpd has not placed in the June 2009 survey except under
the "Market Share for Top Servers Across the Million Busiest Sites"
survey where it places sixth. [10]"

I agree that apache certainly seems to be the most common webserver.

Jason

[William Stein]

On Mon, Aug 31, 2009 at 12:39 PM, Brian Granger <ellisonbg.net@gmail.com> wrote:

I would expect apache to come with any relatively recent Unix or
unix-like box. Perhaps it's not installed by default though. (It is on
Solaris).

I just checked and see the latest Apache is 6.6 MB, so not small, but
not particularly huge.

Apache really is a standard. I don't see it has any serious competition.

For deploying the type of web application that the Sage notebook is, there are definitely alternatives to apache.  As William has mentioned, there are many different aspect of the Sage notebook performance and scalability.  Then there is deployability - where apache is not that great even on linux (proper apache config is non-trivial).

I think the best solution is to abstract the Sage notebook web application away from the web server using something like WSGI.  Ideally, the Sage notebook could be deployed using any of a number of different web servers depending on a users needs.  Welding the notebook to a particular web server implementation is a second rate choice in many respects.

I very strongly agree with this.     My impression is that codenode has sort of done this, in that they use Django, which abstracts away and uses WSGI.

William
 

 

There is no way that the Sage notebook will ever *depend* on Apache.  That doesn't even make sense to suggest.

Whew!

Cheers,

Brian

 

William

[Robert Bradshaw]

Though we're all just pulling numbers out of the air until we have
actual benchmark data, I agree with you that performance probably
isn't a big issue here. I see the advantages o Apache being that
authentication, ssl, etc. are abstracted away and often already
integrated into the existing environment which is one less thing to
worry about.

That being said, I'm a strong proponent of the benefits of WSGI.

- Robert

[Tim Dumol]

Reading through the code base, it seems switching to Django will
entail replacing the user authentication system with Django's -- or at
least it seems the cleanest way to do it. Django views can be
relatively easily implemented by just translating the code in
`twist.py`. The authentication seems to pose the largest obstacle.

I've made a new ticket regarding this: http://trac.sagemath.org/sage_trac/ticket/6897

On Aug 31, 11:20 pm, William Stein <wst...@gmail.com> wrote:
> <SNIP>

[William Stein]

On Sat, Sep 5, 2009 at 8:22 PM, Tim Dumol<timd...@gmail.com> wrote:
>
> Reading through the code base, it seems switching to Django will
> entail replacing the user authentication system with Django's -- or at
> least it seems the cleanest way to do it. Django views can be
> relatively easily implemented by just translating the code in
> `twist.py`. The authentication seems to pose the largest obstacle.
>
> I've made a new ticket regarding this: http://trac.sagemath.org/sage_trac/ticket/6897

Excellent. Thanks for doing this. What you say above makes perfect
sense to me (that is, I agree).

William

>
> On Aug 31, 11:20 pm, William Stein <wst...@gmail.com> wrote:
>> <SNIP>
>>
>> I'm not at all convinced that using twisted in any way (web or web2) is a
>> good idea for the Sage notebook.   I plan to revisit this in late
>> September.    The first thing I plan to do is consider switching from
>> twisted to Django, as is done in codenode -- seehttp://codenode.org/--
>> hopefully, even sharing code with that project.   Of course, twisted could
>> still get used at a certain level behind the scenes, but the Sage notebook
>> would then no longer explicitly use it.
>>
>> So if you want to help as you describe above, perhaps you could accelerate
>> this.  This involves:
>>
>>   (1) getting familiar with Django, if you don't already know it.
>>
>>   (2) reading through the current codenode codebase
>>
>>   (3) then formulating a plan to replace server/notebook/twist.py with
>> something based either directly on Django, or possibly using codenode in
>> some way.
>>
>> The above is what I would do, but if you do it first that would be
>> spectacular.
>>
>> William
> >
>