Somebody who knows the notebook might be able to understand why it is so
slow. The problem is not the machine, because the load is not high and
other notebooks on the same server are much faster. So it is truly an
implementation problem.
Jeroen.
Hi,
To solve this slowness problem will require probably at least a month
of focused work to completely reimplement basically from scratch the
storage and server architecture of the notebook in a much more
scalable way. Nobody is working on this, though if nobody else does
it, then I'll do it sometime in the next year. So the options are:
(1) I simply totally disable www.sagenb.org,
(2) I leave things as they are.
(3) Somebody reimplements the notebook in a way that scales.
Writing scalable web applications is not easy. It's possible, but it
requires much different techniques than are used for the current
notebook, e.g., one must have multiple server instances, and all data
has to be stored in a database that can accessed from any server, etc.
-- William
>
> Jeroen.
>
> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
>
--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org
But IIRC, we aren't sure exactly where the bottlenecks even are right
now. You've mentioned before that the first step is someone developing
some test suites that will hammer a server and profile the server to see
where the bottlenecks are. Has anyone done this?
Thanks,
Jason
We've got a bug days coming up next year (January), and a man-month
between you, me, Alex Leon, and Mike Hansen (provided they're willing
to play along) -- plus whoever else is willing to join.
Tasks:
(1) Write testing code to identify bottlenecks, and generally
improve robustness.
(2) Convert all notebook data structures to a database architecture.
(3) rewrite twist.py to use flask.
(4) Use mod_wsgi and apache to make it scale massively.
(pre-3): evaluate whether we want to change from twisted to something
else. For example, it seems like flask will not support websockets for
a long time, but Alex has some really cool demos/ideas for applications
of websockets in the notebook.
> (4) Use mod_wsgi and apache to make it scale massively.
>
(4b) package nginx or lighttpd or something with Sage so that Sage still
comes with a webserver?
Thanks,
Jason
Maybe keep www.sagenb.org as it is but remove the link from the Sage
front page?
I'm not sure whether there is much I can do, but I'll gladly join if
there is something I can do.
Jeroen.
A half-way solution is to run multiple notebooks servers, partitioning
the users based on their username (perhaps using a different
front-end/proxy). I agree though that the notebook needs to seriously
be fixed, probably by stripping out twisted (maybe replacing with
flaks, maybe not). Fortunately, most of the javascript would probably
be relatively unaffected by such a rewite. I don't think there are any
major technical obstructions, it's mostly a matter of the poeple with
the know-how finding the time.
- Robert
Alex had some really exciting demos, and last I saw, things were
pretty far from being ready for prime time. I'm not sure that these
things have anything to do with eachother, though. My preference is
to keep the single-user functionality more or less the same, and hook
a new robust server structure into place for sagenb.org and other
large multiuser environments.
>
>> (4) Use mod_wsgi and apache to make it scale massively.
>>
>
> (4b) package nginx or lighttpd or something with Sage so that Sage still
> comes with a webserver?
This is a must -- otherwise, we lose the ability to use the notebook
as the GUI. Nobody plans for that to happen.
I've put up a page on the wiki about this project. Any interested
parties are strongly encouraged to get their input down there.
http://wiki.sagemath.org/Notebook_scalability
>
> Thanks,
>
> Jason
> I've put up a page on the wiki about this project. Any interested
> parties are strongly encouraged to get their input down there.
>
> http://wiki.sagemath.org/Notebook_scalability
- Alex
There is also http://demo.sagenb.org and http//demo2.sagenb.org.
William
> Perhaps other people
> are in my condition.
>
> Thanks
>
> Tiziano
>
> On Nov 21, 3:18 am, Jeroen Demeyer <jdeme...@cage.ugent.be> wrote:
>> On 2010-11-20 22:45, William Stein wrote:
>>
>> > (1) I simply totally disablewww.sagenb.org,
>>
>> Maybe keepwww.sagenb.orgas it is but remove the link from the Sage
>> front page?
>
It makes absolutely no sense that that could happen. Please give a
specific example.
-- William
> For more options, visit this group at http://groups.google.com/group/sage-devel
More precisely, please define exactly what you mean by "slower". Do you mean:
(1) it "feels" slower,
(2) the time from press shift-enter until you see output is longer, or
(3) When you use "time a_specific_command(...)", it takes longer.
What I don't think could happen is (3). Obviously (1) or (2) could
happen, due to the client server architecture.
However, this would have absolutely nothing to do with numpy, and you
would notice the same slowdown with
any code in Sage.
William
This has absolutely nothing to do with the notebook. You can't have
this happen if you are actually running the same code. I just tried
this and I get about 5 seconds both in the notebook and on the command
line. *Precisely* how are you running the code on the command line?
You can't just paste it in, since it isn't formatted for that. So
maybe you are saving it to a .py file, then using load or import? If
you do that, then you'll get much, much different times, because of
the Sage preparser, which is not being used on .py files.
Try the following in the notebook:
(1) Make a cell with exactly this content, and evaluate it:
%python
(2) Make another cell with this content and evaluate it:
import time
initial=time.clock()
m=cellular(num2rule(90), 100)
print time.clock()-initial
You'll get a fast timing.
It would be good for you to learn about the Sage preparser more, since
in some cases turning it off (or at least explicitly not using it) can
lead to better performance, especially for numerical code. Turning it
off can also lead to much slower (or wrong) code, but usually in the
context of symbolic (not numerical) computation.
William
> For more options, visit this group at http://groups.google.com/group/sage-devel
No problem. I bet somebody else will find this message in the
archives later or via a search, and it will help them out.
Also, your example is a good handy example to illustrate the
potentially hugely negative speed consequences of the
preparser on numerical code. So thanks!
William
> For more options, visit this group at http://groups.google.com/group/sage-devel
Do they filter port 8000? That is not an uncommon port for HTTP traffic?
http://t2nb.math.washington.edu:8000/
I used to run an SSH server on port 443 when my company filtered port 22!!
If your company does filter 8080. is possible I could make
http://t2nb.math.washington.edu:8000/
also work as
http://t2nb.math.washington.edu:8080/
> Thanks
>
> Tiziano
> On Nov 21, 3:18 am, Jeroen Demeyer<jdeme...@cage.ugent.be> wrote:
> On 2010-11-20 22:45, William Stein wrote:
>
>> (1) I simply totally disablewww.sagenb.org,
>
> Maybe keepwww.sagenb.orgas it is but remove the link from the Sage
> front page?
Dave
That would be good. It's important that things be well thought
through and truly implementable in the time we have -- it's pointless
to try to implement something just to run out of time.
Here's a data point, by the way. Right now there are 42346 accounts
on sagenb.org. If the damn site were made to scale properly, I bet
we could easily hit 100,000 accounts during 2011. The logfile,
which records when people actually do stuff, has about 1 billion
entries since Sept 1. It's almost a 1GB file, but if somebody wants
to study it, email me and let me know, or if you have an account on
boxen.math.washington.edu, look in /sagenb/sagenb/logs/.
Another good thing to do as part of this sprint would be to greatly
improve the logging ability of the notebook server. The logs could,
of course, get stored to a database, for increased flexibility and
ease of querying. They could be studied from a web admin page.
Probably we can copy ideas from Django or something.
>
>> I've put up a page on the wiki about this project. Any interested
>> parties are strongly encouraged to get their input down there.
>>
>> http://wiki.sagemath.org/Notebook_scalability
>
> - Alex
>
> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
>
--
Yes! I would love that. Right now, I have little idea how much the
public KAIST servers are being used -- I can see logins, but I'd like
more information. This would be helpful to me, and also make it easier
to convince administrators / grant funders to drop money on servers.
Dan
--
--- Dan Drake
----- http://mathsci.kaist.ac.kr/~drake
-------
On 2010-11-20 23:33:22, William Stein wrote on the wiki:
>
> We are not going to shoot too high with this project. In particular,
> our goals do *not* include adding new authentication systems or
> making it easy to organize worksheets into folders, etc. We just
> want to solve exactly one problem: make the notebook scalable.
I agree with this sentiment more than I want to see logging.