notebook rewrite

13 views
Skip to first unread message

Ondrej Certik

unread,
Jul 21, 2009, 12:02:34 AM7/21/09
to sage-...@googlegroups.com, codenod...@googlegroups.com
Hi,

I finally learned javascript and AJAX, so that I can help with the
notebook. I also studied it's sources.

First things I like:

* I like the user interface, it's usable, especially the attention to
little details, like borders around the cells, tab completion, tab
indentation and things like that.

Things I don't like:

* the javascript is really hackish overall, but two things really
caught my attention:
a) the keyboard handling is horrific, why not to use some standard
library for that, that works across all browsers
b) it uses some custom format for transfering data (which has bugs,
like http://groups.google.com/group/sage-devel/browse_thread/thread/5ecd104b0aa85439),
why not to use JSON?
* it doesn't run on the google appengine (William mentioned in the
past, that he doesn't see any benefit to do that, or that it would be
slow)

Well, talk is cheap, so here is the code (a sample Firefox screenshot
is also attached in case it didn't work in your browser):

http://pythonnb.appspot.com/

it uses jQuery all over, it uses a keyboard plugin for jQuery, it uses
JSON and it runs on the google appengine (and anywhere else too, it's
just a standard django app). I tested in Firefox and IE8. The keyboard
works, there are just some subtle bugs on IE8, see here:

http://github.com/certik/notebook/blob/375a2026ee7ea721904d05068724b3a7663d018e/todo

but none of it seems major to me, the keyboard seems to be working
just fine (or is IE8 not the most problematic? I'll try to test in
other browsers like Opera and Safari too). Here is the index.html with
all the javascript that I wrote:

http://github.com/certik/notebook/blob/375a2026ee7ea721904d05068724b3a7663d018e/templates/index.html

It handles most of the keyboard interaction. It doesn't have TAB
completion and inspection yet.

Well, let me say that I really like to run things on the appengine,
rather than to constantly maintain our own servers. I see no reason
why the notebook cannot run on the appengine, only the AJAX would talk
to our own server with Sage to actually evaluate the cells (and for
many people, I think appengine itself could actually be enough). I
have to think though what the best way to transfer data to the
database with worksheets is though.

I wanted to ask --- which parts of the Sage notebook are BSD licensed?
I used a bit of the CSS styles and and maybe one javascript function,
everything else was written by me. If possible, I'd like to use the
BSD license for the notebook (if I find time to work on it further),
so that ipython can use it by default.

Also, question to all, do you like the In [3] and Out[3] lines? I
don't have an opinion on it yet myself, so I implemented them, to see
how it looks like. Also, please let me know if it works in your
browser.

Ondrej

notebook.png

William Stein

unread,
Jul 21, 2009, 12:21:09 AM7/21/09
to codenod...@googlegroups.com, sage-...@googlegroups.com
On Mon, Jul 20, 2009 at 9:02 PM, Ondrej Certik<ond...@certik.cz> wrote:
> Hi,
>
> I finally learned javascript and AJAX, so that I can help with the
> notebook. I also studied it's sources.
>
> First things I like:
>
> * I like the user interface, it's usable, especially the attention to
> little details, like borders around the cells, tab completion, tab
> indentation and things like that.
>
> Things I don't like:
>
> * the javascript is really hackish overall, but two things really
> caught my attention:
>  a) the keyboard handling is horrific, why not to use some standard
> library for that, that works across all browsers

Tom Boothby wrote all that in early 2006, and there wasn't something
good then. I don't think jquery even existed then.

>  b) it uses some custom format for transfering data (which has bugs,
> like http://groups.google.com/group/sage-devel/browse_thread/thread/5ecd104b0aa85439),
> why not to use JSON?

That would be a good idea.

> * it doesn't run on the google appengine (William mentioned in the
> past, that he doesn't see any benefit to do that, or that it would be
> slow)

Just because I don't see a benefit to something, doesn't mean there
aren't tons of benefits.

> Well, talk is cheap, so here is the code (a sample Firefox screenshot
> is also attached in case it didn't work in your browser):
>
> http://pythonnb.appspot.com/
>
> it uses jQuery all over,

Cool!

> it uses a keyboard plugin for jQuery, it uses
> JSON and it runs on the google appengine (and anywhere else too, it's
> just a standard django app). I tested in Firefox and IE8. The keyboard
> works, there are just some subtle bugs on IE8, see here:
>
> http://github.com/certik/notebook/blob/375a2026ee7ea721904d05068724b3a7663d018e/todo
>
> but none of it seems major to me, the keyboard seems to be working
> just fine (or is IE8 not the most problematic? I'll try to test in
> other browsers like Opera and Safari too). Here is the index.html with
> all the javascript that I wrote:
>
> http://github.com/certik/notebook/blob/375a2026ee7ea721904d05068724b3a7663d018e/templates/index.html
>
> It handles most of the keyboard interaction. It doesn't have TAB
> completion and inspection yet.

How are you doing the auto input cell resizing?

> Well, let me say that I really like to run things on the appengine,
> rather than to constantly maintain our own servers. I see no reason
> why the notebook cannot run on the appengine, only the AJAX would talk
> to our own server with Sage to actually evaluate the cells (and for
> many people, I think appengine itself could actually be enough). I
> have to think though what the best way to transfer data to the
> database with worksheets is though.
>
> I wanted to ask --- which parts of the Sage notebook are BSD licensed?
> I used a bit of the CSS styles and and maybe one javascript function,
> everything else was written by me. If possible, I'd like to use the
> BSD license for the notebook (if I find time to work on it further),
> so that ipython can use it by default.

Make precise what you used and we'll get it BSD licensed for you. We
have to see who wrote the particular code you're using.

> Also, question to all, do you like the In [3] and Out[3] lines?
> I don't have an opinion on it yet myself, so I implemented them, to see
> how it looks like. Also, please let me know if it works in your
> browser.
>
> Ondrej

-- William

Alex Clemesha

unread,
Jul 21, 2009, 12:30:51 AM7/21/09
to codenod...@googlegroups.com, sage-...@googlegroups.com
Hi Ondrej,

I'll reply from a purely codenode point of view. You sent this
email to both lists, but I'm only qualified to describe the details
of codenode's current architecture.


>  a) the keyboard handling is horrific, why not to use some standard
> library for that, that works across all browsers

There is an *excellent* jQuery library for this called "js-hotkeys"
http://code.google.com/p/js-hotkeys, which is surely the one you are mentioning
that just did not exist when both notebooks began to really get going.
That said, it would be extremely beneficial to delegate the key-handling
to that library.


>  b) it uses some custom format for transfering data (which has bugs,
> like http://groups.google.com/group/sage-devel/browse_thread/thread/5ecd104b0aa85439),
> why not to use JSON?

codenode only sends data encoded in JSON. This is very important because
it totally decouples data from presentation. This is in fact one reason why the
switch to Django went very smoothly.


> * it doesn't run on the google appengine (William mentioned in the
> past, that he doesn't see any benefit to do that, or that it would be
> slow)

The codenode backend (as you know) does run on app-engine, and
I feel that this is the most important part because this is where all the
arbitrary code execution (the big security risk) happens. codenode
is now mostly Django so it does seem feasible to make everything work on
app-engine, but this would take a little work.


> I wanted to ask --- which parts of the Sage notebook are BSD licensed?
> I used a bit of the CSS styles and and maybe one javascript function,
> everything else was written by me. If possible, I'd like to use the
> BSD license for the notebook (if I find time to work on it further),
> so that ipython can use it by default.

We are actually going to be completely switching the codenode license to BSD,
(as nothing we depend on is GPL) and we hope to allow more people
to utilize what codenode has to offer.

Dorian and I have talked about this, and we feel that it is best. The
scipy/numpy/sympy/matplotlib
communities are ones that we know can benefit from a really good notebook,
and we hope that all our efforts combined can make it so.

We have not made the official switch yet, but we will be officially switching
to the BSD license in the next couple weeks.


thanks,
Alex

John H Palmieri

unread,
Jul 21, 2009, 12:31:16 AM7/21/09
to sage-devel
On Jul 20, 9:02 pm, Ondrej Certik <ond...@certik.cz> wrote:

[snip]

> Also, question to all, do you like the In [3] and Out[3] lines? I
> don't have an opinion on it yet myself, so I implemented them, to see
> how it looks like.

How easy would it be to add a way to toggle them on and off?

> Also, please let me know if it works in your
> browser.

In a brief test, it works in Safari and Firefox on my intel mac.

John

killian koepsell

unread,
Jul 21, 2009, 2:18:26 AM7/21/09
to sage-...@googlegroups.com
Hi Ondrej,

very nice work!

On Mon, Jul 20, 2009 at 9:02 PM, Ondrej Certik<ond...@certik.cz> wrote:

>  a) the keyboard handling is horrific, why not to use some standard
> library for that, that works across all browsers
>  b) it uses some custom format for transfering data (which has bugs,
> like http://groups.google.com/group/sage-devel/browse_thread/thread/5ecd104b0aa85439),
> why not to use JSON?

another option of course would be to use pyjamas:
http://code.google.com/p/pyjamas/
It has a lot of features and also the option to run it standalone,
without a browser, as a
desktop app.

Kilian

Ondrej Certik

unread,
Jul 21, 2009, 2:44:25 AM7/21/09
to codenod...@googlegroups.com, sage-...@googlegroups.com

I take the text, count number of "\n", handle line wrapping, calculate
the number of lines *occupied* in the textbox and set the number of
rows of the textbox. It just works in firefox, there is a little
glitch in IE8, that I have to put the backspace and enter into the
text before the calculation (e.g. the text is updated after the
keyboard handler). But I don't need to put the text to some div first,
measure it's height and set the height.

Seems like a similar glitch is in Opera.

As to which functions I used, I used this one:

function get_selection_range(input) {
/*
Return the start and end positions of the currently selected text
in the input text area (a DOM object).

INPUT:
input -- a DOM object (a textarea)

OUTPUT:
an array of two nonnegative integers
*/
// If the attribute input.selectionStart is present, use that:
if (input.selectionStart || input.selectionStart == 0) {
return Array(input.selectionStart, input.selectionEnd);
} else {
var start, end;
var range = document.selection.createRange();

var tmprange = range.duplicate();
tmprange.moveToElementText(input);
tmprange.setEndPoint("endToStart", range);
start = tmprange.text.length;

tmprange = range.duplicate();
tmprange.moveToElementText(input);
tmprange.setEndPoint("endToEnd", range);
end = tmprange.text.length;

return Array(start, end);
}
}

(I rewrote it it a bit, and I may have broken it on IE8, but I'll fix it. :)

Besides that, I used the following styles (again, I modifed them a
bit, but left the Sage notebook borders, because I like them). I fixed
the padding, so that (at least on firefox) if you focus a cell, only
the border changes, but the text doesn't move (in Sage notebook, the
text moves by 1 pixel, and I find it annoying).

textarea.cell_input {
color:#000000;
background-color: white;
border: 1px solid #a8a8a8;
font-family: monospace;
font-size:12pt;
overflow:hidden;
padding-left:6px;
padding-top:4px;
padding-bottom:4px;
margin-bottom:0px;
margin-top:0px;
line-height:1.2em;
float: left;
}

textarea.cell_input_active {
background-color: white;
border: 2px solid #8888FE;
font-family: monospace;
font-size:12pt;
overflow:hidden;
padding-left:5px;
padding-top:3px;
padding-bottom:4px;
margin-bottom:0px;
margin-top:0px;
line-height:1.2em;
float: left;
}

Besides that, I wrote everything from scratch.

Ondrej

Ondrej

Ondrej Certik

unread,
Jul 21, 2009, 2:58:14 AM7/21/09
to codenod...@googlegroups.com, sage-...@googlegroups.com
On Mon, Jul 20, 2009 at 10:30 PM, Alex Clemesha<clem...@gmail.com> wrote:
>
> Hi Ondrej,
>
> I'll reply from a purely codenode point of view.  You sent this
> email to both lists, but I'm only qualified to describe the details
> of codenode's current architecture.

Yes. In fact, one reason I wrote it is so that you can use it in
codenode if you like it --- I really like the Sage style with borders
around cells etc.

>
>
>>  a) the keyboard handling is horrific, why not to use some standard
>> library for that, that works across all browsers
> There is an *excellent* jQuery library for this called "js-hotkeys"
> http://code.google.com/p/js-hotkeys, which is surely the one you are mentioning
> that just did not exist when both notebooks began to really get going.
> That said, it would be extremely beneficial to delegate the key-handling
> to that library.

Yes, that's exactly what I use. It seems to be working just fine
everywhere and the interface is really nice and super easy, you just
attach a function for every key combination --- no need to have one
ugly handler for everything.

>
>
>>  b) it uses some custom format for transfering data (which has bugs,
>> like http://groups.google.com/group/sage-devel/browse_thread/thread/5ecd104b0aa85439),
>> why not to use JSON?
> codenode only sends data encoded in JSON.  This is very important because
> it totally decouples data from presentation.  This is in fact one reason why the
> switch to Django went very smoothly.

Yes, that's the way to go.

>
>
>
>
>> * it doesn't run on the google appengine (William mentioned in the
>> past, that he doesn't see any benefit to do that, or that it would be
>> slow)
> The codenode backend (as you know) does run on app-engine, and
> I feel that this is the most important part because this is where all the
> arbitrary code execution (the big security risk) happens.  codenode
> is now mostly Django so it does seem feasible to make everything work on
> app-engine, but this would take a little work.

In fact, the backend can only run on the appengine if it's pure python
(like sympy), but not if it's some heavy C++ stuff, like our FEM
solvers. But the frontend can run in there, that's my idea.

>
>
>
>
>> I wanted to ask --- which parts of the Sage notebook are BSD licensed?
>> I used a bit of the CSS styles and and maybe one javascript function,
>> everything else was written by me. If possible, I'd like to use the
>> BSD license for the notebook (if I find time to work on it further),
>> so that ipython can use it by default.
>
> We are actually going to be completely switching the codenode license to BSD,
> (as nothing we depend on is GPL) and we hope to allow more people
> to utilize what codenode has to offer.
>
> Dorian and I have talked about this, and we feel that it is best. The
> scipy/numpy/sympy/matplotlib
> communities are ones that we know can benefit from a really good notebook,
> and we hope that all our efforts combined can make it so.
>
> We have not made the official switch yet, but we will be officially switching
> to the BSD license in the next couple weeks.


Ah, that is very nice! Indeed, there should be some default notebook
for python stuff, I view it like a part of the common platform, that
everyone needs.

How hard would be to (maybe optionally) use the Sage like look & feel
to codenode?

Ondrej

Ondrej Certik

unread,
Jul 21, 2009, 3:02:17 AM7/21/09
to sage-...@googlegroups.com
On Mon, Jul 20, 2009 at 10:31 PM, John H Palmieri<jhpalm...@gmail.com> wrote:
>
> On Jul 20, 9:02 pm, Ondrej Certik <ond...@certik.cz> wrote:
>
> [snip]
>
>> Also, question to all, do you like the In [3] and Out[3] lines? I
>> don't have an opinion on it yet myself, so I implemented them, to see
>> how it looks like.
>
> How easy would it be to add a way to toggle them on and off?

That should be easy. It's written in a way, so that people can just
trivially modify the html template and very easily change things like
this.

>
>> Also, please let me know if it works in your
>> browser.
>
> In a brief test, it works in Safari and Firefox on my intel mac.

Thanks!

On Tue, Jul 21, 2009 at 12:18 AM, killian koepsell<koep...@gmail.com> wrote:
>
> Hi Ondrej,
>
> very nice work!
>

> On Mon, Jul 20, 2009 at 9:02 PM, Ondrej Certik<ond...@certik.cz> wrote:

>> a) the keyboard handling is horrific, why not to use some standard
>> library for that, that works across all browsers
>> b) it uses some custom format for transfering data (which has bugs,
>> like http://groups.google.com/group/sage-devel/browse_thread/thread/5ecd104b0aa85439),
>> why not to use JSON?
>

> another option of course would be to use pyjamas:
> http://code.google.com/p/pyjamas/
> It has a lot of features and also the option to run it standalone,
> without a browser, as a
> desktop app.

Yes, that's the next thing that I want to learn and try to rewrite the
thing that I wrote so far into pyjamas, so that we can have everything
in Python.

I wanted to get my hands dirty first, to learn how javascript works,
because even though theoretically you don't have to touch it with
pyjamas, but in practise I am sure I will need to debug it why it
doesn't work in some particular browser.

Ondrej

Ondrej Certik

unread,
Jul 21, 2009, 3:14:30 AM7/21/09
to codenod...@googlegroups.com, sage-...@googlegroups.com
>> Things I don't like:
>>
>> * the javascript is really hackish overall, but two things really
>> caught my attention:
>
> I'm a self-taught javascript hacker -- I learned how to hack javascript long before browsers started adhering to standards, and it shows.  If William writes hackish javascript, it's probably because he was going by my example.
>
> Also -- I've been supporting Opera 8.  jQuery doesn't work there, because it's buggy and just behaves strangely -- many of the hacks you see can be blamed on Opera 8.  Now that Opera 9 is out and jQuery supports it, I am fully behind a complete transition to jQuery.


You did an amazing work. I didn't realize it was already in 2005. It
must have been terrible to make sure it works everywhere.

Fortunately, today there seems to be good javascript libraries for
everything and they seem to work pretty well almost everywhere.

>
>>  a) the keyboard handling is horrific, why not to use some standard
>> library for that, that works across all browsers
>

> As noted, I did this before jQuery existed -- I searched hard and long before deciding to write my own keyboard handler, and every "clean" approach I took failed in a browser or two -- the "horrific" result works in every platform I've tried, so long as one stays away from the alt key in Safari, IIRC.
>
> Now that Opera is no longer an obstruction, there's only one reason not to use a standard library: it's been written, and it works.  Rewrite it!  I love seeing my javascript get rewritten!

I am thinking of using pyjamas, if it works for this, *that* be
awesome. Having everything in Python.

>
>>  b) it uses some custom format for transfering data (which has bugs,
>> like http://groups.google.com/group/sage-devel/browse_thread/thread/5ecd104b0aa85439),
>> why not to use JSON?
>

> Again... it worked after we wrote it.  It became too much work to replace, so we kept cobbling more on.

Right. By no means it was meant to criticize your work. :) I was just
saying that we can do better today with all those nice js libraries.

>
>> * it doesn't run on the google appengine (William mentioned in the
>> past, that he doesn't see any benefit to do that, or that it would be
>> slow)
>>
>> Well, talk is cheap, so here is the code (a sample Firefox screenshot
>> is also attached in case it didn't work in your browser):
>>
>> http://pythonnb.appspot.com/
>

> Might have to take back what I said earlier...  Shift-enter causes an extra newline to be placed in the cell below the current one in Opera 9.

This newline is a bick hackish still ---- basically the textarea
really sucks, it doesn't have a function for getting a cursor position
and it cannot resize automatically. Everything has to be written
indirectly.

>
>>
>> it uses jQuery all over, it uses a keyboard plugin for jQuery, it uses
>> JSON and it runs on the google appengine (and anywhere else too, it's
>> just a standard django app). I tested in Firefox and IE8. The keyboard
>> works, there are just some subtle bugs on IE8, see here:
>>
>> http://github.com/certik/notebook/blob/375a2026ee7ea721904d05068724b3a7663d018e/todo
>>
>> but none of it seems major to me, the keyboard seems to be working
>> just fine (or is IE8 not the most problematic? I'll try to test in
>> other browsers like Opera and Safari too). Here is the index.html with
>> all the javascript that I wrote:
>>
>> http://github.com/certik/notebook/blob/375a2026ee7ea721904d05068724b3a7663d018e/templates/index.html
>>
>> It handles most of the keyboard interaction. It doesn't have TAB
>> completion and inspection yet.
>

> Initial reaction: NICE!!!  But... I only see about 20% of the functionality we really need, and the last 10% typically takes as long as the first 90%.

That's right.

>
> Criticism: when one presses the up arrow accidentally at the top of a cell, it is obnoxious for the cursor to jump to the top of the next cell up.

Yes, in fact this is the first thing in my the TODO file:

http://github.com/certik/notebook/blob/375a2026ee7ea721904d05068724b3a7663d018e/todo

>
> Suggestion: the introspection interface, as written, is utter shit.  It's literally the first thing that I got to work, and it's never been reworked.  I've been wanting to move the introspect "window" to a floating div that can be torn out of the window -- but I have little skill when it comes to using the new-fangled javascript libraries, so I haven't done this.  At the very least, I think it should appear on the right-hand side of the window so one can both read the documentation, and the text at the top of their long cell.


Yes, I think this is the second thing that I want to write. Maybe
after pyjamas --- now, when I understand how to debug all those AJAX
requests, I am eager to look into that.

I really tried to avoid javascript and all this AJAX thing, but I must
say it's really exciting! :)

I think I will try to write GUI for our FEM stuff in the browser.
Browser is the best thing.

>
>>
>> Well, let me say that I really like to run things on the appengine,
>> rather than to constantly maintain our own servers. I see no reason
>> why the notebook cannot run on the appengine, only the AJAX would talk
>> to our own server with Sage to actually evaluate the cells (and for
>> many people, I think appengine itself could actually be enough). I
>> have to think though what the best way to transfer data to the
>> database with worksheets is though.
>>
>> I wanted to ask --- which parts of the Sage notebook are BSD licensed?
>> I used a bit of the CSS styles and and maybe one javascript function,
>> everything else was written by me. If possible, I'd like to use the
>> BSD license for the notebook (if I find time to work on it further),
>> so that ipython can use it by default.
>

> Every single line I have written for the notebook is BSD licensed.  However, William, Alex Clemesha, Jason Grout, and Robert Bradshaw have all contributed javascript code, so I'd like to hear from them from making a blanket statement about the file.  I believe that Dorian Raymer and Mike Hansen may have contributed, too.  Am I missing anybody?  Robert Miller?


>
>> Also, question to all, do you like the In [3] and Out[3] lines? I
>> don't have an opinion on it yet myself, so I implemented them, to see
>> how it looks like. Also, please let me know if it works in your
>> browser.
>

> NO!  I think they're terrible.  The more space a cell can occupy, the better.  I dislike how much border & space the current Sage notebook has.

That was another thing --- I really want the notebook to be
configurable, so that it's easy to rebrand it (e.g. change Sage to
something else), easy to change look & feel, like the thing above.
Ideally just by changing the cell html prototype and CSS styles.

Ondrej

Stan Schymanski

unread,
Jul 21, 2009, 3:46:59 AM7/21/09
to codenod...@googlegroups.com, sage-...@googlegroups.com
Hi Ondrej

I think it would be great to have the notebook (linked to Sage) run in
Google Apps.

Ondrej Certik wrote:
> [snip]


>
> Also, question to all, do you like the In [3] and Out[3] lines? I
> don't have an opinion on it yet myself, so I implemented them, to see
> how it looks like. Also, please let me know if it works in your
> browser.
>

I am used to the In [3] and Out [3] display from Mathematica and I liked
it. The sage notebook has similar tags internally, but for some reason
they are not displayed in the notebook. The good thing is that if you do
Evaluate all, the In [1] etc. are numbered consecutively and you can use
this to refer to bits of notebook code in a published notebook or a pdf
version of your notebook. This comes in very handy if you use a notebook
to derive equations used in e.g. Fortran and you would like to point to
the right place in the notebook in your Fortran code.

Stan

Jason Grout

unread,
Jul 21, 2009, 3:50:30 AM7/21/09
to sage-...@googlegroups.com
Ondrej Certik wrote:
> Hi,
>
> I finally learned javascript and AJAX, so that I can help with the
> notebook. I also studied it's sources.
>
> First things I like:
>
> * I like the user interface, it's usable, especially the attention to
> little details, like borders around the cells, tab completion, tab
> indentation and things like that.
>
> Things I don't like:
>
> * the javascript is really hackish overall, but two things really
> caught my attention:
> a) the keyboard handling is horrific, why not to use some standard
> library for that, that works across all browsers
> b) it uses some custom format for transfering data (which has bugs,
> like http://groups.google.com/group/sage-devel/browse_thread/thread/5ecd104b0aa85439),
> why not to use JSON?
> * it doesn't run on the google appengine (William mentioned in the
> past, that he doesn't see any benefit to do that, or that it would be
> slow)
>
> Well, talk is cheap, so here is the code (a sample Firefox screenshot
> is also attached in case it didn't work in your browser):
>
> http://pythonnb.appspot.com/
>

Very nice! The log shows you've been committing to it for only one day!
That's amazing.

It seems to work on Firefox 3.5.1 on Ubuntu 9.04 32-bit.

Jason

Robert Bradshaw

unread,
Jul 21, 2009, 3:58:46 AM7/21/09
to sage-...@googlegroups.com
On Jul 20, 2009, at 9:02 PM, Ondrej Certik wrote:

> Hi,
>
> I finally learned javascript and AJAX, so that I can help with the
> notebook. I also studied it's sources.
>
> First things I like:
>
> * I like the user interface, it's usable, especially the attention to
> little details, like borders around the cells, tab completion, tab
> indentation and things like that.
>
> Things I don't like:
>
> * the javascript is really hackish overall, but two things really
> caught my attention:
> a) the keyboard handling is horrific, why not to use some standard
> library for that, that works across all browsers
> b) it uses some custom format for transfering data (which has bugs,
> like http://groups.google.com/group/sage-devel/browse_thread/thread/
> 5ecd104b0aa85439),
> why not to use JSON?
> * it doesn't run on the google appengine (William mentioned in the
> past, that he doesn't see any benefit to do that, or that it would be
> slow)

Very cool! AJAX, and javascript libraries, and browsers have improved
a lot since the notebook was first written--I think a lot of this can
be cleaned up now.

> Well, talk is cheap, so here is the code (a sample Firefox screenshot
> is also attached in case it didn't work in your browser):
>
> http://pythonnb.appspot.com/
>
> it uses jQuery all over, it uses a keyboard plugin for jQuery, it uses
> JSON and it runs on the google appengine (and anywhere else too, it's
> just a standard django app). I tested in Firefox and IE8. The keyboard
> works, there are just some subtle bugs on IE8, see here:
>
> http://github.com/certik/notebook/blob/
> 375a2026ee7ea721904d05068724b3a7663d018e/todo
>
> but none of it seems major to me, the keyboard seems to be working
> just fine (or is IE8 not the most problematic? I'll try to test in
> other browsers like Opera and Safari too). Here is the index.html with
> all the javascript that I wrote:
>
> http://github.com/certik/notebook/blob/
> 375a2026ee7ea721904d05068724b3a7663d018e/templates/index.html
>
> It handles most of the keyboard interaction. It doesn't have TAB
> completion and inspection yet.
>
> Well, let me say that I really like to run things on the appengine,
> rather than to constantly maintain our own servers. I see no reason
> why the notebook cannot run on the appengine, only the AJAX would talk
> to our own server with Sage to actually evaluate the cells (and for
> many people, I think appengine itself could actually be enough). I
> have to think though what the best way to transfer data to the
> database with worksheets is though.

+1, though for Sage we rely heavily on compiled code. I wonder how
much introduced latency there would be if the backend were served on
a university computer, and the front end in appengine.

> I wanted to ask --- which parts of the Sage notebook are BSD licensed?
> I used a bit of the CSS styles and and maybe one javascript function,
> everything else was written by me. If possible, I'd like to use the
> BSD license for the notebook (if I find time to work on it further),
> so that ipython can use it by default.

I release everything I've contributed under sage/server/* under BSD.

Here's a complete list. It looks longer than it is, and I bet most of
these people only contributed once. It'll be cleaner when it's
separate into a separate spkg.

$ hg log sage/server/*/*.py* | grep "user:" | sort | uniq
user: "Justin C. Walker <jus...@mac.com>"
user: 'Martin Albrecht <ma...@informatik.uni-bremen.de>'
user: Alex Clemesha <clem...@gmail.com>
user: Alexandru Ghitza <agh...@alum.mit.edu>
user: Bobby Moretti <mor...@u.washington.edu>
user: Carl Witty <cwi...@newtonlabs.com>
user: Christian Wuthrich <christian...@gmail.com>
user: Dan Drake <dr...@kaist.edu>
user: Dan Drake <dr...@mathsci.kaist.ac.kr>
user: Dorian Raymer <deld...@gmail.com>
user: Harald Schilly <harald....@gmail.com>
user: Igor Tolkov <ito...@gmail.com>
user: J. H. Palmieri <palm...@math.washington.edu>
user: Jason Grout <gr...@rayunion.org>
user: Jason Grout <jason...@creativetrax.com>
user: John H. Palmieri <palm...@math.washington.edu>
user: Karl-Dieter Crisman <kcri...@gmail.com>
user: Marshall Hampton <hamp...@gmail.com>
user: Martin Albrecht <ma...@informatik.uni-bremen.de>
user: Mike Hansen <mha...@gmail.com>
user: Mitesh Patel <qed...@gmail.com>
user: Nick Alexander <ncale...@gmail.com>
user: Paul Dehaye <pauloli...@gmail.com>
user: Paul Zimmermann <zimm...@loria.fr>
user: Rob Beezer <bee...@ups.edu>
user: Robert Bradshaw <robe...@math.washington.edu>
user: Robert L. Miller <r...@rlmiller.org>
user: Robert Miller <rlmil...@gmail.com>
user: Timothy Clemans <timothy...@gmail.com>
user: Tom Boothby <boo...@u.washington.edu>
user: Wilfried Huss <hu...@finanz.math.tugraz.at>
user: William Stein <wst...@gmail.com>
user: William Stein <wst...@ucsd.edu>
user: Yi Qiang <yqi...@gmail.com>
user: agc@kubuntu
user: boo...@eight.math.washington.edu
user: boothby@localhost
user: boo...@localhost.localdomain
user: boo...@u.washington.edu
user: mabshoff@localhost
user: mabs...@sage.math.washington.edu
user: root@sage
user: sa...@ubuntu-server.localdomain
user: w...@bsd.local
user: w...@keyah.local
user: was@localhost
user: w...@localhost.localdomain
user: was@ubuntu
user: wst...@gmail.com

> Also, question to all, do you like the In [3] and Out[3] lines?

No, but maybe that's just me.

> I don't have an opinion on it yet myself, so I implemented them, to
> see
> how it looks like. Also, please let me know if it works in your
> browser.

Works great for me.

- Robert

Jason Grout

unread,
Jul 21, 2009, 4:18:07 AM7/21/09
to sage-...@googlegroups.com
Robert Bradshaw wrote:

> I release everything I've contributed under sage/server/* under BSD.

I also release everything I've contributed up to this point under
sage/server/* under BSD.

Jason

--
Jason Grout

Marshall Hampton

unread,
Jul 21, 2009, 4:38:34 AM7/21/09
to sage-devel
My contributions to the server code have been pretty trivial, but in
case it matters:

I release everything I've contributed under sage/server/* under BSD.

I'm excited to see this work on the notebook; I've been trying to
learn some jQuery and it seems quite nice. It seems in the future
there will be more possibilities of moving things to javascript, as it
gets nicer and the implementations speed up (for example, the recent
work by William and John Palmieri on animations using javascript).

Cheers,
Marshall Hampton

Tim Dumol

unread,
Jul 21, 2009, 6:47:13 AM7/21/09
to sage-devel
It's working great in Firefox 3.5, Windows XP x32 and Linux x86_64.

I'm doing some work on converting the notebook to Jinja (
http://trac.sagemath.org/sage_trac/ticket/6568 ). It shouldn't be too
hard to convert my work from Jinja templates to Django templates, or
to switch the Django templating engine to Jinja.

I'd love to help in the notebook rewrite after I finish the
conversion. I can fork your project at that time, and help out.

Pat LeSmithe

unread,
Jul 21, 2009, 11:52:35 AM7/21/09
to sage-...@googlegroups.com

Very promising! Just yesterday, I found Google's tutorial on writing a
Python application for their App Engine:

http://code.google.com/appengine/docs/python/gettingstarted/

There are some sample projects at

http://code.google.com/p/google-app-engine-samples/

Actually, I was motivated not to rewrite the notebook, but to adapt a
Python web server for controlling and monitoring the process of building
and testing Sage.

For example, we might use a local web dashboard to run doctests and
quickly get a list of the failures. Machines on a build farm could
occupy individual tabs. Maybe individual Sage developers could send
automated build reports (and logs, as necessary) to sagemath.org or a cloud.

How soon into the build process can we bring Sage to life, that is,
start running at least a minimal server?

Ondrej Certik

unread,
Jul 21, 2009, 12:24:50 PM7/21/09
to sage-...@googlegroups.com

Awesome! I'll wait after you do it and then I'll just use your
templates. Just use jinja, it works great and django can use it too.

Ondrej

Ondrej Certik

unread,
Jul 21, 2009, 12:27:52 PM7/21/09
to codenod...@googlegroups.com, sage-...@googlegroups.com
On Tue, Jul 21, 2009 at 3:49 AM, James Casbon<cas...@gmail.com> wrote:
>
> 2009/7/21 Ondrej Certik <ond...@certik.cz>:

>> but none of it seems major to me, the keyboard seems to be working
>> just fine (or is IE8 not the most problematic? I'll try to test in
>> other browsers like Opera and Safari too). Here is the index.html with
>> all the javascript that I wrote:
>
> Safari works for me.

>
>> Also, question to all, do you like the In [3] and Out[3] lines? I
>> don't have an opinion on it yet myself, so I implemented them, to see
>> how it looks like. Also, please let me know if it works in your
>> browser.
>
> They're ok, but they implied to me I could use, e.g., _4 as a variable
> (in the same way as ipython) - but you can't.  Also, when you insert a

I added this to the TODO.

> new cell the numbering gets out of order which looks messy.  What is
> the value in having them numbered?

The cells have to have some numbers, but they can be internal of
course. It helped me as a developer to see which cell is what,
especially when merging them. It seems most people don't like the
In/Out labels, so I will make them off by default and implement an
option to turn them on.

>
> By the way,  if you print you don't see the results.

Yes, I need to catch stdout and send it to the browsers. I added it to
the TODO list.

Many thanks for the feedback.

Ondrej

William Stein

unread,
Jul 21, 2009, 12:31:09 PM7/21/09
to codenod...@googlegroups.com, sage-...@googlegroups.com
On Tue, Jul 21, 2009 at 9:27 AM, Ondrej Certik<ond...@certik.cz> wrote:
>
> On Tue, Jul 21, 2009 at 3:49 AM, James Casbon<cas...@gmail.com> wrote:
>>
>> 2009/7/21 Ondrej Certik <ond...@certik.cz>:
>>> but none of it seems major to me, the keyboard seems to be working
>>> just fine (or is IE8 not the most problematic? I'll try to test in
>>> other browsers like Opera and Safari too). Here is the index.html with
>>> all the javascript that I wrote:
>>
>> Safari works for me.
>>
>>> Also, question to all, do you like the In [3] and Out[3] lines? I
>>> don't have an opinion on it yet myself, so I implemented them, to see
>>> how it looks like. Also, please let me know if it works in your
>>> browser.
>>
>> They're ok, but they implied to me I could use, e.g., _4 as a variable
>> (in the same way as ipython) - but you can't.  Also, when you insert a
>
> I added this to the TODO.
>
>> new cell the numbering gets out of order which looks messy.  What is
>> the value in having them numbered?
>
> The cells have to have some numbers, but they can be internal of
> course. It helped me as a developer to see which cell is what,
> especially when merging them. It seems most people don't like the
> In/Out labels, so I will make them off by default and implement an
> option to turn them on.

Long ago Sage used to have numbers in the cells, but after a typical
session with lots of random insertions of new cells, the numbers can
easily get confusing as one is just presented with a random list of
numbers.

That said, the capability of referring to the output of previous cells
via a notation like Out[17] is very handy, and a *lot* of users really
like it (Sage doesn't really have that). So a way to easily toggle
the numbers on and off like you suggest is probably best.

William

>>
>> By the way,  if you print you don't see the results.
>
> Yes, I need to catch stdout and send it to the browsers. I added it to
> the TODO list.
>
> Many thanks for the feedback.
>
> Ondrej
>
> >
>



--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

Ondrej Certik

unread,
Jul 21, 2009, 12:39:34 PM7/21/09
to sage-...@googlegroups.com
On Tue, Jul 21, 2009 at 1:58 AM, Robert
Bradshaw<robe...@math.washington.edu> wrote:
>
> On Jul 20, 2009, at 9:02 PM, Ondrej Certik wrote:

>> Well, let me say that I really like to run things on the appengine,
>> rather than to constantly maintain our own servers. I see no reason
>> why the notebook cannot run on the appengine, only the AJAX would talk
>> to our own server with Sage to actually evaluate the cells (and for
>> many people, I think appengine itself could actually be enough). I
>> have to think though what the best way to transfer data to the
>> database with worksheets is though.
>
> +1, though for Sage we rely heavily on compiled code. I wonder how
> much introduced latency there would be if the backend were served on
> a university computer, and the front end in appengine.

I think none, it would be as fast as it is now (e.g. the browser
communicating directly with the engine).

I would like to decouple Sage as the *engine* from the rest. The
engine should handle evaluating cells and storing and retrieving the
state (I guess). Then it can be used in services like Google Wave that
Harald is experimenting with etc.

The AJAX in the browser should be talking directly to the engine (e.g.
just like it is now). Where the rest of it is running, that doesn't
really matter imho and it should be possible to run it on the
appengine.

Ondrej

William Stein

unread,
Jul 21, 2009, 12:44:01 PM7/21/09
to sage-...@googlegroups.com
On Tue, Jul 21, 2009 at 9:39 AM, Ondrej Certik<ond...@certik.cz> wrote:
>
> On Tue, Jul 21, 2009 at 1:58 AM, Robert
> Bradshaw<robe...@math.washington.edu> wrote:
>>
>> On Jul 20, 2009, at 9:02 PM, Ondrej Certik wrote:
>
>>> Well, let me say that I really like to run things on the appengine,
>>> rather than to constantly maintain our own servers. I see no reason
>>> why the notebook cannot run on the appengine, only the AJAX would talk
>>> to our own server with Sage to actually evaluate the cells (and for
>>> many people, I think appengine itself could actually be enough). I
>>> have to think though what the best way to transfer data to the
>>> database with worksheets is though.
>>
>> +1, though for Sage we rely heavily on compiled code. I wonder how
>> much introduced latency there would be if the backend were served on
>> a university computer, and the front end in appengine.
>
> I think none, it would be as fast as it is now (e.g. the browser
> communicating directly with the engine).

How is it "none", given that there are now three separate computers
involved instead of two? There would have to be a little extra
latency, i.e., whatever there is between appengine and the "sage
engine". That said, the internet is pretty fast these days :-). And
the scalability of a decoupled approach like we're talking about is a
big plus, if it works.

By the way, if you haven't already, I personally think you should
start a mailing list, web page, trac, etc. for a separate notebook
project, since you're already writing code. There's already some
confusion about where we are supposed to have this discussion -- and a
funny mix of sage-devel and codenode doesn't seem right.

> I would like to decouple Sage as the *engine* from the rest. The
> engine should handle evaluating cells and storing and retrieving the
> state (I guess). Then it can be used in services like Google Wave that
> Harald is experimenting with etc.
>
> The AJAX in the browser should be talking directly to the engine (e.g.
> just like it is now). Where the rest of it is running, that doesn't
> really matter imho and it should be possible to run it on the
> appengine.
>
> Ondrej
>
> >
>



Ondrej Certik

unread,
Jul 21, 2009, 1:21:38 PM7/21/09
to sage-...@googlegroups.com
On Tue, Jul 21, 2009 at 10:44 AM, William Stein<wst...@gmail.com> wrote:
>
> On Tue, Jul 21, 2009 at 9:39 AM, Ondrej Certik<ond...@certik.cz> wrote:
>>
>> On Tue, Jul 21, 2009 at 1:58 AM, Robert
>> Bradshaw<robe...@math.washington.edu> wrote:
>>>
>>> On Jul 20, 2009, at 9:02 PM, Ondrej Certik wrote:
>>
>>>> Well, let me say that I really like to run things on the appengine,
>>>> rather than to constantly maintain our own servers. I see no reason
>>>> why the notebook cannot run on the appengine, only the AJAX would talk
>>>> to our own server with Sage to actually evaluate the cells (and for
>>>> many people, I think appengine itself could actually be enough). I
>>>> have to think though what the best way to transfer data to the
>>>> database with worksheets is though.
>>>
>>> +1, though for Sage we rely heavily on compiled code. I wonder how
>>> much introduced latency there would be if the backend were served on
>>> a university computer, and the front end in appengine.
>>
>> I think none, it would be as fast as it is now (e.g. the browser
>> communicating directly with the engine).
>
> How is it "none", given that there are now three separate computers
> involved instead of two?  There would have to be a little extra

What I meant is that the latency in typing 1+1 into the cell and get
the output cell saying 2 should not change at all, because the
javascript in the browser sends a POST request to the Sage engine
(e.g. a web app with the url interface, just like it is now) and it
returns it back directly to the browser.

What changes is the database storage, e.g. either the javascript in
the browser, once it receives the output of the cells also sends it to
the appengine (or whenever the database is running), or the engine
sends it itself, I don't know yet which approach is better. So there
are some issues involved, like if one of those connections fail etc.
But as long as both connections are up and running, the user would not
recognize anything at all.

> latency, i.e., whatever there is between appengine and the "sage
> engine".  That said, the internet is pretty fast these days :-).  And
> the scalability of a decoupled approach like we're talking about is a
> big plus, if it works.

Right, it has to be tried to see if it works. But I think it's worthy.

>
> By the way, if you haven't already, I personally think you should
> start a mailing list, web page, trac, etc. for a separate notebook
> project, since you're already writing code.   There's already some
> confusion about where we are supposed to have this discussion -- and a
> funny mix of sage-devel and codenode doesn't seem right.

Well, I hope codenode guys could pick this up and they would be the
notebook. I unfortunately probably can't spend too much time on this,
until september. But I wanted to get this going to see which approach
to take.

I wrote the above in about 2 days (roughly), but it's only the first
90%, e.g. the cells sort of works, but the rest 10%, like tab
completion, worksheets, saving. loading, publishing, users, fixing it
so that it works 100% in all browsers..... That would take a lot more,
and I can't do it yet. But I hope it's encouraging to all of you to
learn some AJAX too till September, so that we can work on this
together. :)

There is one more thing I want to try -- pyjamas, as pointed out
above. I already played with it yesterday, and what I saw so far is
*impressive*. So my next step will be to rewrite what I did into
pyjamas (e.g. just pure python both on the server and in the browser).
If that works and I think it could, well, that would be the way to go,
since I could debug all those functions like for calculating cursor
positions etc. in Python.

Ondrej

William Stein

unread,
Jul 21, 2009, 2:53:48 PM7/21/09
to sage-...@googlegroups.com

Thanks for the clarification, since I clearly misunderstood you.  Robert said "backend were served on a university computer, and the front end in appengine."  You seem to be eliminating the frontend completely when computations are done.  I.e., do you imagine appengine *just* serving some javascript and a database interface, and basically nothing else?  So what would happen is the following:

1. User visits the appengine server and gets the javascript for the sage notebook (after authenticating). 
2. User starts a worksheet.   The javascript in the browser requests a "sage engine token", and the appengine allocates a "compute engine" somewhere for use by that user's worksheet.
3. The user types "factor(2^197-1)" and their javascript *directly* connects to the compute engine and runs the code "factor(2^197-1)".  It also connects to the appengine and stores that "factor(2^197-1)" was input in the database.
4. The javascript in the browser gets back the answer to the factor query and displays the result.
5. The javascript in the browser later also stores the result in the app engine database.

I think there could be some weird security issues/tricks involved with the javascript in the browser directly doing AJAX calls to the "compute engine" above, but there are hacks to get around that.  There's also twice the communications overhead between the user's javascript and remote machines than in the current Sage notebook model where everything goes through the notebook server.    E.g., if the output of a Sage command (in step 4 and 5 above)  is large, e.g., a 10MB image, then that image is going to go all over the place, both uploaded and downloaded, which will be incredibly expensive.
 
What changes is the database storage, e.g. either the javascript in
the browser, once it receives the output of the cells also sends it to
the appengine (or whenever the database is running), or the engine
sends it itself, I don't know yet which approach is better. So there
are some issues involved, like if one of those connections fail etc.
But as long as both connections are up and running, the user would not
recognize anything at all.

This is an interesting design. It hadn't occured to me before.  It would be interesting to see whether it is any good or not (I can't tell).   

I can tell you one thing, which is that when I start working on the notebook again seriously this September, my first goal will be to create a powerful system for simulating the load of n people all using the notebook at once in a potentially heterogenous way (say from several different computers, etc.).  This testing code will be hopefully generic enough to work with codenode, sagenb, etc.   I think having actual benchmark testing code will in the longrun be a better litmus test for designs than us just thinking about them in the abstract. 

I could pronounce the design you suggest above as "bad" for several reasons, but what if I'm wrong and in fact the design above, with some tweaks and insights that would result from testing, turns out to be amazingly good? 


> latency, i.e., whatever there is between appengine and the "sage
> engine".  That said, the internet is pretty fast these days :-).  And
> the scalability of a decoupled approach like we're talking about is a
> big plus, if it works.

Right, it has to be tried to see if it works. But I think it's worthy.

>
> By the way, if you haven't already, I personally think you should
> start a mailing list, web page, trac, etc. for a separate notebook
> project, since you're already writing code.   There's already some
> confusion about where we are supposed to have this discussion -- and a
> funny mix of sage-devel and codenode doesn't seem right.

Well, I hope codenode guys could pick this up and they would be the
notebook. I unfortunately probably can't spend too much time on this,
until september. But I wanted to get this going to see which approach
to take.

Hey, same here.  Yeah for September.

I wrote the above in about 2 days (roughly), but it's only the first
90%, e.g. the cells sort of works, but the rest 10%, like tab
completion, worksheets, saving. loading, publishing, users, fixing it
so that it works 100% in all browsers..... That would take a lot more,
and I can't do it yet. But I hope it's encouraging to all of you to
learn some AJAX too till September, so that we can work on this
together. :)

There is one more thing I want to try -- pyjamas, as pointed out
above. I already played with it yesterday, and what I saw so far is
*impressive*. So my next step will be to rewrite what I did into
pyjamas (e.g. just pure python both on the server and in the browser).
If that works and I think it could, well, that would be the way to go,
since I could debug all those functions like for calculating cursor
positions etc. in Python.

I strongly encourage you to test pyjamas with the above.  I think that's the best possible next step.

 -- William 

Tim Dumol

unread,
Jul 21, 2009, 3:08:03 PM7/21/09
to sage-devel
On Jul 22, 12:24 am, Ondrej Certik <ond...@certik.cz> wrote:
> On Tue, Jul 21, 2009 at 4:47 AM, Tim Dumol<timdu...@gmail.com> wrote:
>
> > It's working great in Firefox 3.5, Windows XP x32 and Linux x86_64.
>
> > I'm doing some work on converting the notebook to Jinja (
> >http://trac.sagemath.org/sage_trac/ticket/6568). It shouldn't be too
> > hard to convert my work from Jinja templates to Django templates, or
> > to switch the Django templating engine to Jinja.
>
> > I'd love to help in the notebook rewrite after I finish the
> > conversion. I can fork your project at that time, and help out.
>
> Awesome! I'll wait after you do it and then I'll just use your
> templates. Just use jinja, it works great and django can use it too.
>
> Ondrej

I'm mostly done with notebook.py -- just have to check for any
dependencies on the functions I replaced before I clean everything up.
The templates output pretty much the same output as the original
functions, as far as I can tell. I'll see if I can make things more
semantic after the migration -- <br>'s to <p>'s, and table layout to
CSS -- if there's no problem with that?

Dorian Raymer

unread,
Jul 21, 2009, 3:11:38 PM7/21/09
to sage-...@googlegroups.com, codenod...@googlegroups.com
Hi Ondrej,

I like what you have done!



This is very cool. The "notebook" is really the encapsulation of at least three different projects.
 - A formal api interface to the Python or Sage interpreter (and that implementation of the interface for each of those systems)
 - Some kind of canonical and portable persistent notebook format (and something to manage all your notebooks)
 - The front end client (javascript/html/css) that is the notebook you actually use (really, a source code/text editor of which many projects simply trying to do this right (in the browser) exist)

In terms of distributing the components/responsibilities of the different parts (like what you are talking about with the AJAX computation requests to the sage server being different from the appengine frontend hosting, and then somehow integrating a possible third database element distinct from app engine (something I definitely want, because I want to own my data!!)), codenodes design is centered around this and has many cool (improving/improvable) solutions.

Although our current use case with app engine is a different permutation, as we are delegating all computation requests to it and keeping the data and frontend on our own server, the architecture is getting there to be able to do any permutation that makes sense.
I have been working on improving the backend and have recently made some great progress: http://github.com/deldotdr/codenode/commit/5a9ed5a19e0f71c48d8f62bb206f8b1aa347d1d6

Some of the key highlights:
 - I want it to be trivial to add different backend engines as Plugins. This means things like Sage, and other non-python interpreters. There are a handful of major configuration items: path to interpreter bin, args, environment variables (the hardest part/most work for sage), and run path.

-  frontend is able to know about multiple different backends (like app engine, a sage server, or another privately hosted server), each of those possibly having different engine plugins (i.e. Python and Sage at least).

- clear decoupling of the communication lines for administering all user data and backend permissions, getting/saving notebook data to a database, evaluating/tab completing/etc. code on an interpreter process.

It would be great to get feedback on this new stuff as I integrate it into the full system. There are a lot of design decisions in the same vein as your questions of the right way to separate the AJAX computation requests from saving to the database, etc.

 

>
> By the way, if you haven't already, I personally think you should
> start a mailing list, web page, trac, etc. for a separate notebook
> project, since you're already writing code.   There's already some
> confusion about where we are supposed to have this discussion -- and a
> funny mix of sage-devel and codenode doesn't seem right.

Well, I hope codenode guys could pick this up and they would be the
notebook. I unfortunately probably can't spend too much time on this,
until september. But I wanted to get this going to see which approach
to take.

Cool! With codenode, we have strived to make the major components as decoupled as possible such that people can hack on any one of them (mostly) independent of the others. Like you, we really want to work on it all the time, full time, but we also have real jobs and other projects that take up our time. I think collaborating would be the best thing that could happen for the project and the evolution of the notebook.

We have put in significant effort in thought and code writing and, although it has not been very obviously presented in the past, we have the foundation for something that we strongly want to be the basis of the effort you are in the midst of now. We have been through the process you are in, and hopefully a fresh review of our source code (especially the javascript and the backend design) will resonate with your current thought process.
 

I wrote the above in about 2 days (roughly), but it's only the first
90%, e.g. the cells sort of works, but the rest 10%, like tab
completion, worksheets, saving. loading, publishing, users, fixing it
so that it works 100% in all browsers..... That would take a lot more,
and I can't do it yet. But I hope it's encouraging to all of you to
learn some AJAX too till September, so that we can work on this
together. :)

;-) Yes, it takes *a lot* more. A quick review of what codenode has sitting there already:

- Generalized notion of a cell in the notebook. Cells can contain anything, even more cells! This is great for handling different output cells (text, traceback, plots), different input formats, and doing col things like Mathematica does with nested sections -- maybe that is too much, but when used in the simplest case, it gets the job done smoothly. We have a cell id scheme worked out that allows any arbitrary addition, deletion, and rearrangement of cells.

- Tab completer, decoupled from cell evaluation. This has it's own javascript source code file, Completer.js

- Event Delegation and pretty simple (definitely refine-able) configuration of key + mouse combination event handling. This doesn't use the jQuery plugin yet, but it works great and shows how far down the path we have already gotten.


The sage notebook and codenode represent two extremes in design ideology/practicality, but they both already have great usable functionality supported by untold man hours of hard work and thought.
I think a lot of progress on a great generally usable notebook can be made with the help and drive of a third perspective like your own [Ondrej].

There is too much work remaining, work ranging from realizing un-implemented big features to last mile refinements, not to mention reconciling the awesome but coupled features of the sage notebook (plotting, interact, etc) into your (and codenode's) more general and decoupled model.
 
So, I am all for combining efforts and becoming more in tune with your end goal and motivation for improving the notebook.

-Dorian
 

Ondrej Certik

unread,
Jul 21, 2009, 3:57:53 PM7/21/09
to sage-...@googlegroups.com

As to me, definitely use CSS for everything and remove all tables if
there are some. CSS is easy to customize by people.

Ondrej

Ondrej Certik

unread,
Jul 21, 2009, 4:01:06 PM7/21/09
to codenod...@googlegroups.com, sage-...@googlegroups.com
Hi Dorian!

I agree with everything you wrote.

Only one suggestion -- could you take my simple frontend for the cells
and incorporate it in codenode? I mean how things *look* like, so that
it looks like the Sage notebook. The default codenode look & feel
doesn't work well in my browser, since I can't figure out where to
click to find the cell, the cursor changes in some weird way and
generally it's confusing to me etc. So that's a major problem, but the
fix is really easy, just change the bit of the javascript + CSS styles
and it will look like Sage. There could be some option to choose
between the two designs if you prefer the current codenode style.

Ondrej

Ondrej Certik

unread,
Jul 21, 2009, 4:13:02 PM7/21/09
to sage-...@googlegroups.com

That's exactly correct.

Another possibility is to change 5) into 5'):

5') the Sage engine talks to the appengine database server directly.

The advantage of 5') over 5) is that the Sage engine should be running
on some fast network anyways (thus the communication Sage engine <->
app engine server will be fast), but the user's laptop can be on some
crappy connection.

>
> I think there could be some weird security issues/tricks involved with the
> javascript in the browser directly doing AJAX calls to the "compute engine"
> above, but there are hacks to get around that.  There's also twice the

Right.

> communications overhead between the user's javascript and remote machines
> than in the current Sage notebook model where everything goes through the
> notebook server.    E.g., if the output of a Sage command (in step 4 and 5
> above)  is large, e.g., a 10MB image, then that image is going to go all
> over the place, both uploaded and downloaded, which will be incredibly
> expensive.

I agree, I think we should use 5'). E.g. if the database engine and
Sage engine is running on the same machine, that's the current design,
but if they are decoupled, but connected using fast internet, it could
work.

The appengine database backend has to have some notion of the engine
anyways, so it might as well retreive from it the results.

I agree that it might be too complex/tricky/error prone. I simply don't know.

>
>>
>> What changes is the database storage, e.g. either the javascript in
>> the browser, once it receives the output of the cells also sends it to
>> the appengine (or whenever the database is running), or the engine
>> sends it itself, I don't know yet which approach is better. So there
>> are some issues involved, like if one of those connections fail etc.
>> But as long as both connections are up and running, the user would not
>> recognize anything at all.
>
> This is an interesting design. It hadn't occured to me before.  It would be
> interesting to see whether it is any good or not (I can't tell).

Me neither.

>
> I can tell you one thing, which is that when I start working on the notebook
> again seriously this September, my first goal will be to create a powerful
> system for simulating the load of n people all using the notebook at once in
> a potentially heterogenous way (say from several different computers,
> etc.).  This testing code will be hopefully generic enough to work with
> codenode, sagenb, etc.   I think having actual benchmark testing code will
> in the longrun be a better litmus test for designs than us just thinking
> about them in the abstract.
>
> I could pronounce the design you suggest above as "bad" for several reasons,
> but what if I'm wrong and in fact the design above, with some tweaks and
> insights that would result from testing, turns out to be amazingly good?

Exactly. I don't know myself and I am not sure about exact technical
details of my design, e.g. 5) vs 5') etc. But my motivation is that I
really want it to be able to run on the appengine completely if
needed, because there are tons of situations, where I just want to
show off some simple thing, be it sympy, or just some simple algorithm
in python and I really *don't* want to maintain my own server for
that.

At the same time however, I really would like to just create a simple
engine with web API (be it Sage, or anything else), and I would like
to maintain just this engine and if it dies, the frontend (running
somewhere else) would just use a different engine, or whatever.

So I would like to have that, but if it's possible to get everything
right and robust and fast, I simply don't know.

> I strongly encourage you to test pyjamas with the above.  I think that's the
> best possible next step.

I will report later on this. It seems to work, but I can already see a
big issue -- it seems a bit slow (e.g. the generated javascript in the
browser). But it's too early to tell, once I implement the same thing,
we can then compare which approach is the best in the long run.

Ondrej

William Stein

unread,
Jul 21, 2009, 4:21:38 PM7/21/09
to sage-...@googlegroups.com
Note that there are new security implications to 5' not in 5. Without
more careful thought, the Sage engine has to have whatever
authentication credentials as the user, since the Sage engine suddenly
gets to change anything in the user's worksheets. This isn't
necessary a problem, but is something to think about.

>
>>
>> I think there could be some weird security issues/tricks involved with the
>> javascript in the browser directly doing AJAX calls to the "compute engine"
>> above, but there are hacks to get around that.  There's also twice the
>
> Right.
>
>> communications overhead between the user's javascript and remote machines
>> than in the current Sage notebook model where everything goes through the
>> notebook server.    E.g., if the output of a Sage command (in step 4 and 5
>> above)  is large, e.g., a 10MB image, then that image is going to go all
>> over the place, both uploaded and downloaded, which will be incredibly
>> expensive.
>
> I agree, I think we should use 5'). E.g. if the database engine and
> Sage engine is running on the same machine, that's the current design,
> but if they are decoupled, but connected using fast internet, it could
> work.
>

Another issue with 5' is that it means the "sage engine" has to be
able to open new outgoing connections to communicate with the database
server. This could be a problem if the sage engine is running in some
sort of locked down sandboxed environment. Again, this isn't
insurmountable, but you should keep it in mind.

> The appengine database backend has to have some notion of the engine
> anyways, so it might as well retreive from it the results.
>
> I agree that it might be too complex/tricky/error prone. I simply don't know.

I don't either. Trickiness is all relative. If you encapsulate
things with a good design, you can sometimes build up very complicated
tricky systems that seem simple.
That sounds like very useful information. Benchmarking is super super
important for something like this, since javascript is already slow.

mmarco

unread,
Jul 21, 2009, 4:37:44 PM7/21/09
to sage-devel
I have tested your demo. It works fine in firefox, but i can't see the
cells in konqueror 3.5.10 (sage notebook doesn't look well in
konqueror, but most of the functionalities work).

Alex Clemesha

unread,
Jul 21, 2009, 4:55:29 PM7/21/09
to codenod...@googlegroups.com, sage-...@googlegroups.com
Hi Ondrej,


>> So, I am all for combining efforts and becoming more in tune with your end
>> goal and motivation for improving the notebook.
>
> I agree with everything you wrote.
>
> Only one suggestion -- could you take my simple frontend for the cells
> and incorporate it in codenode? I mean how things *look* like, so that
> it looks like the Sage notebook. The default codenode look & feel
> doesn't work well in my browser, since I can't figure out where to
> click to find the cell, the cursor changes in some weird way and
> generally it's confusing to me etc. So that's a major problem, but the
> fix is really easy, just change the bit of the javascript + CSS styles
> and it will look like Sage. There could be some option to choose
> between the two designs if you prefer the current codenode style.

I'm going to make supporting the 'Sage Notebook style' a top priority of mine.
I'll put it in as a setting, like the "open notebook in new window" setting
(which was motivated by you... thanks for that!)

Please keep requests like this coming.

--

I'm absolutely confident that the core architecture of codenode is
solid - a massive amount of effort has been put into designing it
to be formed from decouple pieces, making it easier to extend in
the long run. Dorian's detailed response describes this.

What codenode is still missing is the "last mile bits" from real world
users like yourself, and with your feedback and involvement,
we'll be able to create something that exactly solves the problems
you want to be solved.


-Alex




>
> Ondrej
>
> >
>



--
Alex Clemesha
clemesha.org

J Elaych

unread,
Jul 21, 2009, 4:56:57 PM7/21/09
to sage-devel

>
> There is one more thing I want to try -- pyjamas, as pointed out
> above. I already played with it yesterday, and what I saw so far is
> *impressive*. So my next step will be to rewrite what I did into
> pyjamas (e.g. just pure python both on the server and in the browser).
> If that works and I think it could, well, that would be the way to go,
> since I could debug all those functions like for calculating cursor
> positions etc. in Python.
>
> Ondrej

Well technically it wouldn't be 'python in the browser', right? You
would compile the pyjamas python code into javascript, store it
on the server and send the js to the client browser.

William Stein

unread,
Jul 21, 2009, 5:35:12 PM7/21/09
to sage-...@googlegroups.com

(1) Can pyjamas cleanly make use of arbitrary javascript libraries?

(2) Is there a list of nontrivial examples where pyjamas is actually
used to implement javascript apps?

-- William

Ondrej Certik

unread,
Jul 21, 2009, 5:39:15 PM7/21/09
to sage-...@googlegroups.com

Yes. But you don't have to write javascript and if you do things
correctly, the same file executes on your desktop using python, thus
you can doctest the whole thing.

I already implemented the textbox resizing, here is the code:

---------------

import pyjd # this is dummy in pyjs.
from pyjamas.ui.RootPanel import RootPanel
from pyjamas.ui.Button import Button
from pyjamas.ui.HTML import HTML
from pyjamas.ui.Label import Label
from pyjamas import Window
from pyjamas.ui.TextArea import TextArea
from pyjamas.ui import KeyboardListener


def greet(fred):
print "greet button"
Window.alert("Hello, AJAX!")

class InputArea(TextArea):

def __init__(self, echo):
TextArea.__init__(self)
self.echo = echo
self.addKeyboardListener(self)
self.addClickListener(self)
self.set_rows(1)
self.setCharacterWidth(80)

def onClick(self, sender):
print "on_click"

def rows(self):
return self.getVisibleLines()

def set_rows(self, rows):
if rows in [0, 1]:
# this is a bug in pyjamas, we need to use 2 rows
rows = 2
# the number of rows seems to be off by 1, another bug in pyjamas
self.setVisibleLines(rows-1)

def cols(self):
return self.getCharacterWidth()

def occupied_rows(self):
text = self.getText()
lines = text.split("\n")
return len(lines)

def cursor_coordinates(self):
"""
Returns the cursor coordinates as a tuple (x, y).

Example:

>>> self.cursor_coordinates()
(2, 3)
"""
text = self.getText()
lines = text.split("\n")
pos = self.getCursorPos()
i = 0
cursor_row = -1
cursor_col = -1
#print "--------" + "start"
for row, line in enumerate(lines):
i += len(line) + 1 # we need to include "\n"
# print len(line), i, pos, line
if pos < i:
cursor_row = row
cursor_col = pos - i + len(line) + 1
break
#print "--------"
return (cursor_col, cursor_row)

def insert_at_cursor(self, inserted_text):
pos = self.getCursorPos()
text = self.getText()
text = text[:pos] + inserted_text + text[pos:]
self.setText(text)

def onKeyUp(self, sender, keyCode, modifiers):
#print "on_key_up"
x, y = self.cursor_coordinates()
rows = self.occupied_rows()
s = "row/col: (%s, %s), cursor pos: %d, %d, real_rows: %d" % \
(self.rows(), self.cols(), x, y, rows)
self.set_rows(rows)
self.echo.setHTML("Info:" + s)

def onKeyDown(self, sender, key_code, modifiers):
if key_code == KeyboardListener.KEY_TAB:
self.insert_at_cursor(" ")
print "TAB"

#def onKeyDownPreview(self, key, modifier):
# print "preview"

def onKeyPress(self, sender, keyCode, modifiers):
#print "on_key_press"
pass


if __name__ == '__main__':
pyjd.setup("../templates/Hello.html")
b = Button("Click me", greet, StyleName='teststyle')
h = HTML("<b>Hello World</b> (html)", StyleName='teststyle')
l = Label("Hello World (label)", StyleName='teststyle')
echo = HTML()
t = InputArea(echo)
RootPanel().add(b)
RootPanel().add(h)
RootPanel().add(l)
RootPanel().add(t)
RootPanel().add(echo)
pyjd.run()


---------------

And it mostly works, up to some bugs in pyjamas (like that the textbox
can't be set to just 1 row, only 2 rows or more), that I will try to
solve with pyjamas developers, hopefully they are simple to fix.

If you look at the cursor_coordinates() function, this is really PITA
to debug in javascript -- I mean, you essentially need tests for the
javascript etc. If this could be avoided, that'd be a huge win. The
generated javascript for the function above is:

cls_definition.cursor_coordinates =
pyjs__bind_method(cls_instance, 'cursor_coordinates', function() {
if (this.__is_instance__ === true) {
var self = this;
} else {
var self = arguments[0];
}
var text = self.getText();
var lines = text.split(String('\x0A'));
var pos = self.getCursorPos();
var i = 0;
var cursor_row = -1;
var cursor_col = -1;
var __temp_row = pyjslib.enumerate(lines).__iter__();
try {
while (true) {
var temp_row = __temp_row.next();
var row = temp_row.__getitem__(0); var
line = temp_row.__getitem__(1);
i += ( pyjslib.len(line) + 1 ) ;
if (pyjslib.bool((pyjslib.cmp(pos, i) == -1))) {
cursor_row = row;
cursor_col = ( ( ( pos - i ) +
pyjslib.len(line) ) + 1 ) ;
break;
}
}
} catch (e) {
if (e.__name__ != 'StopIteration') {
throw e;
}
}
return new pyjslib.Tuple([cursor_col, cursor_row]);
}


So that doesn't look bad. But it's definitely slower than it could be
if used javascript for loop directly. But as I said, let's wait until
I implement the whole thing and let's see. Also pyjamas allow to embed
javascript code, so we may write the critical code in javascript
itself.

Ondrej

Rob Beezer

unread,
Jul 21, 2009, 5:42:43 PM7/21/09
to sage-devel
William Stein wrote:
> (1) Can pyjamas cleanly make use of arbitrary javascript libraries?

Here's an example of a pyjamas "application" using a JQuery slider
(its along the bottom edge of the map)

Demo: http://web2py.gdw2.com/maps/default/index

To my untrained eye it looks like there was some trivial overhead to
integrate the JQuery library, so it is not automatic. But maybe
"clean"?

Code: http://pastebin.com/fb45e73b

This is from the last post in:

http://groups.google.com/group/pyjamas-dev/browse_thread/thread/f403232f9216e135

Ondrej Certik

unread,
Jul 21, 2009, 5:44:34 PM7/21/09
to sage-...@googlegroups.com
On Tue, Jul 21, 2009 at 3:35 PM, William Stein<wst...@gmail.com> wrote:
>
> On Tue, Jul 21, 2009 at 1:56 PM, J Elaych<micro...@gmail.com> wrote:
>>
>>
>>>
>>> There is one more thing I want to try -- pyjamas, as pointed out
>>> above. I already played with it yesterday, and what I saw so far is
>>> *impressive*. So my next step will be to rewrite what I did into
>>> pyjamas (e.g. just pure python both on the server and in the browser).
>>> If that works and I think it could, well, that would be the way to go,
>>> since I could debug all those functions like for calculating cursor
>>> positions etc. in Python.
>>>
>>> Ondrej
>>
>> Well technically it wouldn't be 'python in the browser', right?  You
>> would compile the pyjamas python code into javascript, store it
>> on the server and send the js to the client browser.
>
> (1) Can pyjamas cleanly make use of arbitrary javascript libraries?

I think it can:

http://groups.google.com/group/pyjamas-dev/browse_thread/thread/639dffd00d6b7c7/

but I am still learning it.

>
> (2) Is there a list of nontrivial examples where pyjamas is actually
> used to implement javascript apps?

the examples directory is full of interesting examples --- well, it
depends what you mean nontrivial. Let me just implement it and let's
see after it.

Ondrej

Rado

unread,
Jul 21, 2009, 7:54:33 PM7/21/09
to sage-devel
Having read a bit of the old notebook code, for the graph editor
(which I should polish and submit in the next few days), I think Sage
will greatly benefit from notebook rewrite. It is very possible to
make front-end applets like the graph editor once it is easy to talk
to the sage kernel, without going through the notebook.

However, both solutions (codenode and ondrej) seem to complicate
things for single users by making them set-up two servers (one not
even local). Is there a way to bundle django engine with twisted's
sage server? Or at least ship it with sage and start it on a separate
port when 'notebook' is executed. So that the user can have the sage
computational engine and the new notebook (probably done with django's
web platform) on the same machine.

Also I think codenode has a lot of potential, but it is lost on non-
Mathematica users who find it hard to manipulate cells. I teach
Mathematica-based math courses and know exactly how non-intuitive
students find the cell structure at first. Actually I thought Sage
model is weird at first (being used to Mathematica), by letting me
jump from cell to cell and erasing them so easy. Now I preferred it to
Mathematica's rigidness.

Rado

ghtdak

unread,
Jul 21, 2009, 8:50:32 PM7/21/09
to sage-devel

This thread has gotten long and there are many subjects embedded
within.

One of the problems I've had with the notebook implementation is that
the sage process supporting the notebook computation blocks on the
pipe between itself and the twistd server which spawns it. This means
that one can't build an asynchronous event handler without using
threads.

More conventional web server architectures use a callback style for
invocation making it much easier to splice in other events for
handling by the main thread (this is the general asynchronous
programming model and the heart of how Twisted works)

Perhaps it is a foregone conclusion that this approach will be taken
in the rewrite. if not, I'd like to suggest that it is an important
consideration.

-glenn

Dr. David Kirkby

unread,
Jul 21, 2009, 9:40:59 PM7/21/09
to sage-...@googlegroups.com

I don't know about how the web server is implemented. I know it did not
work on my Solaris box, but that is another matter.

But actually including Apache might be a sensible choice. A lot of
people know how to administer Apache. It offers a lot of flexibility.
You can for example only serve pages to particular IP addresses.

Worth a thought anyway.

dave

Ondrej Certik

unread,
Jul 21, 2009, 11:03:58 PM7/21/09
to sage-...@googlegroups.com
Hi,

here is an early preview of the pyjamas version:

http://2.latest.pythonnb.appspot.com/

So far my experience is:

* it doesn't work in IE8 (that's a showstopper)
* it's fast enough
* implementing the cursor positions and resizing was a piece of cake
(I was very impressed)
* learning the whole framework took me some time, one has to read sources a lot
* if it doesn't work, it's a bit difficult to debug (because it's not
just javascript, I need to figure out where I made the mistake in the
python code), I basically use git a lot and always do a small change
and test, small change and test. If it fails, I break my changes in
half and test, etc.

Essentially, pyjamas provide a complete DOM access (just like jQuery),
but in Python, and then builds its own widgets on top of it.

I am now learning how to do AJAX with it. So far only the cursor
movement and cells work (the focus is not yet shown by a blue line,
I'll do that later). Try this:

def f(x):
<hit TAB couple times>

and then hit <backspace>, you will see that it deletes 4 spaces, but
in a clever way, e.g. if you are at a position 7, it goes to 4 first
and then to 0. This is how my vim is setup for python editing and I
like it a lot.

Let me know if it works in your browser so far. I only tested firefox
3.5, that works fine. IE8 doesn't load the javascript, e.g. you will
see no textbox. Also, in Firefox I get frequent error messages
(printed in the actual HTML):

"
JavaScript Error: Permission denied to get property
HTMLDivElement.parentNode at line number 9254. Please inform
webmaster.
"

It's a bug in pyjamas.

Ondrej

Alex Clemesha

unread,
Jul 21, 2009, 11:06:30 PM7/21/09
to sage-...@googlegroups.com, codenod...@googlegroups.com
On Tue, Jul 21, 2009 at 4:54 PM, Rado<rki...@gmail.com> wrote:
>
> Having read a bit of the old notebook code, for the graph editor
> (which I should polish and submit in the next few days), I think Sage
> will greatly benefit from notebook rewrite. It is very possible to
> make front-end applets like the graph editor once it is easy to talk
> to the sage kernel, without going through the notebook.
>
> However, both solutions (codenode and ondrej) seem to complicate
> things for single users by making them set-up two servers (one not
> even local). Is there a way to bundle django engine with twisted's
> sage server?
This is exactly how codenode works - there is only one webserver,
and that is twisted. Django is running off of twisted's wsgi implementation.

The command-line utility, "codenode-admin", hides all this,
making what happens behind the scenes totally transparent to the
normal, every day user.



Or at least ship it with sage and start it on a separate
> port when 'notebook' is executed. So that the user can have the sage
> computational engine and the new notebook (probably done with django's
> web platform) on the same machine.


> Also I think codenode has a lot of potential, but it is lost on non-
> Mathematica users who find it hard to manipulate cells. I teach
> Mathematica-based math courses and know exactly how non-intuitive
> students find the cell structure at first. Actually I thought Sage
> model is weird at first (being used to Mathematica), by letting me
> jump from cell to cell and erasing them so easy. Now I preferred it to
> Mathematica's rigidness.
This is yet another excellent data point making it clear to me that
we must adopt the easier-to-user sage style interface. This will be done soon.


thanks,
Alex






>
> Rado
> >
>



--
Alex Clemesha
clemesha.org

Ondrej Certik

unread,
Jul 21, 2009, 11:07:38 PM7/21/09
to sage-...@googlegroups.com

Also cell joining is not yet implemented. But it works in the Chrome
browser, so at least something.

Ondrej

Ondrej Certik

unread,
Jul 22, 2009, 5:40:26 AM7/22/09
to sage-...@googlegroups.com

I implemented the AJAX thing as well, here is a working example (sort of):

http://3.latest.pythonnb.appspot.com/media_files/output/index.html

Besides what I wrote above, there is some problem with CSS styles,
which shows up when you evaluate some cell and see the output, the
"insert new cell blue thin line" is misplaced.

I implemented it in the "pyjamas" branch in my repository (link is on
the webpage), so there are some leftovers (like the jQuery.js library,
which is *not* needed anymore, etc.). The django backend didn't change
at all and now the whole notebook doesn't contain a single line of
javascript (everything is pure python). I think that itself is pretty
impressive. Here is the file, that gets translated:

http://github.com/certik/notebook/blob/001b4ddf444b480822adf9216419afa1adaf4818/media/index.py

In terms of lines of code, it's about the same as my previous version
in javascript:

http://github.com/certik/notebook/blob/ca4e6a90a3f0c10c78c8c99d4d55055ba5019c28/templates/index.html

But there are still some things missing (e.g. joining and deleting the
cells). The python code should be refactored first though, there
should be a class Cell, that should have references to it's children,
like the input/output cells etc. and this class should now how to turn
on/off the output cell etc. Currently I am using the DOM directly in a
bit hackish way, this should be polished. Essentially I was fighting
pyjamas to access the elements in the DOM, for example jQuery's
analogs of insert before and after are not available (resp. it's
tricky).

Nevertheless, overall, I like the pyjamas approach and the above
things can be fixed. Also now there is a nice possibility to mock up
the controls and run regular python unittests on the whole thing.
That's a big plus.

The remaining big problem is the IE8 support and the errors that
sometimes popup in firefox. I reported it here:

http://groups.google.com/group/pyjamas-dev/browse_thread/thread/f170c3709c7f12ed

and I was told I am pretty much on my own with IE8. So that's very
disappointing of course, but maybe the fix is easy. If it is not, then
that's a big problem.

Ondrej

javier

unread,
Jul 22, 2009, 6:00:56 AM7/22/09
to sage-devel
For the pyjamas version, with Firefox 3.5.1 and Safari 4 under OS-X
10.4 I cannot see the outputs. In Firefox, I also get some

"JavaScript Error: Permission denied to access property 'parentNode'
from a non-chrome context at line number 9254. Please inform
webmaster."

The AJAX version (the one starting with a 3) throws a
" JavaScript Error: Permission denied to access property 'parentNode'
from a non-chrome context at line number 9488. Please inform
webmaster."
in Firefox. In Safari, no JS errors, but the output text appears
"cut" (you can only see the lower half of it) until you mouse over
it.

Will test under linux when get to my office.

Cheers
Javier
> http://github.com/certik/notebook/blob/001b4ddf444b480822adf9216419af...
>
> In terms of lines of code, it's about the same as my previous version
> in javascript:
>
> http://github.com/certik/notebook/blob/ca4e6a90a3f0c10c78c8c99d4d5505...
>
> But there are still some things missing (e.g. joining and deleting the
> cells). The python code should be refactored first though, there
> should be a class Cell, that should have references to it's children,
> like the input/output cells etc. and this class should now how to turn
> on/off the output cell etc. Currently I am using the DOM directly in a
> bit hackish way, this should be polished. Essentially I was fighting
> pyjamas to access the elements in the DOM, for example jQuery's
> analogs of insert before and after are not available (resp. it's
> tricky).
>
> Nevertheless, overall, I like the pyjamas approach and the above
> things can be fixed. Also now there is a nice possibility to mock up
> the controls and run regular python unittests on the whole thing.
> That's a big plus.
>
> The remaining big problem is the IE8 support and the errors that
> sometimes popup in firefox. I reported it here:
>
> http://groups.google.com/group/pyjamas-dev/browse_thread/thread/f170c...

Ondrej Certik

unread,
Jul 22, 2009, 1:12:38 PM7/22/09
to sage-...@googlegroups.com
On Wed, Jul 22, 2009 at 4:00 AM, javier<veng...@gmail.com> wrote:
>
> For the pyjamas version, with Firefox 3.5.1 and Safari 4 under OS-X
> 10.4 I cannot see the outputs. In Firefox, I also get some
>
> "JavaScript Error: Permission denied to access property 'parentNode'
> from a non-chrome context at line number 9254. Please inform
> webmaster."
>
> The AJAX version (the one starting with a 3) throws a
> " JavaScript Error: Permission denied to access property 'parentNode'
> from a non-chrome context at line number 9488. Please inform
> webmaster."
> in Firefox. In Safari, no JS errors, but the output text appears
> "cut" (you can only see the lower half of it) until you mouse over
> it.
>
> Will test under linux when get to my office.

Many thanks for testing it. That's a bad news. On the very top of
pyjamas website they wrote:

"
Also, the AJAX library takes care of all the browser interoperability
issues on your behalf, leaving you free to focus on application
development instead of learning all the "usual" browser
incompatibilities.
"

But the reality is that it works, unless you run Firefox and unless
you run linux. :(

We'll see if it can be fixed. If not, we'll have to abandon this
approach and just use jQuery.

Ondrej

Ondrej Certik

unread,
Jul 22, 2009, 1:13:27 PM7/22/09
to sage-...@googlegroups.com
On Wed, Jul 22, 2009 at 11:12 AM, Ondrej Certik<ond...@certik.cz> wrote:
[..]

>
> But the reality is that it works, unless you run Firefox and unless
> you run linux. :(

I mean as long as.

O.

ghtdak

unread,
Jul 22, 2009, 2:12:35 PM7/22/09
to sage-devel


On Jul 21, 6:40 pm, "Dr. David Kirkby" <david.kir...@onetel.net>
wrote:
I came across this post from the Twisted folks. It looks like they do
WSGI and run Django quite well...

http://blog.dreid.org/2009/03/twisted-django-it-wont-burn-down-your.html

At least there are alternatives to apache which might be simpler.

The core question remains: While the presentation layer support is
defined by django, pyjamas etc, the integration with the underlying
sage process process is the issue, not the web presentation.

In the current architecture, a twistd daemon spawns a notebook server
which is responsible for doing "sage" stuff. twistd is fully
asynchronous, but the notebook process itself is a pexpect based
blocking process connected with pipes to twistd. As such, the block
on read by pexpect precludes the sage process servicing asynchronous
events.

IMHO, this architecture is incorrect and limited... Perhaps this is
part of what is being rethought... if not, I believe it should be.

A preferable architecture is an event loop which dispatches requests
within the sage process. Since Sage is written in python, I would
suggest Twisted for this but there might be better alternatives (I'd
be surprised, but its possible)

Using this approach, one could easily add other elements to the core
event loop to support asynchronous processing (timers, communication,
etc) without threads which are, in this case, unnecessary. Threads
when necessary are bad enough, when they're introduced because of
unnecessary blocking, one gets all the threading nightmare without any
benefit. (reminds me of health care reform)

Another benefit is that since asynchronous event processing is the
widely accepted approach to this type of problem, there are lots of
libraries / packages to "make it so".

-glenn

>
> dave

William Stein

unread,
Jul 22, 2009, 2:18:25 PM7/22/09
to sage-...@googlegroups.com
Here's the relevant code that you're talking about (starting at line
3007 of worksheet.py):

try:
done, out, new = S._so_far(wait=wait,
alternate_prompt=SAGE_END+str(self.synchro()))
except RuntimeError, msg:
verbose("Computation was interrupted or failed.
Restarting.\n%s"%msg)
self.__comp_is_running = False
self.start_next_comp()
return 'w', C

The Sage notebook server does a blocking read from the
pexpect-controlled subprocess for "wait" seconds. The default value
of wait is 0.2 seconds. So indeed there is a block, but it is never
for more than 0.2 seconds. During that time other asynchronous
events can't be dealt with.

> IMHO, this architecture is incorrect and limited... Perhaps this is
> part of what is being rethought... if not, I believe it should be.
>
> A preferable architecture is an event loop which dispatches requests
> within the sage process.  Since Sage is written in python, I would
> suggest Twisted for this but there might be better alternatives (I'd
> be surprised, but its possible)
>
> Using this approach, one could easily add other elements to the core
> event loop to support asynchronous processing (timers, communication,
> etc) without threads which are, in this case, unnecessary.  Threads
> when necessary are bad enough, when they're introduced because of
> unnecessary blocking, one gets all the threading nightmare without any
> benefit. (reminds me of health care reform)
>
> Another benefit is that since asynchronous event processing is the
> widely accepted approach to this type of problem, there are lots of
> libraries / packages to "make it so".
>
> -glenn
>
>>
>> dave
> >
>



ghtdak

unread,
Jul 22, 2009, 5:19:01 PM7/22/09
to sage-devel
I'm not sure I understand, so let me restate what I think you're
saying:

The notebook "server" is doing a blocking read with a timeout. The
server is a twistd process which multiplexes and processes in support
the browsers.

The "pexpect-controlled subprocess" is where Sage itself is.

The issue I was raising was with the latter (subprocess) although I
now think the server implementation might also be an issue.

when twisted spawns a sub-process, typically one registers a callback
to handle information from that sub-process. There should be no
reason to check for input. Generally a callback for signals is also
registered to sense the sub-process dying, the pipe being disconnected
etc. Unless I misunderstand (very possible), you're "polling" the set
of spawned Sage sub-processes for data.

While that doesn't sound good either, thats not the problem I was
describing.

My primary problem is that the Sage subprocess is blocking forever on
the other side of the pipe when its not computing... Therefore, I
can't have a Sage sub-process that I'm using in the notebook that is
also able to communicate with other processes as I can't
asynchronously receive data (or get timing interrupts). I've gotten
around this in the past by using threads as it was the only choice I
had.

-glenn

William Stein

unread,
Jul 22, 2009, 5:23:27 PM7/22/09
to sage-...@googlegroups.com
Exactly right.

> The "pexpect-controlled subprocess" is where Sage itself is.

Yes.

> The issue I was raising was with the latter (subprocess) although I
> now think the server implementation might also be an issue.
>
> when twisted spawns a sub-process, typically one registers a callback
> to handle information from that sub-process.

Twisted does not spawn the subprocess that does the computations.
The notebook server spawns that subprocess and controls it using pexpect.

>  There should be no
> reason to check for input.  Generally a callback for signals is also
> registered to sense the sub-process dying, the pipe being disconnected
> etc.  Unless I misunderstand (very possible), you're "polling" the set
> of spawned Sage sub-processes for data.

Correct.

> While that doesn't sound good either, thats not the problem I was
> describing.
>
> My primary problem is that the Sage subprocess is blocking forever on
> the other side of the pipe when its not computing... Therefore, I
> can't have a Sage sub-process that I'm using in the notebook that is
> also able to communicate with other processes as I can't
> asynchronously receive data (or get timing interrupts).  I've gotten
> around this in the past by using threads as it was the only choice I
> had.

Thanks for the clarification. Since I don't really understand the
problem, without further clarification I don't think it will get fixed
in the near future.

William

ghtdak

unread,
Jul 22, 2009, 7:41:46 PM7/22/09
to sage-devel

>
> > My primary problem is that the Sage subprocess is blocking forever on
> > the other side of the pipe when its not computing... Therefore, I
> > can't have a Sage sub-process that I'm using in the notebook that is
> > also able to communicate with other processes as I can't
> > asynchronously receive data (or get timing interrupts).  I've gotten
> > around this in the past by using threads as it was the only choice I
> > had.
>
> Thanks for the clarification.  Since I don't really understand the
> problem, without further clarification I don't think it will get fixed
> in the near future.

Basically, the problem is that the Sage sub-process loses control when
its done servicing a request from the server. Instead of entering an
event loop, it blocks on the pipe.

The alternative would be to return after any request to an event
loop. Clearly, the primary requestor would be the notebook server,
but if you had a general event loop, the user could register any
number of other asynchronous sources or timers to respond to.

The adjustment I propose is in the sub-process (and probably would
extend this to the notebook but I'll withhold that for the time
being). Instead of initializing and entering the read-process-write
cycle, where read blocks waiting for a request, the sub-process would
initialize, register a callback to handle requests from the server
pipe, and enter the event loop. When a request came in, the primary
sage callback would "do the right thing" and return the result, hence
returning to the event loop.

Now, if a user wanted to handle other events or timers, they would
simply add those asynchronous systems to the event loop using whatever
means were provided. This is, essentially, the purpose of the Twisted
Reactor.

This sub-process adjustment could (maybe) be done initially without
changing anything on the notebook itself. The pipe callback would
receive the string, call a slightly modified sub-process method, get
the result and return it. Even if there are pexpect elements in the
sub-process, the callback could feed that.

Of course, once you put an event loop in the sub-process, much of
pexpect probably becomes unnecessary. Seems to me that Twisted could
do the spawning of the twisted sub-process and just shuffle strings
across. I don't really know pexpect but my guess is the things it
handles are also handled by twisted (although I don't actually know).
Its doubtful that there is sufficient complexity that pickle couldn't
handle the marshalling of data across the pipe.

Once this change was made, you'd have a full infrastructure with which
to build much more flexible applications yet still have the notebook
interface. This would also facilitate building distributed
computation engines, data collectors etc.

Furthermore, as things evolved, truly dynamic AJAX could be built
because the underlying Sage process could be asynchronously receiving
data, talking with other Sage processes, periodically polling other
servers (e.g. yahoo finance)

It seems this would be a fairly easy task and have substantial
payoff. I would have done it myself, but got bogged down when looking
through the pexpect code and the strings shoved over the pipe. I
figured that a few hours with the author would save a lot of time but
the runaway stack interfered.

-glenn

>
> William
>
>
>
>
>
> > -glenn
>

Dorian Raymer

unread,
Jul 22, 2009, 11:00:34 PM7/22/09
to sage-...@googlegroups.com, codenod...@googlegroups.com
Hi Glenn,

On Wed, Jul 22, 2009 at 4:41 PM, ghtdak <gl...@tarbox.org> wrote:


>
> > My primary problem is that the Sage subprocess is blocking forever on
> > the other side of the pipe when its not computing... Therefore, I
> > can't have a Sage sub-process that I'm using in the notebook that is
> > also able to communicate with other processes as I can't
> > asynchronously receive data (or get timing interrupts).  I've gotten
> > around this in the past by using threads as it was the only choice I
> > had.
>
> Thanks for the clarification.  Since I don't really understand the
> problem, without further clarification I don't think it will get fixed
> in the near future.

Basically, the problem is that the Sage sub-process loses control when
its done servicing a request from the server.  Instead of entering an
event loop, it blocks on the pipe.

The alternative would be to return after any request to an event
loop.  Clearly, the primary requester would be the notebook server,

The essence of what you have described is architecture of Codenode. With out getting too much into a side by side comparison, this is how it works using our terminology:

An Engine is a computation process (like what you call a Sage sub-process).
An Engine contains two parts: First part, an object representing the Python (or Sage) interpreter, with methods like evaluate, and tab complete. Second part, a simple RPC server that adapts the interpreter object to a network transport; the current working implementation is an XML-RPC server.

The Interpreter object knows nothing about the RPC server, and the RPC server only has to map it's rpc methods to the interpreter object methods.

A Backend server creates and manages Engines.
The Backend server is a Twisted event loop.
It spawns, monitors, and terminates Engine Processes.
It relays all notebook AJAX requests to the appropriate engine via an XML-RPC client (which is non-blocking).

Since the Backend is a Twisted event loop, it handles all process spawning/monitoring, AJAX requests, and XML-RPC requests totally asynchronously.

It has no dependencies on the interpreter libraries that it runs, and does not need to understand the data going in and out of the engines.

Have a look at all the pieces currently work here.
I am currently improving this Backend concept and have a completely new iteration of code (still in development) that should be a bit more digestible (it's cleaner, among other improvements).
 

Once this change was made, you'd have a full infrastructure with which
to build much more flexible applications yet still have the  notebook
interface.  This would also facilitate building distributed
computation engines, data collectors etc.

Yes! Exactly. Combined with the third element (called the Frontend web application server who's most important job is storing users notebook data in a database), you have distributed notebook computation. We have successfully done deployments where the Frontend runs on one server, and the Backend on another server, usually in another City!
When the next piece of development is finished, a Frontend will be capable of supporting multiple Backends, again, no matter where they are on the network.
 

Furthermore, as things evolved, truly dynamic AJAX could be built
because the underlying Sage process could be asynchronously receiving
data, talking with other Sage processes, periodically polling other
servers (e.g. yahoo finance)

This is interesting, and sounds like something different than just a "Computation Engine".
I interpret this as a formalization of some kind of information base into a service accessible by the the notebook Engine processes.

The simple limit of this is the formalization of how a processes OS Environment Variables are set. For example, I could want to set up a environment supporting scientific analysis, so I would want to configure numpy and scipy to be in my Engine process python path.
This environment could also facilitate access to some kind of data set, like one produced by an Astrophysicist's galaxy simulation. The data set could be a file, a set of files, etc. Or, in the large limit, and coming back to your example, it could be a network service explicitly supported by the Backend, with accessibility permissions from each engine process explicitly configurable.

I'm not sure it makes sense, architecturally, for an Engine process to run an event loop, because it's principal purpose is to execute arbitrary code, and that must be generally assumed to be a blocking procedure. Maybe you can elaborate more on this idea; it's very interesting.

Take a look at codenode!

-Dorian