Proposal: move SageNB back to Sage

[Jeroen Demeyer]

Hello all,

I propose to make SageNB no longer a separate package but to move it
back into the Sage git tree. For purposes of installation and use of
SageNB, it will still be a separate Python package, but the sources will
be in $SAGE_ROOT/src/sagenb instead of a separate git repo. The changes
to the Sage build system to support this move will be minimal.

The reason is that SageNB is truly in maintenance mode currently. Making
new SageNB releases regularly to fix things is a burden for the SageNB
release manager Karl-Dieter Crisman. On #14840 [1], he said "the sooner
sagenb gets back in Sage proper, the better!"

The original reason to split SageNB from Sage was to enable quick
development. That worked for a while, but now that development has
stalled, this reason no longer applies. A secondary reason was to make
SageNB truly independent from Sage, but that also never happened. So I
see no reason to keep SageNB split from Sage currently.

I know this is a controversial proposal, but it will certainly be easier
to maintain SageNB this way. I also want to stress that this is
orthogonal to any future deprecation or removal of SageNB: we can still
do that from the Sage git tree.

Jeroen.



[1] http://trac.sagemath.org/ticket/14840#comment:58

[Sébastien Labbé]

I propose to make SageNB no longer a separate package but to move it
back into the Sage git tree.

+1

I agree.

[Kwankyu Lee]

+1

[Dima Pasechnik]

+1

[mmarco]

+1

[Jonathan Gutow]

+1

Things really have stalled.  My fixes to the code to handle properly putting the notebook behind a proxy since the server mechanism in SageNB is not robust have languished for over a year.

Jonathan

                        Dr. Jonathan H. Gutow
Chemistry Department                                 gu...@uwosh.edu
UW-Oshkosh                                                Office:920-424-1326
800 Algoma Boulevard                                 FAX:920-424-2042
Oshkosh, WI 54901
                http://www.uwosh.edu/facstaff/gutow/

--
You received this message because you are subscribed to the Google Groups "sage-notebook" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-noteboo...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-notebook.
For more options, visit https://groups.google.com/d/optout.

[Dima Pasechnik]

Who does have commit access to https://github.com/sagemath/sagenb ? 

(not me).

Karl is obviously overwhelmed with other things.

If I had this access I could have reviewed at least some of these tickets

(we still would want to keep upstream on github, right?)

Dima

[William Stein]

On Fri, Apr 15, 2016 at 7:05 AM, Dima Pasechnik <dim...@gmail.com> wrote:
> Who does have commit access to https://github.com/sagemath/sagenb ?
> (not me).
> Karl is obviously overwhelmed with other things.
> If I had this access I could have reviewed at least some of these tickets
> (we still would want to keep upstream on github, right?)

I've added you. There are 9 people who can commit (some long missing)
who can commit, including you, which I think you can see here:
https://github.com/orgs/sagemath/teams/sage-notebook

Anybody who wants to be added, say so and I (or maybe Dima) can add them.

William

--
William (http://wstein.org)

[William Stein]

-1 to moving the sagenb code back into the sage git repo. The day I
removed sagenb from the sage git repo was, for me, a very happy day.

But +1 for having people who care have the ability to push to the repo
on github and generally take over maintenance.

--
William (http://wstein.org)

[Jeroen Demeyer]

On 2016-04-15 16:15, William Stein wrote:
> But +1 for having people who care have the ability to push to the repo
> on github and generally take over maintenance.

Any volunteers? If there is indeed somebody who is seriously willing to
manage SageNB, that would be a good solution. However, by lack of such a
person, I think the best solution is to use the resources of Sage to
manage SageNB.

[John H Palmieri]

+1.

For those who disagree, please recognize that the current situation is unmanageable: there are interdependencies between sage and sagenb, so certain changes in sage require changes in sagenb. Getting those changes done in sagenb is difficult, because sagenb is not really actively maintained. So if you disagree, please suggest a concrete alternative, because (as Jeroen says) the status quo is not working.

  John

[Dima Pasechnik]

On Friday, April 15, 2016 at 5:06:34 PM UTC+1, John H Palmieri wrote:

+1.

For those who disagree, please recognize that the current situation is unmanageable: there are interdependencies between sage and sagenb, so certain changes in sage require changes in sagenb. Getting those changes done in sagenb is difficult, because sagenb is not really actively maintained.

Could you perhaps point out at these?

Are there any issues/pull requests on sagenb related to such changes?

[John H Palmieri]

On Friday, April 15, 2016 at 10:40:25 AM UTC-7, Dima Pasechnik wrote:

On Friday, April 15, 2016 at 5:06:34 PM UTC+1, John H Palmieri wrote:

+1.

For those who disagree, please recognize that the current situation is unmanageable: there are interdependencies between sage and sagenb, so certain changes in sage require changes in sagenb. Getting those changes done in sagenb is difficult, because sagenb is not really actively maintained.

Could you perhaps point out at these?

Are there any issues/pull requests on sagenb related to such changes?

I don't know if there are any right now, but I think that many of the recent changes in sagenb have come because there were changes in the Sage library. Maybe "difficult" is not the right word, but this adds an artificial layer of complexity: instead of making a few possibly trivial changes in the relevant sagenb files within Sage, for example when the location of the built documentation moved, it takes a pull request, someone to handle that request, someone to put together a new sagenb release, etc. It often (almost always?) ends up being the same people working on the Sage trac ticket and the sagenb pull request, so we're adding some inefficiencies to the system. If we could deal with the issues with just a trac ticket, that would be better.
 

[Jeroen Demeyer]

On 2016-04-15 22:06, John H Palmieri wrote:
> I don't know if there are any right now, but I think that many of the
> recent changes in sagenb have come because there were changes in the
> Sage library. Maybe "difficult" is not the right word, but this adds an
> artificial layer of complexity: instead of making a few possibly trivial
> changes in the relevant sagenb files within Sage, for example when the
> location of the built documentation moved, it takes a pull request,
> someone to handle that request, someone to put together a new sagenb
> release, etc. It often (almost always?) ends up being the same people
> working on the Sage trac ticket and the sagenb pull request, so we're
> adding some inefficiencies to the system. If we could deal with the
> issues with just a trac ticket, that would be better.

+1, this is an excellent summary of the problem.

[Dima Pasechnik]

On Wednesday, November 23, 2016 at 6:42:40 PM UTC, Jack Dyson wrote:

Hi everyone,

With the greatest respect, I disagree strongly - chopping and changing the notebook this way leads to a lot of instability in the code and confusion for anyone who wants to get into developing on sagenb projects.

Added to the fact that none of us would have time to document changes in detail causes new contributions stagnate, which wastes effort and randomizes progress.

Actually the functionality of the current notebook is good, the look and ui is very dated and as many are aware, a bit on the unpolished side.

Remembering Samuel Ainsworth's really good work a few year's back, I would like to ask why was that system not developed and integrated into the local sagemath distributable?

From the trials he conducted in 2012 it ran well on Sage 5.3 and was in my opinion a decent step forward. I'll post this to a separate question as I wanted to explore the possibilities of getting that up and running again, even if only for private use here.

It doesn't seem to connect to sage 7.3 so I wanted to see if anyone knew why ?

I understand that opinions on usability of https://github.com/sagemath/sagenb/tree/newui

diverge.  (and with the breakneck speed javascript

frameworks are developed, one may ask whether something written in 2012 is still a great idea)

You ask why sagenb is not developed further.

1) the sagenb's design is really dated, and jupyter notebook seems a better (and much better

supported) alternative. 

2) Another actively developed alternative is SMCs notebook, which can be run

locally.

 

Very best to all
Jack

[Jan Groenewald]

Hi

On 23 November 2016 at 22:05, Dima Pasechnik <dim...@gmail.com> wrote:

I understand that opinions on usability of https://github.com/sagemath/sagenb/tree/newui

diverge.  (and with the breakneck speed javascript

frameworks are developed, one may ask whether something written in 2012 is still a great idea)

You ask why sagenb is not developed further.

1) the sagenb's design is really dated, and jupyter notebook seems a better (and much better

supported) alternative. 

2) Another actively developed alternative is SMCs notebook, which can be run

locally. 

In Africa we have connectivity and power issues, which requires a local notebook.

Would jupyter be a possible replacement for sagenb if sagenb is discontinued from active development?  Is it already on a stand-alone sagemath install?

Regards,

Jan

--

  .~.
  /V\     Jan Groenewald
 /( )\    www.aims.ac.za
 ^^-^^ 

[Dima Pasechnik]

On Wednesday, November 23, 2016 at 8:11:27 PM UTC, Jan Groenewald wrote:

Hi

On 23 November 2016 at 22:05, Dima Pasechnik <dim...@gmail.com> wrote:

I understand that opinions on usability of https://github.com/sagemath/sagenb/tree/newui

diverge.  (and with the breakneck speed javascript

frameworks are developed, one may ask whether something written in 2012 is still a great idea)

You ask why sagenb is not developed further.

1) the sagenb's design is really dated, and jupyter notebook seems a better (and much better

supported) alternative. 

2) Another actively developed alternative is SMCs notebook, which can be run

locally. 

In Africa we have connectivity and power issues, which requires a local notebook.

Would jupyter be a possible replacement for sagenb if sagenb is discontinued from active development?

yes, it is already there:

Start sage as follows:

sage --notebook='jupyter'

 

  Is it already on a stand-alone sagemath install?

yes. For instance, even a docker container.

[William Stein]

On Wed, Nov 23, 2016 at 12:11 PM, Jan Groenewald <j...@aims.ac.za> wrote:
>
> Hi
>
> On 23 November 2016 at 22:05, Dima Pasechnik <dim...@gmail.com> wrote:
>>
>>
>> I understand that opinions on usability of https://github.com/sagemath/sagenb/tree/newui
>> diverge. (and with the breakneck speed javascript
>> frameworks are developed, one may ask whether something written in 2012 is still a great idea)
>>
>> You ask why sagenb is not developed further.
>> 1) the sagenb's design is really dated, and jupyter notebook seems a better (and much better
>> supported) alternative.
>> 2) Another actively developed alternative is SMCs notebook, which can be run
>> locally.
>
>
> In Africa we have connectivity and power issues, which requires a local notebook.

SMC **can be run locally**:

1. Install docker.

2. Type this (all one line)

docker run --name=smc -v ~/smc:/projects -p 80:80 -p 443:443
sagemathinc/sagemathcloud

Now you're running SMC locally (visit http://localhost). See
https://github.com/sagemathinc/smc/blob/master/src/dev/docker/README.md



--
William (http://wstein.org)

[Jan Groenewald]

Hi

We don't want to install docker. We want to install a debian package.

Regards,

Jan

[William Stein]

OK, then it's not possible to use SMC, since we do not provide it as a
Debian package. Sorry,

-- William

[Dima Pasechnik]

On Wednesday, November 23, 2016 at 9:29:44 PM UTC, Jan Groenewald wrote:

Hi

On 23 November 2016 at 22:59, William Stein <wst...@gmail.com> wrote:

On Wed, Nov 23, 2016 at 12:11 PM, Jan Groenewald <j...@aims.ac.za> wrote:
>
> Hi
>
> On 23 November 2016 at 22:05, Dima Pasechnik <dim...@gmail.com> wrote:
>>
>>
>> I understand that opinions on usability of https://github.com/sagemath/sagenb/tree/newui
>> diverge.  (and with the breakneck speed javascript
>> frameworks are developed, one may ask whether something written in 2012 is still a great idea)
>>
>> You ask why sagenb is not developed further.
>> 1) the sagenb's design is really dated, and jupyter notebook seems a better (and much better
>> supported) alternative.
>> 2) Another actively developed alternative is SMCs notebook, which can be run
>> locally.
>
>
> In Africa we have connectivity and power issues, which requires a local notebook.


SMC **can be run locally**:

1. Install docker.

2. Type this (all one line)

    docker run --name=smc -v ~/smc:/projects -p 80:80 -p 443:443
sagemathinc/sagemathcloud

Now you're running SMC locally (visit http://localhost).  See
   https://github.com/sagemathinc/smc/blob/master/src/dev/docker/README.md

We don't want to install docker. We want to install a debian package.

but sure, do it!

https://packages.debian.org/sid/docker.io

[Jack Dyson]

On Friday, April 15, 2016 at 10:44:21 AM UTC+2, Jeroen Demeyer wrote:

Hi everyone,

With the greatest respect, I disagree strongly - chopping and changing the notebook this way leads to a lot of instability in the code and confusion for anyone who wants to get into developing on sagenb projects.

Added to the fact that none of us would have time to document changes in detail causes new contributions stagnate, which wastes effort and randomizes progress.

Actually the functionality of the current notebook is good, the look and ui is very dated and as many are aware, a bit on the unpolished side.

Remembering Samuel Ainsworth's really good work a few year's back, I would like to ask why was that system not developed and integrated into the local sagemath distributable?

From the trials he conducted in 2012 it ran well on Sage 5.3 and was in my opinion a decent step forward. I'll post this to a separate question as I wanted to explore the possibilities of getting that up and running again, even if only for private use here.

It doesn't seem to connect to sage 7.3 so I wanted to see if anyone knew why ?

[Jack Dyson]

Hi Dima,

Thankyou for your quick reply - I appreciate what you said about development of sagenb as a whole, and actually I do see the point.

Logically therefore, as you indicate, iPython is a good alternative. Unfortunately, it is not fully compatible with sagemath, for example R doesn't access it properly in all respects or the 3D graphics formatting was of last time I checked.

The current sagenb is excellent on both those things even if the structure is dated: I want to just make clear it does work extremely well and that's what a user spends 80% of their time doing.

It is great that SMC's notebook is actually available in some way: and here I feel that a decision needs to be made:

therefore one of these should answer well:

1)  "phase out" sagenb completely and integrate SMC into the sage distributable quickly so that it is the default option - as was hinted at a few years ago. It was in fact stated then that developing sagenb was a "waste of developer resources"

2) dump jmol and make iPython the default notebook and address its remaining incompatibilities with packages (which would need a rewrite of the sagemath API so it is no longer view model dependent I suppose)

3) create a new independent team around sagenb : decide what we need and continue Sam's work in whatever framework we want, the functional model of the code made independent from the implementation

The corollary is clear: not all sagemath users want to specialize to SMC for various reasons, and the sagemath userbase (universities for example) as a distributable is going to be under threat without a viable modern local interface just like Maxima was years back before wxmaxima.

I think William wrote something below about installing a docker and getting SMC up, very welcome indeed : I'll give that a go for starters.

Best from here,

Jack

[William Stein]

For a more local interface, also consider the possibility of building
on nteract:

https://github.com/nteract/nteract

It's basically an Electron app rewrite from scratch of Jupyter.

[Jack Dyson]

Right cheers for that William! will give it a go, hopefully we'll have it as standard at some point soon.

Best from here,

J

[William Stein]

On Wed, Nov 23, 2016 at 3:02 PM, Jack Dyson <jackd...@gmail.com> wrote:
> Right cheers for that William! will give it a go, hopefully we'll have it as
> standard at some point soon.

If you do want to work on that, some thoughts:

- There is a "painful" dependency on RethinkDB, which is a C++
program that takes a while to compile. In a few months I predict this
will be replaced by either sqlite or postgreSQL, which are both much
less painful dependencies...

Actually, I don't have much in the way of other thoughts, but don't
hesitate to ask.

[Jack Dyson]

Thanks I will! enough there to think about already !
J
PS nteract looks cool

> You received this message because you are subscribed to a topic in the Google Groups "sage-notebook" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-notebook/t11JSxxCgpw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to sage-noteboo...@googlegroups.com.

[Jack Dyson]

Hi William,
installed the docker version - exciting stuff - just brilliant!
thanks. We need this "sagewide" asap really. Be looking into it as I
understand more.
small question when you have a moment:
docker starts correctly, everything works-I have SMC locally but I
can't get in sage dedicated modes like R or shell. Some of them work
like maxima, python, gap. If I create a terminal in my project, I can
start R. The code is there - just not accessible from a sage file in
the project: any idea ?
best,
J

PS : an example:

%r
1+1

Error in lines 1-1
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/smc_sagews/sage_server.py",
line 968, in execute
exec compile(block+'\n', '', 'single') in namespace, locals
File "", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/smc_sagews/sage_server.py",
line 1009, in execute_with_code_decorators
code = code_decorator(code)
File "/usr/local/lib/python2.7/dist-packages/smc_sagews/sage_salvus.py",
line 2117, in r
r.jupyter_kernel = jupyter("ir")
File "/usr/local/lib/python2.7/dist-packages/smc_sagews/sage_jupyter.py",
line 31, in __call__
return _jkmagic(kernel_name, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/smc_sagews/sage_jupyter.py",
line 111, in _jkmagic
km, kc = jupyter_client.manager.start_new_kernel(kernel_name = kernel_name)
File "/usr/lib/sagemath/local/lib/python2.7/site-packages/jupyter_client/manager.py",
line 429, in start_new_kernel
km.start_kernel(**kwargs)
File "/usr/lib/sagemath/local/lib/python2.7/site-packages/jupyter_client/manager.py",
line 230, in start_kernel
kernel_cmd = self.format_kernel_cmd(extra_arguments=extra_arguments)
File "/usr/lib/sagemath/local/lib/python2.7/site-packages/jupyter_client/manager.py",
line 170, in format_kernel_cmd
cmd = self.kernel_spec.argv + extra_arguments
File "/usr/lib/sagemath/local/lib/python2.7/site-packages/jupyter_client/manager.py",
line 82, in kernel_spec
self._kernel_spec =
self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
File "/usr/lib/sagemath/local/lib/python2.7/site-packages/jupyter_client/kernelspec.py",
line 175, in get_kernel_spec
raise NoSuchKernel(kernel_name)
NoSuchKernel: No such kernel named ir

[Harald Schilly]

On Thu, Nov 24, 2016 at 10:38 AM, Jack Dyson <jackd...@gmail.com> wrote:

any idea ?

​The error is: NoSuchKernel: No such kernel named ir

​

That means, that the "ir" kernel for R (and many others) aren't available in that image. I mean, you're running a local version of smc with some minimal subset of the necessary stuff on the back-end -- that's it. In total there is much much more, but that's outside the scope of smc. ​

Here is a most likely working copy of that kernel:

https://github.com/sagemathinc/smc/tree/master/src/smc-build/jupyter-kernels/ir

-- h

[Jack Dyson]

ok cheers Harald !

[William Stein]

On Thu, Nov 24, 2016 at 1:38 AM, Jack Dyson <jackd...@gmail.com> wrote:
> Hi William,
> installed the docker version - exciting stuff - just brilliant!
> thanks. We need this "sagewide" asap really. Be looking into it as I
> understand more.
> small question when you have a moment:
> docker starts correctly, everything works-I have SMC locally but I
> can't get in sage dedicated modes like R or shell. Some of them work
> like maxima, python, gap. If I create a terminal in my project, I can
> start R. The code is there - just not accessible from a sage file in
> the project: any idea ?

Unfortunately, we haven't added the code to support those Jupyter
kernels to the docker image, which is why it doesn't work yet. I've
made an issue for this:

https://github.com/sagemathinc/smc/issues/1304

[William Stein]

Emmanuel Charpentier wrote:
> 2) In contrast, the Sage notebook, while quite advanced *for its time*, has
> remained a Sage-only interface. Yes, you can use a number of other tools
> with i, *as long as they are known by Sage*.
> This simultaneously enhances and limits its utility. For example, one can
> use the Sage notebook for Maxima (and even use Maxima, R or gap cells in a
> notebook mainly containing Sage code). But only with Sege-suported version
> of Maxima, R or gap. Don't even think of having Lisp or (heavens !) Fortran
> cells...

Sagenb has support for "Fortran cells" since when Josh Kantor and I
added it in 2007... This support integrated with f2py and numpy, to
make fortran functions available automatically. Sagenb also has
%lisp cells.

> 3) The same is true, with both aggravation and mitigation, for SMC and its
> related tools. I like the idea of a \LaTeX editor that supports sage (or
> other) snippets. But, again, *only* with the tools integrated in Sage...

This is wrong. 1) SMC Sage worksheets support using all Jupyter
kernels via the jupyter('kernel name') command, and (2) SMC also
embeds Jupyter notebooks as one of the editor types.

> In contrast, emacs+SageTeX(+R+knitr) give me a better service...

SMC is mainly a collaborative remote website, whereas emacs + ... is a
local application -- apples to oranges.

> Similarly, while the current Jupyter notebook won't display the graphs
> produced by R,

Jupyter notebooks using the R kernel have been able to display graphs
produced by R for years, and do so pretty nicely...

https://cloud.sagemath.com/projects/4a5f0542-5873-4eed-a85c-a18c706e8bcd/files/support/2016-11-24-r-graph.ipynb

> The SMC, however, will probably succeed for
> its initial goal (serving a large group of users using exclusively Sage and
> its federated tools via the Web),

That is not our goal. A significant proportion of users of SMC don't
use Sage at all -- they use Latex or Jupyter notebooks or terminals...
SMC just happens to make it easy to use Sage, among other things.

> for which it seems probably unbeatable ;
> but for people needing a local installation, or needing an interface to
> tools not (yet) integrated with Sage, Jupyter is simply better.

SMC is to Jupyter like Sage is to GAP or Pari. Just as Sage doesn't
compete with GAP, SMC isn't competing with Jupyter notebooks.


--
William (http://wstein.org)

[Jack Dyson]

Hi William

cheers for getting back on that - what Harald told me this morning led
me to this video you did:

https://www.youtube.com/watch?v=GOuy07Kift4

which should sort me out hopefully : I'll install all of it on a vm

If you do get the SMC docker thing completely up *for single users* -
it is one distribution route, could even be scripted.

The only if or but would be how to use docker containers to access the
host filesystem and tinker with the Sage components (remember a while
back I used the system R because it allows packages to run)

just a thought.

Best

J

[kcrisman]

Just listening in - curious what the proposed move would do for sagenb server installs?  Is there a realistic porting option for those?  I assume that authentication etc. would not just port at all to either SMC or Jupyterhub, assuming the latter is even usable (I don't know the answer to that).

[Jack Dyson]

Hi Harald, just installed all of smc onto Ubuntu 15.10 from the
sagemathinc repo as shown in the instructions:

got a problem and I need some direction if you might say a couple of words:

1) the tests pass upto the smc-hub : rethinkdb times out
2) no url from the info.py : the log confirms rethinkdb gremlins :

2016-11-25T20:30:14.075Z - debug: RethinkDB._connect: error connecting
to localhost -- {"name":"ReqlDriverError","msg":"Could not connect to
localhost:28015.\nconnect ECONNREFUSED
127.0.0.1:28015","message":"Could not connect to
localhost:28015.\nconnect ECONNREFUSED 127.0.0.1:28015"}

3) smc server starts fine - just no web url because I think rethinkdb
is not connected

I installed the latest rethinkdb, with nodejs 5.12 as recommended

cheers if you can help,

Jack

On 24 November 2016 at 11:49, Harald Schilly <har...@schil.ly> wrote:
>