SageNB -> Jupyter conversion needs testers

[Jeroen Demeyer]

Hello,

At https://trac.sagemath.org/ticket/22433 there is a ticket implementing
a move from SageNB towards Jupyter. It does conversion of SageNB
notebooks to Jupyter, hopefully with little loss of functionality (but I
know it's not 100% perfect and it never will be). If you are currently
using SageNB, please test this ticket and tell us your experience.

I propose that the SageMath version after 7.6 will be 8.0 and that this
is merged in 8.0.

Trac #22433 is really a merge of 3 tickets:

* Trac #19740: the new default notebook (what you get when running "sage
-n") is sagenb_export, which allows exporting SageNB notebooks to
Jupyter notebooks. It also provides buttons to run either the Jupyter
notebook or the old SageNB.

* Trac #21267: port SageNB interacts to Jupyter IPyWidgets. This tickets
aims to keep old SageNB interacts working within Jupyter. This mainly
means defining functions like slider() returning a Jupyter widget
instead of a SageNB control.

* Trac #22432: this updates the Jupyter notebook to the latest stable
release. It is probably not strictly required, it's just to test that
stuff works with that version.

If you encounter trouble while testing, please put a comment in the
corresponding Trac ticket.

A lot of this work was done for the OpenDreamKit grant, in particular
D4.5, see https://github.com/OpenDreamKit/OpenDreamKit/issues/94
Credit also goes to Volker Braun for developing sagenb_export.


Thanks,
Jeroen.

[Volker Braun]

Yay!

[kcrisman]

At https://trac.sagemath.org/ticket/22433 there is a ticket implementing
a move from SageNB towards Jupyter. It does conversion of SageNB
notebooks to Jupyter, hopefully with little loss of functionality (but I
know it's not 100% perfect and it never will be). If you are currently
using SageNB, please test this ticket and tell us your experience.

I propose that the SageMath version after 7.6 will be 8.0 and that this
is merged in 8.0.

Thank you SO much for your diligent work on this.  I have had virtually no Sage development time for a couple years (largely due to involvement with Sage-related things like MBX and ask.sagemath) but really appreciate everyone who is helping make this transition robust and not abandonware for so many users (such as many reporting on ask.sagemath).  Hopefully a good spring break project for me to do some testing of my sizable notebook collection.  I think that it's reasonable to make 8.0 more or less about this, with sufficient testing.

My big question is what would happen in the server situation?    Has anyone tested what would be different for those running sagenb servers, from launch to running?   One of the disappointments is that there still doesn't seem to be an easy-to-install-and-use Jupyter equivalent for the multi-user server which has served many people so well, even if imperfectly.  Perhaps Jupyterhub is now ready for this - but I imagine it would be challenging to have a migration from sagenb to that?

[Enrique Artal]

We have trying in my university some solutions usingy jupyterhub; we have some ideas about sharing and we still do not know how safe it is. 

[kcrisman]

This is now default in the Sage 8.0 beta series, and we strongly welcome testers!  For a limited time only I have posted a binary of the app bundle at 

http://www.math.gordon.edu/~kcrisman/sage-8.0.beta0-OSX_10.11.6-x86_64.app.dmg

This should work on both Mac 10.11 and 10.12.   For bandwidth reasons I would prefer if those with the ability simply upgrade and build from scratch, but testing this app is also necessary.

PLEASE TRY THIS and report back if you are at all interested in making sure that Sage 8.0 doesn't come with nasty surprises for long-time sagenb users.  Preliminary testing makes it seem as if we are okay, but it would be very helpful to have feedback about better/more informative messages for end users.

Note if you haven't downloaded a new one for a while: because it will first have to patch some names to make sure all paths are correct, the first launch will take a LONG time.

[dimpase]

it does not work for me, or at least I must be doing something wrong.

I get to the page listing the notebooks to convert, and then, if I click on a notebook, I get a weird page asking for a password or a token.

Password or token:

 Log in

Token authentication is enabled. You need to open the notebook server with its first-time login token in the URL, or enable a password in order to gain access. The command:

jupyter notebook list

will show you the URLs of running servers with their tokens, which you can copy and paste into your browser. For example:

Currently running servers:
http://localhost:8888/?token=c8de56fa... :: /Users/you/notebooks

Or you can paste just the token value into the password field on this page.

Cookies are required for authenticated access to notebooks.

What I see above doesn't look as a valid URL, and then WTF is the token I can paste instead?

I tried c8de56fa, it didn't work. I tried clicking on the URL, it brings me back to the page to choose

the notebooks...

 Arrgh... Give me my console any time...

[Dima Pasechnik]

anyhow, this process, clicking on these old notebooks, does provide copies of converted to ipython sagenb notebooks,

but the whole process is very stressful at the moment, I think. This popup with password prompt really should not be there...

On Tuesday, April 11, 2017 at 9:44:03 PM UTC+1, Dima Pasechnik wrote:

it does not work for me, or at least I must be doing something wrong.

I get to the page listing the notebooks to convert, and then, if I click on a notebook, I get a weird page asking for a password or a token.

Password or token:

 Log in

Token authentication is enabled. You need to open the notebook server with its first-time login token in the URL, or enable a password in order to gain access. The command:

jupyter notebook list

will show you the URLs of running servers with their tokens, which you can copy and paste into your browser. For example:

Currently running servers:
http://localhost:8888/?token=c8de56fa... :: /Users/you/notebooks

Or you can paste just the token value into the password field on this page.

Cookies are required for authenticated access to notebooks.

What I see above doesn't look as a valid URL, and then WTF is the token I can paste instead?

I tried c8de56fa, it didn't work. I tried clicking on the URL, it brings me back to the page to choose

the notebooks...

 Arrgh... Give me my console any time...

a 

[Jeroen Demeyer]

On 2017-04-11 22:43, dimpase wrote:
> I get to the page listing the notebooks to convert, and then, if I click
> on a notebook, I get a weird page asking for a password or a token.

Thanks for the feedback. The big issue is that this problem is
non-deterministic. Sometimes, you get this password prompt, but not
always. It completely boggles my mind...

[Dima Pasechnik]

here is what I see on the console:

$ sage -n

┌────────────────────────────────────────────────────────────────────┐

│ SageMath version 8.0.beta1, Release Date: 2017-04-06               │

│ Type "notebook()" for the browser-based notebook interface.        │

│ Type "help()" for help.                                            │

└────────────────────────────────────────────────────────────────────┘

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓

┃ Warning: this is a prerelease version, and it may be unstable.     ┃

┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

Please wait while the SageNB export server starts...

[W 09:12:41.199 NotebookApp] server_extensions is deprecated, use nbserver_extensions

[I 09:12:41.201 NotebookApp] Using MathJax: nbextensions/mathjax/MathJax.js

[I 09:12:41.458 NotebookApp] Serving notebooks from local directory: /home/dima/tmp

[I 09:12:41.458 NotebookApp] 0 active kernels 

[I 09:12:41.458 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/?token=fd22b498813a324306a40c338c8aeeccc070b7f4fbdd4922

[I 09:12:41.458 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

[C 09:12:41.460 NotebookApp] 

    

    Copy/paste this URL into your browser when you connect for the first time,

    to login with a token:

        http://localhost:8888/?token=fd22b498813a324306a40c338c8aeeccc070b7f4fbdd4922

[12684:12684:0412/091241.637533:ERROR:child_thread_impl.cc(762)] Request for unknown Channel-associated interface: ui::mojom::GpuMain

Created new window in existing browser session.

[W 09:12:41.746 NotebookApp] 404 GET /favicon.ico (::1) 22.43ms referer=http://localhost:8888/sagenb?token=ab665726c7d12093377334a8594ece82cfcbe53707e1f3a7

('POST', HTTPServerRequest(protocol='http', host='localhost:8888', method='POST', uri='/sagenb/export', version='HTTP/1.1', remote_ip='::1', headers={'Origin': 'http://localhost:8888', 'Content-Length': '7', 'Accept-Language': 'en-US,en;q=0.8', 'Accept-Encoding': 'gzip, deflate, br', 'X-Xsrftoken': '2|4686c908|4688dfe919597975862c3eb95cd49778|1490003278', 'Host': 'localhost:8888', 'Accept': '*/*', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.21 Safari/537.36', 'Connection': 'keep-alive', 'Referer': 'http://localhost:8888/sagenb?token=ab665726c7d12093377334a8594ece82cfcbe53707e1f3a7', 'Cookie': '_xsrf=2|4686c908|4688dfe919597975862c3eb95cd49778|1490003278; username-localhost-8889="2|1:0|10:1491943472|23:username-localhost-8889|44:NzQ4NDQ3MTVhNDRjNDc4MWE1MzI5ZmRkY2ZmZTIzNTc=|e8ee9977b1bd627b0bcb667545275966b1a145018957eaa79594645133118583"; nb_session_8080=; session="MnYU0L8X1xm8mjlYHbIupc7Ntms=?username=UydhZG1pbicKcDAKLg=="', 'Content-Type': 'text/plain;charset=UTF-8'}), 'admin:4')

[I 09:12:54.666 NotebookApp] 302 GET /notebooks/sarag%20(3).ipynb (::1) 0.84ms

[1:1:0412/091254.756792:ERROR:KeyboardEventManager.cpp(424)] Not implemented reached in static bool blink::KeyboardEventManager::currentCapsLockState()

[1:1:0412/091412.086219:ERROR:KeyboardEventManager.cpp(424)] Not implemented reached in static bool blink::KeyboardEventManager::currentCapsLockState()

^C[I 09:15:05.380 NotebookApp] interrupted

Serving notebooks from local directory: /home/dima/tmp

0 active kernels 

The Jupyter Notebook is running at: http://localhost:8888/?token=fd22b498813a324306a40c338c8aeeccc070b7f4fbdd4922

Shutdown this notebook server (y/[n])? n

So far I get this password browser popup all the time.

[kcrisman]

On Tuesday, April 11, 2017 at 10:04:44 AM UTC-4, kcrisman wrote:

This is now default in the Sage 8.0 beta series, and we strongly welcome testers!  For a limited time only I have posted a binary of the app bundle at 

This should now be available at

http://files.sagemath.org/osx/intel/development/index.html

and shortly mirrors will be updated as well.

[Volker Braun]

The token is a security measure so no other (local) user can run arbitrary code in your notebook. It is printed during startup (see below). Usually, jupyter will launch its own browser so the token is used automatically...

$ sage -n

┌────────────────────────────────────────────────────────────────────┐

│ SageMath version 8.0.beta2, Release Date: 2017-04-12               │

│ Type "notebook()" for the browser-based notebook interface.        │

│ Type "help()" for help.                                            │

└────────────────────────────────────────────────────────────────────┘

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓

┃ Warning: this is a prerelease version, and it may be unstable.     ┃

┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

Please wait while the SageNB export server starts...

[W 16:07:47.160 NotebookApp] server_extensions is deprecated, use nbserver_extensions

[I 16:07:47.165 NotebookApp] Using MathJax: nbextensions/mathjax/MathJax.js

[I 16:07:47.197 NotebookApp] Writing notebook server cookie secret to /run/user/1006/jupyter/notebook_cookie_secret

[I 16:07:48.266 NotebookApp] Serving notebooks from local directory: /mnt/disk/home/release/Sage

[I 16:07:48.266 NotebookApp] 0 active kernels 

[I 16:07:48.266 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/?token=4b2b23550be648f42c955b4dfe2f8ab6890553699fb8b386

[I 16:07:48.266 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

[C 16:07:48.267 NotebookApp] 

    

    Copy/paste this URL into your browser when you connect for the first time,

    to login with a token:

        http://localhost:8888/?token=4b2b23550be648f42c955b4dfe2f8ab6890553699fb8b386

[Ivan Andrus]

Sorry.  I just want to be clear here.  You have the password prompt when running from the terminal, so it’s not a Mac app specific problem, right?  Or is that a Terminal that was opened the app?

-Ivan

--
You received this message because you are subscribed to the Google Groups "sage-notebook" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-noteboo...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-notebook.
For more options, visit https://groups.google.com/d/optout.

[Volker Braun]

Is it really non-deterministic?

When going from the exportsagenb webapp to the jupyter webapp we don't forward the jupyter token, isn't that the issue?

Authentication can be disabled wih

sage --notebook=jupyter --NotebookApp.token=''

but imho thats a bad idea (or at least as bad as running SageNB and/or older jupyter versions) as it doesn't defend against other local users.

[Dima Pasechnik]

Perhaps it's a good idea to provide a prompt at this point, something like "Launch the browser? (yes/no)"

[Dima Pasechnik]

On Monday, April 17, 2017 at 8:30:38 AM UTC+1, Volker Braun wrote:

Is it really non-deterministic?

the the exportsagenb webapp seems to be deterministic on my gentoo linux laptop - I always end up with a browser window with a password  which does not work. I don't see any need for it, this should not happen.

[Volker Braun]

Well at least in your case it seems that chrome is detected and launched automatically, this is IMHO preferable to a commandline prompt.

[Jeroen Demeyer]

On 2017-04-17 09:30, Volker Braun wrote:
> Is it really non-deterministic?

At some point I was seeing the token login screen every time, so I was
debugging that. After about 30 minutes of debugging (without changing
any functionality), I couldn't reproduce the token login screen anymore.

It could be a caching issue or something, but it's certainly not fully
deterministic.

[dimpase]

On Monday, April 17, 2017 at 10:12:54 AM UTC+1, Volker Braun wrote:

Well at least in your case it seems that chrome is detected and launched automatically, this is IMHO preferable to a commandline prompt.

Why would any more browser windows need to be launched after one selects a sagenb notebook to convert?

I just don't get it. 

[Volker Braun]

On Monday, April 17, 2017 at 11:17:33 AM UTC+2, Jeroen Demeyer wrote:

At some point I was seeing the token login screen every time, so I was
debugging that. After about 30 minutes of debugging (without changing
any functionality), I couldn't reproduce the token login screen anymore.

Thats a stored cookie/localstorage, after you enter the token once there is no prompt in later sessions. Delete cookies for localhost to reproduce.

[Dima Pasechnik]

Do you mean to say that sagenb notebook converter tries to do something clever with jupyter authentication, like overriding it?

And this sometimes does not work? Perhaps this can be simply dropped, i.e. let the user authenticate once, and then proceed?

  

[Jeroen Demeyer]

On 2017-04-17 12:30, Volker Braun wrote:
> Thats a stored cookie/localstorage, after you enter the token once there
> is no prompt in later sessions. Delete cookies for localhost to reproduce.

Thanks A LOT!

Your answer is correct, even though it makes no sense to me... I would
guess that a cookie is only relevant for a particular Jupyter session.
But even after shutting down and restarting the Jupyter server (which
changes the startup token), the cookie is somehow still used.

[Jeroen Demeyer]

On 2017-04-11 22:43, dimpase wrote:

> it does not work for me, or at least I must be doing something wrong.
> I get to the page listing the notebooks to convert, and then, if I click
> on a notebook, I get a weird page asking for a password or a token.

Fixed in https://trac.sagemath.org/ticket/22831

(With many thanks to Min RK)

[P Purkayastha]

Hi all,

Many thanks for this script! I have been able to convert most of my notebooks into Jupyter notebooks. Some of my observations regarding the conversion:

1. The script recognizes only sage_notebook.sagenb directories. I had a lot of notebooks in individual folders, i.e., not in ~/.sage, and having their own directory names. In such cases, I used the following procedure to list and convert the notebooks:
   
   cd <directory_containing_notebook>
   ln -s notebook_name.sagenb sage_notebook.sagenb
   ~/Installations/sage/local/bin/sagenb-export --list --dot-sage "$PWD
   
   
   
2. Some of these sagenb notebook directories had sage documentation notebooks. Not sure how and why they got in there:
   
   » ~/Installations/sage/local/bin/sagenb-export --list --dot-sage "$PWD" 

    

   Unique ID       | Notebook Name

   -------------------------------------------------------------------------------

   _sage_:0        | Welcome to Sage’s Reference Manual! -- Sage Reference Manual v4.7

   _sage_:1        | Graph Theory -- Sage Reference Manual v4.7
   admin:0         | 2011-09-08 - Eigenvalues, eigenvectors, diagonalization
   
   
   These do not play well with the script because of some unicode characters in them, and the sagenb-export script also does not like the output getting piped to anything else other than stdout! In such cases, it was possible to move the sage documentation directory away and proceed with the conversion:
   
   » readlink lectures.sagenb/home/_sage_
   __store__/9/9b/9bb/9bbe/_sage_
   » mv lectures.sagenb/home/__store__/9 .
   
   Then the script works quite well.
   
   
3. I finally converted the notebooks by renaming the individual notebooks into the column under "Notebook Name" and ending with the extension .ipynb. The output of --list command can be piped to a while loop which helps convert all the notebooks en-mass.

Regards,

 basu.