SageNB, publishing and error 500

[Jori Mäntysalo]

Strange little error: both publishing and stopping to publish give "500:
Internal server error." Publishing works, it's just little oddity. Logfile
has this line:

BuildError: Could not build url for endpoint 'worksheet_publish' with
values ['id', 'username']. Did you mean 'worksheet.worksheet_publish'
instead?

I have no clue about how to debug this.

--
Jori Mäntysalo

[kcrisman]

Hmm.  Probably related: https://ask.sagemath.org/question/38486/build-error-when-uploading-notebooks-in-sage/ and/or https://ask.sagemath.org/question/38468/internal-server-error-after-deleting-or-suspending-a-user/ 

[Jori Mäntysalo]

On Fri, 18 Aug 2017, kcrisman wrote:

> BuildError: Could not build url for endpoint 'worksheet_publish' with
> values ['id', 'username']. Did you mean 'worksheet.worksheet_publish'
> instead?
>

> Hmm.  Probably related: https://ask.sagemath.org/question/38486/build-error-when-uploading-notebooks-in-sage/ and/or
> https://ask.sagemath.org/question/38468/internal-server-error-after-deleting-or-suspending-a-user/ 

Yes, there seems to be the same error type "Could not build url for
endpoint - -". But I did not see how to correct this on those pages.

--
Jori Mäntysalo

[kcrisman]

 You'll note these are all errors about the custom notebook conversion app that is now the default, not sagenb per se.  I wonder whether that app is interfering somehow with the normal running of sagenb.  What happens if you ignore that and directly do sage --notebook=sagenb?  (This is just a wild guess.)

[Jori Mäntysalo]

On Tue, 22 Aug 2017, kcrisman wrote:

> You'll note these are all errors about the custom notebook conversion app that is now the default, not sagenb per se.
> I wonder whether that app is interfering somehow with the normal running of sagenb.  What happens if you ignore that
> and directly do sage --notebook=sagenb?  (This is just a wild guess.)

I will try that (later, as this is from a shared server). Currently I am
running Sage as

sage -c "notebook(secure=False, interface=\"localhost\", port=8321, timeout=36000, server_pool=[\"sagecalc@localhost\"])

I must first modify --notebook=sagenb to this form.

--
Jori Mäntysalo

[Jori Mäntysalo]

On Tue, 22 Aug 2017, kcrisman wrote:

> What happens if you ignore that and directly do sage --notebook=sagenb?

Tested, no help. The same internal error, but the publishing and
unpublishing works.

--
Jori Mäntysalo

[Jori Mäntysalo]

Is there an easy way to see what has been done in
.../sage/local/lib/python2.7/site-packages/flask/ and in
.../sage/local/lib/python2.7/site-packages/sagenb/flask_version/ between
version 8.0 and the one before it? Something like git blame?

With Google I found for example
https://stackoverflow.com/questions/9023488/build-error-with-variables-and-url-for-in-flask
but still can not see what bit I should flip.

--
Jori Mäntysalo

[Dima Pasechnik]

check out the upstream sagenb repo and do git blame, indeed.

[Dima Pasechnik]

although it very well might be due to its dependencies, like flask, etc., updated to newer versions, e.g. for the purpose of python3 compatibility.

[Jori Mäntysalo]

OK. What packages or parts have been updated lately?

--
Jori Mäntysalo

[kcrisman]

See perhaps https://trac.sagemath.org/ticket/22431 or https://trac.sagemath.org/ticket/23066 though I don't see immediately which packages were upgraded there. 

[Dima Pasechnik]

sagenb is installed by pip, which pulls packages from the net; whatever comes up as the "stable" version,

gets installed. We used to pin down versions, but don't do this any more.

You can look at the output of 

./sage --pip list

to see what versions you have.

Sagenb explicitly needs the following, as you can see from its setup.py:

install_requires = [
	'twisted>=11.0.0',
	'flask>=0.10.1',
	'flask-oldsessions>=0.10',
	'flask-openid',
	'flask-autoindex',
	'flask-babel'
	]

[Jori Mäntysalo]

On Thu, 24 Aug 2017, Dima Pasechnik wrote:

> sagenb is installed by pip, which pulls packages from the net; whatever comes up as the "stable" version,gets
> installed. We used to pin down versions, but don't do this any more.
> You can look at the output of 
>
> ./sage --pip list
>
> to see what versions you have.

$ ./sage --pip list | egrep -i 'sage|flask'
Flask (0.10.1)
Flask-AutoIndex (0.6)
Flask-Babel (0.9)
Flask-OldSessions (0.10)
Flask-OpenID (1.2.5)
Flask-Silk (0.2)
sage (8.1b1)
sagenb (1.0.1)
sagenb-export (3.2)
sagetex (3.0)

--
Jori Mäntysalo

[Jori Mäntysalo]

On Thu, 24 Aug 2017, kcrisman wrote:

> OK. What packages or parts have been updated lately?
>
>
> See perhaps https://trac.sagemath.org/ticket/22431 or https://trac.sagemath.org/ticket/23066 though I don't see
> immediately which packages were upgraded there. 

#22431 is a new ticket, but #23066 is closed in 8.0 and has title "sagenb
update to 1.0". It upgrades sagenb from 0.13 to 1.0.1.

Uh, too much code to check.

--
Jori Mäntysalo

[Dima Pasechnik]

perhaps, git bisect ?

 

--
Jori Mäntysalo

[kcrisman]

And keep in mind you can see what was merged in sagenb on Github - nearly everything couldn't have had this effect.  

But see that https://github.com/sagemath/sagenb/commit/7bff646746bfad908a431621eedeb55296594d85 fixes something similar to this, I am guessing, based on https://stackoverflow.com/a/25496041/782821   In fact, https://github.com/sagemath/sagenb/issues/420 seems very similar indeed.  So we seem to have forgotten some of those, somehow.  

Probably one of the instances in /flask_version/worksheet.py - you could check in the full error log you are getting.

[Jori Mäntysalo]

On Fri, 25 Aug 2017, kcrisman wrote:

> But see that https://github.com/sagemath/sagenb/commit/7bff646746bfad908a431621eedeb55296594d85 fixes something similar
> to this, I am guessing, based on https://stackoverflow.com/a/25496041/782821   In
> fact, https://github.com/sagemath/sagenb/issues/420 seems very similar indeed.  So we seem to have forgotten some of
> those, somehow.  
>
> Probably one of the instances in /flask_version/worksheet.py - you could check in the full error log you are getting.

I tried to read the log and the source code. No help. I guess there is
something more dynamic going, not something that could be corrected by
changing href="foo" to href=".foo".

Would be nice if Someone Else(tm) could check this.

--
Jori Mäntysalo

[Jori Mäntysalo]

Kcrisman said to check logs. Here they are:

Traceback (most recent call last):

File
"/home/jm58660/sage/local/lib/python2.7/site-packages/flask/app.py", line
1475, in full_dispatch_request
rv = self.dispatch_request()

File
"/home/jm58660/sage/local/lib/python2.7/site-packages/flask/app.py", line
1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)

File
"/home/jm58660/sage/local/lib/python2.7/site-packages/sagenb/flask_version/decorators.py",
line 22, in wrapper
return f(*args, **kwds)

File
"/home/jm58660/sage/local/lib/python2.7/site-packages/sagenb/flask_version/worksheet.py",
line 51, in wrapper
return f(username, id, **kwds)

File
"/home/jm58660/sage/local/lib/python2.7/site-packages/sagenb/flask_version/worksheet.py",
line 140, in wrapper
return f(*args, **kwds)

File
"/home/jm58660/sage/local/lib/python2.7/site-packages/sagenb/flask_version/worksheet.py",
line 879, in worksheet_publish
print worksheet_publish.url_for(worksheet)

File
"/home/jm58660/sage/local/lib/python2.7/site-packages/sagenb/flask_version/worksheet.py",
line 147, in wc_url_for
return url_for(f.__name__, *args, **kwds)

File
"/home/jm58660/sage/local/lib/python2.7/site-packages/flask/helpers.py",
line 312, in url_for
return appctx.app.handle_url_build_error(error, endpoint, values)

File
"/home/jm58660/sage/local/lib/python2.7/site-packages/flask/app.py", line
1641, in handle_url_build_error
reraise(exc_type, exc_value, tb)

File
"/home/jm58660/sage/local/lib/python2.7/site-packages/flask/helpers.py",
line 305, in url_for
force_external=external)

File
"/home/jm58660/sage/local/lib/python2.7/site-packages/werkzeug/routing.py",
line 1758, in build

raise BuildError(endpoint, values, method, self)

BuildError: Could not build url for endpoint 'worksheet_publish' with
values ['id', 'username']. Did you mean 'worksheet.worksheet_publish'
instead?

I have tried to grep the source, but without help.

* * *

However, this seems to be documented bug already:

https://github.com/sagemath/sagenb/issues/432

--
Jori Mäntysalo

[kcrisman]

   File
"/home/jm58660/sage/local/lib/python2.7/site-packages/sagenb/flask_version/worksheet.py",
line 879, in worksheet_publish
     print worksheet_publish.url_for(worksheet)

   File
"/home/jm58660/sage/local/lib/python2.7/site-packages/sagenb/flask_version/worksheet.py",
line 147, in wc_url_for
     return url_for(f.__name__, *args, **kwds)

"/home/jm58660/sage/local/lib/python2.7/site-packages/werkzeug/routing.py",
line 1758, in build

     raise BuildError(endpoint, values, method, self)
BuildError: Could not build url for endpoint 'worksheet_publish' with
values ['id', 'username']. Did you mean 'worksheet.worksheet_publish'
instead?

This suggests hacking line 879 ... but you are right that things don't quite correspond. 

I suspect perhaps https://github.com/sagemath/sagenb/commit/9dd5c0f8c783de7cb0ae21c9f445ab8260b5a0ac is the culprit, and put something on #432 to the author of that change.  It seems like the kind of thing that might happen when a module is updated, as that change did, though I could be barking up the wrong tree too.  Thanks for pursuing this.

 

However, this seems to be documented bug already:

https://github.com/sagemath/sagenb/issues/432

Could be, though they didn't give the traceback :( 

[Jori Mäntysalo]

On Wed, 6 Sep 2017, kcrisman wrote:

> I suspect perhaps https://github.com/sagemath/sagenb/commit/9dd5c0f8c783de7cb0ae21c9f445ab8260b5a0ac is the culprit, and
> put something on #432 to the author of that change.  It seems like the kind of thing that might happen when a module is
> updated, as that change did, though I could be barking up the wrong tree too.  Thanks for pursuing this.

Another question: Can I downgrade SageNB when waiting correction for this
bug? Is so, how?

--
Jori Mäntysalo

[Dima Pasechnik]

Jori, 

sorry for crying out loud, but noone is going to work on sagenb, besides maybe you, I think.

Messing around with 11 years old web technology is very hard indeed.

What exactly are you missing from jupyter nb, or jupyter hub, or some version of SMC

people managed to get running on their intranets?

Dima

PS. Reverting to an older sagenb should work, via reverting the corresponding commits, etc...

[Jori Mäntysalo]

On Thu, 7 Sep 2017, Dima Pasechnik wrote:

> What exactly are you missing from jupyter nb, or jupyter hub, or some version of SMC
> people managed to get running on their intranets?

I don't actually know; SageNB has just been a working piece of software.

What we need is

- User management from LDAP or Shibboleth, so that everyone with an
account on them can just sign in to Sage.
- Local user accounts too.
- Security, so that a user A can not see worksheets for user B.
- Importing SageNB worksheet to a new system worksheet.

--
Jori Mäntysalo

[Enrique Artal]

And probably the ability of sharing worksheets, I did not find it in jupyter or jupyterhub; of course SMC would be OK but I see only the option of campus accounts or personal docker copies.

[Dima Pasechnik]

did you see this:

https://benjamin-hackl.at/2016/01/16/jupyterhub-with-sagemath-kernel/

It seems that all of the requirements you list are there...

[Volker Braun]

You can share worksheets just by uploading them to github, for example:

http://nbviewer.jupyter.org/github/vbraun/torsion_cup_product/blob/master/Y0_cohomology_ring.ipynb 

https://github.com/vbraun/torsion_cup_product/blob/master/Y0_cohomology_ring.ipynb

[Jori Mäntysalo]

On Thu, 7 Sep 2017, Volker Braun wrote:

> You can share worksheets just by uploading them to github, for example:

Isn't that publishing worksheet, not sharing?

--
Jori Mäntysalo

[Dima Pasechnik]

well, in principle you can put it in a private repo, and control access to it the github way.

Although I agree this is not very easy as opposed to sagenb or SMC worksheet sharing.

[kcrisman]

On Thursday, September 7, 2017 at 11:16:14 AM UTC-4, Dima Pasechnik wrote:

did you see this:

https://benjamin-hackl.at/2016/01/16/jupyterhub-with-sagemath-kernel/

It seems that all of the requirements you list are there...

This seems very interesting, especially if it could be packaged up in a box, so to speak, for deployment. 

[kcrisman]

Yeah, that's really quite different and more involved.

It's true that no one who has time and expertise will be working much on sagenb, it appears.  But it does have features which still aren't easily available in something people can "just put up there" so I agree there isn't a drop-in replacement yet, and a few of us will try to give advice as we are able.

[Enrique Artal]

It is quite useful, indeed. We have installed it in our servers.

[kcrisman]

On Saturday, September 9, 2017 at 6:12:49 AM UTC-4, Enrique Artal wrote:

It is quite useful, indeed. We have installed it in our servers.

Could it be made "even easier" to deploy?  Sagenb really just required someone who knew how to give things a domain name and commands to start a server. 

Also, more formal testing of the requirements for memory would be helpful - many local sagenb installs only had a couple gig of RAM available on a virtual machine and so couldn't handle more than a dozen or so users at a time.

[Enrique Artal]

It is a server with 8 cores and 32GB of RAM, but with several other sagenb processes and twenty users on jupyterhub plus 60 on the sagenb instances cause no performance issue (usually)

[kcrisman]

On Saturday, September 9, 2017 at 6:54:38 PM UTC-4, Enrique Artal wrote:

It is a server with 8 cores and 32GB of RAM, but with several other sagenb processes and twenty users on jupyterhub plus 60 on the sagenb instances cause no performance issue (usually)

Oh yeah, 32 GB would be plenty :) 

[Jori Mäntysalo]

On Thu, 7 Sep 2017, Dima Pasechnik wrote:

> did you see
> this:https://benjamin-hackl.at/2016/01/16/jupyterhub-with-sagemath-kernel/
>
> It seems that all of the requirements you list are there...

Not sharing notebooks, but it seems that it has been done later:

https://github.com/jupyterhub/hubshare/issues/15

Maybe I wait for this to stabilize and then try to install all parts and
see if it works.

--
Jori Mäntysalo