Jupyterhub feature Qs

[Andrew Melo]

Hi all,

I run a JupyterHub instance @ Vanderbilt and had the need/desire for a
couple features. Before I went down the rabbit hole of trying to write
the code myself, I figured I'd ask if someone had already implemented
them :)

1) Occasionally users will hit the memory limit, which causes their
notebook container to be killed, causing JupyterHub to return 503
errors when the proxy tries to access the non-existent backend. Does
anyone have a suggestion of how to give the user some type of message
that says "Hey User, your notebook was killed, probably because you
exceeded the memory limits". I guess there would be two parts a) some
mechanism to queue "sysadmin alerts" to a user and b) some hooks in
the spawner to send an alert to the user.

The "announcements" feature looks like it would be close (by writing
the announcement to a shared path and symlinking it), but I don't know
of how I could let the user "dismiss" the announcement once they saw
it.

2) There is often interesting data sent to the stdout/stderr of the
notebook process (particularly with Spark) which, in our case, dumps
to the stdout/err of the docker container, which the user doesn't have
access to. Is there a way to capture that output and forward it to a
page that would be visible the user facing side of the app?

Any resources/suggestions would be greatly appreciated.
Thanks!
Andrew

[Aaron Watters]

Regarding (2) I think in principle you could mount the log area of the docker host into the running instance.

That folder could either be directly visible to the Jupyter file browser or you could process it using Python

code.  I suppose you might want to restrict the user to only see their own log, which might complicate

the implementation.

My 2p for what it is worth.  -- Aaron Watters

--
You received this message because you are subscribed to the Google Groups "Jupyter at Research Facilities" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter-research-fa...@googlegroups.com.
To post to this group, send email to jupyter-resea...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter-research-facilities/CAJY4aWGtoPPnaC__4QYrTPG3EPu9bq4g%3Dev3hNCiYCLKXCu5QQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[MinRK]

On Tue, Dec 11, 2018 at 6:42 PM Andrew Melo <andre...@gmail.com> wrote:

Hi all,

I run a JupyterHub instance @ Vanderbilt and had the need/desire for a
couple features. Before I went down the rabbit hole of trying to write
the code myself, I figured I'd ask if someone had already implemented
them :)

1) Occasionally users will hit the memory limit, which causes their
notebook container to be killed, causing JupyterHub to return 503
errors when the proxy tries to access the non-existent backend. Does
anyone have a suggestion of how to give the user some type of message
that says "Hey User, your notebook was killed, probably because you
exceeded the memory limits". I guess there would be two parts a) some
mechanism to queue "sysadmin alerts" to a user and b) some hooks in
the spawner to send an alert to the user.

The "announcements" feature looks like it would be close (by writing
the announcement to a shared path and symlinking it), but I don't know
of how I could let the user "dismiss" the announcement once they saw
it.

This is often desired and a little tricky. Announcements might be able to be used, but that's mostly for all-users broadcasts. It could be tricky to do it per-user.

You could write a custom template specifically for 503 errors (create a 503.html in the templates directory). If this usually happens due to resource exhaustion, you could have a static message there that just says "this was probably caused by running out of memory, here are some resources/actions...". That isn't what you've asked for, but would probably the the least-effort solution to at least improve the situation somewhat.

If you want to show messages only after they restart the notebook server, this is easier, because there is better access to customize what happens on a notebook page (extensions and the like) than there is on hub-served pages. You would need to:

1. log the event somewhere,

2. write an nbextension and/or server extension that finds this event and displays it

 

2) There is often interesting data sent to the stdout/stderr of the
notebook process (particularly with Spark) which, in our case, dumps
to the stdout/err of the docker container, which the user doesn't have
access to. Is there a way to capture that output and forward it to a
page that would be visible the user facing side of the app?

Depending on the circumstance, wurlitzer might be useful. It captures low-level C output and forwards it to the frontend, so things written to the terminal will actually show up in the notebook. It has an IPython extension for enabling automatically and everything. (disclaimer: I made it for exactly this purpose).

An alternative is to change the command used to start the notebook server to a custom script that uses `tee`, e.g.

#!/bin/sh

exec jupyterhub-singleuser "$@" 2>&1 | tee server.log

This should duplicate stdout + stderr into the file `server.log` which users can read. You can then modify `c.Spawner.cmd` to point to your launch script instead of the default.

 

Any resources/suggestions would be greatly appreciated.
Thanks!
Andrew