Jupyterhub and nbgrader problem in RedHat

[DVD PS]

Hello there!!

Thanks for these amazing tools built to make jupyter a better tool for teaching.  I am planning to use nbgrader in the python course I teach for researchers that starts this week. However, I'm having a problem with the machine that's running jupyterhub and I cannot figure it out... nor know what to search to find the problem.

I've run the same setup on a centos7 in a virtual box and it works. The only different setup is centos7 vs RedHat and local users vs LDAP.

The problem happens when initialising the jupyterhub which has in the config the following:

c.JupyterHub.services = [
    {
        'name': 'formgrader-worldmusic',
        'admin': True,
        'command': ['nbgrader', 'formgrade', '--debug'],
        'url': 'http://127.0.0.1:9000',
        'cwd': '/home/myuser/nbgrader/WorldMusic'
    }
]
 

Looking at the debug logs this is what I get:

[I 2017-05-02 20:32:19.900 JupyterHub app:1453] Hub API listening on http://127.0.0.1:8081/hub/
[I 2017-05-02 20:32:19.904 JupyterHub app:1176] Starting proxy @ http://127.0.0.1:8000/
[D 2017-05-02 20:32:19.904 JupyterHub app:1177] Proxy cmd: ['configurable-http-proxy', '--ip', '127.0.0.1', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--default-target', 'http://127.0.0.1:8081', '--error-target', 'http://127.0.0.1:8081/hub/error', '--ssl-key', '/etc/httpd/jupyterhub-staging-certs/jh.dev.ucl.ac.uk.key', '--ssl-cert', '/etc/httpd/jupyterhub-staging-certs/jh.dev.ucl.ac.uk.crt']
[D 2017-05-02 20:32:20.121 JupyterHub app:1205] Proxy started and appears to be up
[I 2017-05-02 20:32:20.122 JupyterHub service:220] Starting service 'formgrader-worldmusic': ['nbgrader', 'formgrade', '--debug']
[I 2017-05-02 20:32:20.127 JupyterHub service:90] Spawning nbgrader formgrade --debug
[E 2017-05-02 20:32:20.141 JupyterHub service:101] Permission denied trying to run '/opt/anaconda/bin/nbgrader'. Does root have access to this file?
[C 2017-05-02 20:32:20.141 JupyterHub app:1468] Failed to start service formgrader-worldmusic
    Traceback (most recent call last):
      File "/opt/anaconda/lib/python3.5/site-packages/jupyterhub/app.py", line 1466, in start
        service.start()
      File "/opt/anaconda/lib/python3.5/site-packages/jupyterhub/services/service.py", line 243, in start
        self.spawner.start()
      File "/opt/anaconda/lib/python3.5/site-packages/jupyterhub/services/service.py", line 95, in start
        cwd=self.cwd or None,
      File "/opt/anaconda/lib/python3.5/subprocess.py", line 947, in __init__
        restore_signals, start_new_session)
      File "/opt/anaconda/lib/python3.5/subprocess.py", line 1551, in _execute_child
        raise child_exception_type(errno_num, err_msg)
    PermissionError: [Errno 13] Permission denied
   
[D 2017-05-02 20:32:20.142 JupyterHub application:642] Exiting application: jupyterhub


I'm very puzzled as root can run `nbgrader formgrade` without problems, and so from myuser.

Does anyone here have an idea of what I could be doing wrong?

Thanks a lot!
David

[DVD PS]

 

Does anyone here have an idea of what I could be doing wrong?

Ok, so I've managed to get rid of that error setting "cwd" to "/tmp/myuser/nbgrader/WorldMusic" - Maybe in this system, root doesn't have access to write on users directories... but I would have imagined this would run as the rest of the notebooks - as per users...

Now though I've got a 503 error when visiting services/formgrader-worldmusic...  and the following when trying to run `nbgrader assign ps1`

    sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) attempt to write a readonly database [SQL: 'INSERT INTO assignment (id, name, duedate) VALUES (?, ?, ?)'] [parameters: ('caa5371e6d6c476c90a74008de1cac4a', 'ps1', None)]

which seems also related with the `root` user...

At least seems I'm moving somehow forward :)

Cheers,
David

[Jessica B. Hamrick]

Hi David,

Hmm, yeah, this seems like a permissions issue to me as well. I would expect you should be able to run the formgrader with the cwd as /home/myuser/nbgrader/WorldMusic with no problems... though I haven't personally use nbgrader with RedHat or LDAP. It might be that LDAP is causing issues. Can you list the contents of /home/myuser/nbgrader/WorldMusic with permissions and also share your nbgrader config?

Cheers,

Jess

--
You received this message because you are subscribed to the Google Groups "Teaching with Jupyter Notebooks" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter-education+unsubscribe@googlegroups.com.
To post to this group, send email to jupyter-education@googlegroups.com.
Visit this group at https://groups.google.com/group/jupyter-education.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter-education/02e27caa-3256-4717-a1b8-2e383cefdc50%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[DVD PS]

Hi Jess, everyone

I've started the below as a simple and quick email... but then I wanted to provide the right error messages... and I've got into this long description of which stesp I tried.

TLDR; It works now so I probably was missing something - but I still don't know the right workflow on how to puppetise a machine where I won't have root access.

After trying different stuff I've found out some other things that may throw some light into the issues.

- The home directories, thought the authentication happens via LDAP is mounted by NFS. Additionaly, since a few days the NFS of the university have been  having problems - I don't know whether that was affecting too.

- If the jupyterhub (via systemctl) starts with the formgrade service and the directory - `cwd` - doesn't exist then jupyterhub fails.

     - if such directory is in the instructor home - then I get the permission denied I showed in the first email - `PermissionError: [Errno 13] Permission denied`

     - if the directory is pointing to a place locally, then I get `FileNotFoundError: [Errno 2] No such file or directory: '/srv/nbgrader/myusers/WorldMusic'`

- if in the other hand the directory exists but it's empty - then the above happens again if in the instructor directory, but works if in the local directory. In that casein creates `gradebook.db` owned by root.
        -rw-r--r-- 1 root    root   128K May  4 12:04 gradebook.db

  and formgrader shows as service no provider (sure, the directory doesn't contain the `nbgrader_config.py`)

  adding the assignment with the config files and source and restarting jupyterhub, it works!!! Yay!!!! - formgrade service is shown!!

- With that working, then I go ahead with the next step: "assign" - this fails as the database is owned by root. If I change from root the owner to myuser, then assign works. However, I won't have access as root in the production machine.

   Let's try to stop jupyterhub, remove the database, assign the course (so the database is created by the instructor) - that works too!!  (you may not feel it in my writting, but I'm jumping of happiness in my office :))

So it's working now... but I have a little problem with the workflow to set up all this. The reason of why I'm trying with an empty directory is that this machine needs to be set up using puppet, so I'm trying to do it in a similar way than in https://github.com/jupyterhub/jupyterhub-deploy-teaching/ - Having some kind of template yaml file for the instructors to write, then run puppet to create the services and start jupyterhub. The problem is that unless the course assigment has already been created (and a gradebook.db) exist in the right location, jupyterhub will create a db owned by root.

The only solution I've found, which it seems to work now is to create an empty file (`touch gradebook.db`), set the owner to the instructor, and start jupyterhub afterwards (which will write to the file). But then I need also the nbconfig_config so the server starts looking at the right place. If I add this then I don't know how to allow the instructors to update (add) new assignments to the course, without having to restart the jupyterhub (which only root can do so - jupyterhub admins can stop it, but I've not found a way to restart it).

The only way I can find to use it is to run formgrade locally in my machine, and keep the jupyterhub only with nbgrader.  Suggestions, ideas or work arounds are very welcome.

On 3 May 2017 at 22:53, Jessica B. Hamrick <jessica....@gmail.com> wrote:

Hi David,

Hmm, yeah, this seems like a permissions issue to me as well. I would expect you should be able to run the formgrader with the cwd as /home/myuser/nbgrader/WorldMusic with no problems... though I haven't personally use nbgrader with RedHat or LDAP. It might be that LDAP is causing issues. Can you list the contents of /home/myuser/nbgrader/WorldMusic with permissions and also share your nbgrader config?

The nbgrade_config I'm using contains the following:

c = get_config()

###############################################################################
# Begin additions by nbgrader quickstart
###############################################################################

# You only need this if you are running nbgrader on a shared
# server set up.
c.NbGrader.course_id = "WorldMusic"

# Update this list with other assignments you want
c.NbGrader.db_assignments = [dict(name="ps1")]

# Change the students in this list with that actual students in
# your course
c.NbGrader.db_students = [
    dict(id="ucasper", first_name="Ben", last_name="Bitdiddle"),
]

###############################################################################
# End additions by nbgrader quickstart
###############################################################################

c.FormgradeApp.ip = "127.0.0.1"
c.FormgradeApp.port = 9000
c.FormgradeApp.authenticator_class = "nbgrader.auth.hubauth.HubAuth"
c.HubAuth.grader_group = "RITS-graders"

which is pretty much the one at http://nbgrader.readthedocs.io/en/stable/configuration/jupyterhub_config.html#configuring-nbgrader-formgrade

Root can access to local files owned by others, but not in the NFS. It seems root is not that powerful after all :)

So, most probably when I send the first email I missed something on the way - I believe the setting on FormgradeApp.ip or they were not the same on both sides.

Thanks a lot!!

David

 

[Jessica B. Hamrick]

Hi David,

Thanks for the further details! They helped me remember an issue that there is with JupyterHub services whic I think is what you're running up against. This is something that should be fixed in JuptyerHub master, but which isn't in a release, yet.

In the ideal case, what I think you want to do is to run the formgrader as the instructor (rather than as root), but this has the effect of not starting it in the correct directory (as specified by the cwd config option): https://github.com/jupyterhub/jupyterhub/issues/935

So what I think the jupyterhub-deploy-teaching setup does is to run the formgrader as root (as you're doing) and then chown the gradebook to the instructor account after it's been created. See https://github.com/jupyter/nbgrader/issues/647 for some more discussion around this issue.

So your options are:

1. Run with JupyterHub master, and run the formgrader as the instructor, which should work out-of-the-box (but let me know if it doesn't!)
2. Run with JupyterHub 0.7.2, and add something to Puppet that either creates the file ahead of time with the right permissions (as your'e doing) or sets them after the file's been created.

Regarding your last question:

> But then I need also the nbconfig_config so the server starts looking at the right place. If I add this then I don't know how to allow the instructors to update (add) new assignments to the course, without having to restart the jupyterhub (which only root can do so - jupyterhub admins can stop it, but I've not found a way to restart it).

You don't need to restart JupyterHub to add new assignments -- you can add them on the command line with `nbgrader db` without need to restart, see: http://nbgrader.readthedocs.io/en/stable/user_guide/managing_the_database.html

Hope that helps!

Cheers,

Jess

--
You received this message because you are subscribed to the Google Groups "Teaching with Jupyter Notebooks" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter-education+unsubscribe@googlegroups.com.
To post to this group, send email to jupyter-education@googlegroups.com.
Visit this group at https://groups.google.com/group/jupyter-education.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter-education/CAD0QkqUgovZOREfZM6Pp4873bfcBDz0d80B-_mM0JP8Gzkwmig%40mail.gmail.com.