setting up nbgrader for multiple graders with jupyterhub

[Jonathan Morgan]

Hello,

For a class I am helping to teach, I am trying to set up the latest nbgrader from pip (0.2.1) in the latest jupyterhub from pip (0.2.0) using Ipython 4.0.0 against python 3.4.0 on an ubuntu 14.04.3 LTS machine.

We would like to have multiple users set up as instructors/graders, able to work in parallel (or serially but trading off, if parallel doesn't really work).

I have reading the nbgrader getthedocs doc trying to figure out how this all should work.  While I think I am starting to understand, I must admit to still being confused.  So, while I am moving different pieces around and seeing what works, I figured I'd also post to this forum and see if I had missed some doc, or if there are generally accepted ways of working with this.

At the highest level, if there is documentation that I've missed that pull this all together in more detail (how to set up the folders and files for nbgrader, general usage for grader and student, and then what a student and a grader need in their home directories to get this all to work), I apologize for not finding it (I promise I looked).  Please point me at it.  If not, once I understand, I'd be happy to help with documentation if you think that would be useful.

What I have so far:

• a single directory where jupyterhub configuration and database live, and jupyterhub up and running.
• nbgrader installed, nbgrader's extensions installed and seeming to work just fine (the Assignment toolbar works, and the Assignment tab shows in Jupyter).
• a directory "nbgrader/courses/example" that I think needs to contain the nbgrader hierarchy, if that is how this works, the nbgrader "gradebook.db" database, and any other related files, so they can be shared among multiple instructors/graders (currently inside of the jupyterhub directory, but could be moved as needed).
• This includes a clone into "source" of a github repository structured such that I think it matches the structure expected of "source", for versioned collaboration on the assignments.
• a set of nbgrader configuration properties for formgrade that are in a file named nbgrader_config.py, currently sitting in the jupyterhub folder, but incomplete.

My questions:

• general
• is there a standard centralized location for a system-level nbgrader_config.py configuration that would be every user's default, or does nbgrader just look for one in the directory where it is run, and you then have to adjust that based on role and needs of individual users?
• is there a standard location for the course folder that contains the nbgrader.db database, the nbgrader file system hierarchy, etc.?
• If you want multiple graders/instructors able to work together on assignments, is there a recommended location for or configuration of the nbgrader course folder such that all users with role grader can be configured to interact with a certain course folder, or do you just have to have a place where graders/instructors all have access and can go and run the nbgrader command?  Would they need this sym-linked into their home directory for formgrade?
• am I courting disaster with multiple graders/instructors?
• for users (graders/instructors and students):
• What files or folders for nbgrader need to be in a given user's home directory?  Sounds like they need an nbgrader_config.py.  Anything else?
• What does nbgrader automatically create for users?
• Is any of this difference between the requirements for graders/instructors and students?  If each has their own nbgrader_config.py, and there isn't a system-wide default, then they'd probably have different contents?
• exchange directory
• what is the default exchange directory?
• can this be the same location as where the lesson "source" folder is?
• is there a centralized place where you can specify this for all users?
• formgrade app
• if this is all configured per-user, can you have more than one user in formgrade at once?

More specific questions from documentation I've looked over:

• http://nbgrader.readthedocs.org/en/stable/user_guide/01_philosophy.html
  
• concept of nbgrader file system structure and hierarchy, but no indication I see of where that needs to live and how you reference it across multiple students or multiple instructors.
• http://nbgrader.readthedocs.org/en/stable/user_guide/03_generating_assignments.html
• mention of needing a "gradebook.db" database, but not specific about where one would put it (implies it is in course folder at same level as "source" folder, I think).
• code to add assignments to database, and how to then release an assignment, and implication, I believe, that an instructor should run this code as well as the "nbgrader" command in the directory where the "source" directory lives (and it looks like that is the expectation throughout, but I don't think I've seen that explicitly stated).
• http://nbgrader.readthedocs.org/en/stable/user_guide/11_jupyterhub_config.html
• here, it tells you how to configure an nbgrader_config.py file for formgrade app, but I don't see where one is supposed to put it or how one tells nbgrader where it is.  Makes more sense if each user gets one, or at least you need one in the same folder where you run nbgrader, but doesn't say that.
• And, if multiple graders, can multiple users log in to an instance of formgrade?  It takes up a port, so if not, per user, would you need separate ports for different instructors, or to coordinate so only one person using at a time?
• c.HubAuth.notebook_url_prefix - the notes on this imply that it will be relative to the current user's home directory - is this to guide the formgrade app to the course folder?
• So does this imply that each grader who might use this app will need at least a symbolic link to the course folder in their home folder?
• Could I make a standard nbgrader_config.py that I put in the course folder that anyone with access could use to launch the formgrader?
• http://nbgrader.readthedocs.org/en/stable/user_guide/07_releasing_assignments.html
• here it talks about more nbgrader_config.py that will get automatically loaded when we run nbgrader, but it contains different information than is in the nbgrader_config.py in 11_jupyterhub_config.
• it also doesn't say explicitly where that file needs to live, or where you are running the nbgrader command from (from the directory that holds the nbgrader file system hierarchy?).
• Sounds like it assumes a single instructor, also?
• http://nbgrader.readthedocs.org/en/stable/user_guide/08_fetching_assignments.html
• what is the default exchange directory?
• looks like each student will need an nbgrader_config.py file - will it just live in their home directory?  Can you change it so that they at least are in a child directory, so they don't clutter the home directory?
• how do the commands a student runs know where the database is?
• http://nbgrader.readthedocs.org/en/stable/user_guide/10_collecting_assignments.html
  
• how would an instructor/grader's nbgrader_config.py file need to be configured for this to work?  Would they need the same explicit definition of the exchange directory, and then need to run the nbgrader command in the course directory?

Any help will be greatly appreciated.

Thanks,

Jonathan Morgan

[Jess Hamrick]

Hey Jon,

Thanks for takig the time to go through everything so carefull! I’ll try to answer your questions as best I can, and this will definitely help me improve the docs a lot :-)

• is there a standard centralized location for a system-level nbgrader_config.py configuration that would be every user's default, or does nbgrader just look for one in the directory where it is run, and you then have to adjust that based on role and needs of individual users?

You can put a system-wide one in /etc/jupyter/nbgrader_config.py. If you want to automatically install a different one per-user, you can put them in $HOME/.jupyter/nbgrader_config.py. I don’t think this is currently documented anywhere, but it definitely should be!

• is there a standard location for the course folder that contains the nbgrader.db database, the nbgrader file system hierarchy, etc.?

No, not at the moment. It might not be a bad idea to have something like that, though. I’ve been toying with the idea of having nbgrader “profiles” (kind of similar to IPython profiles) and if I added that then I think it would make sense to specify a profile when you run nbgrader, and then be able to specify what the root directory for that profile is, with some sensible default. In the meantime I could just add an option to nbgrader to specify the root of the class files directory.

• If you want multiple graders/instructors able to work together on assignments, is there a recommended location for or configuration of the nbgrader course folder such that all users with role grader can be configured to interact with a certain course folder, or do you just have to have a place where graders/instructors all have access and can go and run the nbgrader command?  Would they need this sym-linked into their home directory for formgrade?

The way I’ve been doing this for my classes is to have only one person run the official commands (nbgrader release, nbgrader collect, nbgrader autograde, nbgrader formgrade) but allow other graders to access the formgrader by adding them to the list of graders. I think probably (for the moment) if you want multiple people to be able to run those commands, the best way to do it would be to create a special “nbgrader” user on the system and give the password to everyone you want to be able to run the nbgrader commands. I think, ideally, that in the future if I do something like the profiles idea above, then it should be possible to configure permissions so that all instructors can run the commands.

• am I courting disaster with multiple graders/instructors?

No :-) We had 5 people doing grading for my class last semester and it worked great, the only thing was that there was just a single person who actually ran the commands — everyone else only ever accessed the formgrader.

• What files or folders for nbgrader need to be in a given user's home directory?  Sounds like they need an nbgrader_config.py.  Anything else?

Nope, unless there’s something specific you want to add. And as I mentioned above you can put the nbgrader_config.py in a system location if you’d prefer to do that.

• What does nbgrader automatically create for users?

By users do you mean students or instructors? For students, not much, but that’s because there’s not really much to create, with the exception that if students run nbgrader fetch (or use the assignment list extension) it will automatically copy the problem set into their home directory for them. For instructors, you need to have notebooks in the `source` directory, and nbgrader will create everything else (`release`, `autograded`, `feedback`). The exception is possibly the `submitted` folder, which will get created automatically if you run nbgrader collect, but if you don’t you’ll need to create it yourself.

• Is any of this difference between the requirements for graders/instructors and students?  If each has their own nbgrader_config.py, and there isn't a system-wide default, then they'd probably have different contents?

The only thing that really needs to be in a student’s config is the path to the exchange directory (if it is different from the default) and the class name. For the user who runs the nbgrader commands, I’ve just put a nbgrader_config.py file in the root of the class files repository itself. The config files will stack, so it first imports the global one and then the one in the local directory.

• what is the default exchange directory?

/srv/nbgrader/exchange

• can this be the same location as where the lesson "source" folder is?

It could be, but I probably wouldn’t recommend it. It is probably better to have it somewhere that’s more of a global system path, and also away from your nbgrader database and stuff.

• is there a centralized place where you can specify this for all users?

Yep, in the nbgrader_config.py if you put it in /etc/jupyter/nbgrader_config.py

• if this is all configured per-user, can you have more than one user in formgrade at once?

Yes. By default, anyone who has the URL can access the formgrader. If you use the HubAuth class, then you can specify a list of JupyterHub users who are allowed to access the formgrader, and then it’s not sufficient to just have the URL.

More specific questions from documentation I've looked over:

These questions are super helpful! If it’s alright with you, I will copy and paste your questions into an issue on github on the nbgrader repo and address them there (or you are welcome to do so if you’d prefer to open the issue yourself).

Cheers,

Jess

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/db88e6ed-4763-4dff-8fbe-d5c5a853f9ad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian Granger]

Jonathan,

Sounds like you are diving in. Jess has given great answers here so I
won't rehash that. Let us know if you run into anything along the way.

Cheers,

Brian

> --
> You received this message because you are subscribed to the Google Groups
> "Project Jupyter" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jupyter+u...@googlegroups.com.
> To post to this group, send email to jup...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jupyter/db88e6ed-4763-4dff-8fbe-d5c5a853f9ad%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
Brian E. Granger
Associate Professor of Physics and Data Science
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgra...@calpoly.edu and elli...@gmail.com

[Jonathan Morgan]

Thanks!  This infomation is great, just what I was looking for.

We made a separate instructor user where we put the course directories, an nbgrader_config.py, and the gradebook, for actually managing assignments (as you suggested).  We are now using the default config file in /etc/juptyer/nbgrader_config.py to start (glad to know of that), and I’m running a test assignment through today.  We are going to use the “nbgrader” commands to move assignments through the flow.  We’re using the default “/srv/nbgrader/exchange” folder.

I'm taking detailed notes, and once I get it all set up, I'll post them here, and then would be happy to help revise the documentation.  I made a ticket with those doc questions, issue 391.

More soon...

Thanks again for the help!

Jon

[Jonathan Morgan]

Hi,

Thanks to Jessica's help (thanks again!), I got everything installed and working over the weekend - jupyterhub and nbgrader for Python 2 and Python 3 kernels.

I documented the steps as I went.  The resulting markdown and PDF are attached.  I can't guarantee that this is complete, but I quickly ran through it a second time on a second server yesterday, just to kick the tires, and that server is up and running, too.

The only issue that remains is a problem with getting formgrade to work with jupyterhub when jupyterhub has SSL enabled.  When I turn SSL on in jupyterhub by specifying an ssl_key and ssl_cert, jupyterhub is fine, but the nbgrader formgrade application becomes inaccessible.  I am not sure what is going on, but nothing shows up in the formgrade logs, and I'm not sure where to find logging from the proxy server.  What you see in the browser when you hit 'https://<domain>/hub/nbgrader/<classname>" is ongoing loading, like what you see when you visit a page using HTTP that is only available in HTTPS.  It doesn't time out quick, it doesn't get an error, it just keeps spinning, and it can go for minutes.  If anyone has any ideas of how to fix this, please let me know.

Thanks,

Jon

[Jess Hamrick]

Hey Jon,

Thanks so much for going through and documenting this so well! I think this is really going to help others who want to set up something similar. I think I will create a wiki on the nbgrader repo and create a “case studies” area — if you wanted to add this there too I that would be really great!

Cheers,

Jess

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/8030d551-c0d2-45d7-a393-34989c476635%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

<jupyterhub_nbgrader_install.markdown><jupyterhub_nbgrader_install.pdf>

[Jonathan Scott Morgan]

Glad to wiki it.  Better to get it somewhere it can live and evolve if there is call for that.

Jon

--

Jonathan Morgan
jons...@gmail.com

You received this message because you are subscribed to a topic in the Google Groups "Project Jupyter" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jupyter/xEiZwsGTdMg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jupyter+u...@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/575CF05B-FAE3-4668-BC61-97DB49EE745A%40gmail.com.