Using batchspawner while not running as root

[Andreas Hilboll]

Dear Jupyter community,

I'm planning to set up a jupyterhub instance on our HPC cluster, where
individual notebooks should be spawned on the compute nodes via SLURM.

From what I'm reading, the batchspawner would be my way to go. However,
I would *also* like the jupyterhub process to run under a different UID
than root. For this, there seems to be the sudospawner. But it seems
that the two are mutually exclusive.

So my question is:

How can I have jupyterhub run as non-root user while using the
batchspawner using SLURM?

Cheers,
Andreas.

--
Dr. Andreas Hilboll

Center for Marine Environmental Sciences (MARUM)
- AND -
Institute of Environmental Physics (IUP)

University of Bremen

NW1 / S3132
Otto-Hahn-Allee 1
D-28359 Bremen
Germany

+49(0)421 218 62133 (phone)
+49(0)421 218 98 62133 (fax)
http://www.iup.uni-bremen.de/~hilboll

[MinRK]

On Wed, Dec 21, 2016 at 2:47 PM, Andreas Hilboll <hil...@uni-bremen.de> wrote:

Dear Jupyter community,

I'm planning to set up a jupyterhub instance on our HPC cluster, where
individual notebooks should be spawned on the compute nodes via SLURM.

From what I'm reading, the batchspawner would be my way to go.  However,
I would *also* like the jupyterhub process to run under a different UID
than root.  For this, there seems to be the sudospawner.  But it seems
that the two are mutually exclusive.

So my question is:

How can I have jupyterhub run as non-root user while using the
batchspawner using SLURM?

There are two default behaviors that need root permissions in the default configuration:

First, is the PAMAuthenticator, which needs access to the PAM service. On a typical debian/ubuntu system, any user in the `shadow` group can check passwords. That's the actual capability that the process needs. If you are using a different Authenticator, such as an SSO/OAuth setup, you may not need any elevated permissions here.

The next step is the Spawner. At a high level, the Spawner needs to be able to start notebook servers 'as' specific users. The default Spawner uses `setuid`, which requires root permissions. The `sudospawner` uses slightly complex `sudo` configuration to grant *restricted* switch-user permissions. What you will need for BatchSpawner is the ability to submit SLURM jobs on behalf of other users. I imagine the 'simplest' version of this is to setuid to the actual user, and submit these jobs. But if there's another way to submit SLURM jobs on behalf of other users without 'becoming' the user first, that should allow you to run the batch spawner without running the server as root. I'm not quite sure what that would be, though.

-Min

 

Cheers,
  Andreas.

--
Dr. Andreas Hilboll

Center for Marine Environmental Sciences (MARUM)
                    - AND -
    Institute of Environmental Physics (IUP)

University of Bremen

NW1 / S3132
Otto-Hahn-Allee 1
D-28359 Bremen
Germany

+49(0)421 218 62133  (phone)
+49(0)421 218 98 62133 (fax)
http://www.iup.uni-bremen.de/~hilboll

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+unsubscribe@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/6k8tr975ry.fsf%40shaula.iup.uni-bremen.de.
For more options, visit https://groups.google.com/d/optout.

[oft.k...@gmail.com]

On Wednesday, December 21, 2016 at 1:47:49 PM UTC, Andreas Hilboll wrote:

[...]

How can I have jupyterhub run as non-root user while using the
batchspawner using SLURM?

Sorry for resurrecting an old thread, but I want to report that it is indeed possible (and not very hard) to run batchspawer as non-root.

The versions I used are:
- anaconda3 5.0.1
- jupyterhub 0.8.1
- batchspawner current master (0f115fe0d6a5249b618255e69d20c44d8af025ee)

No modifications of the batchspawner code are necessary. It looks like all interactions of batchspawner with the batch system already go through sudo, so you only need to setup the sudo configuration. We use SGE, and one snag is that the relevant environment for the SGE commands seems to get lost in the sudo calls, so I had to set up a few wrapper scripts like the following

$ cat /usr/local/sbin/qsub
#!/bin/sh
. /usr/local/sge6.2u5/default/common/settings.sh
exec qsub "$@"

and likewise for qdel and qstat. With this, the relevant part of /etc/sudoers is

Runas_Alias     JUPYTER_USERS = %student    # or whatever your jupyter users are
Cmnd_Alias      JUPYTER_CMD = /usr/local/sbin/qsub, /usr/local/sbin/qdel, /usr/local/sbin/qstat
jupyterhub  ALL=(JUPYTER_USERS) NOPASSWD:SETENV:JUPYTER_CMD

where "jupyterhub" is the user account (with disabled login) that runs the jupyterhub process. Note the SETENV flag which is needed so that the jupyterhub can use the "sudo -E" calls issued by batchspawner.

For Slurm, from looking at the batchspawner code I believe the batch commands to wrap would be sbatch, scancel, and squeue.

The final piece is that I needed to add the path to the jupyterhub install in the batch script template, which can be set in the jupyterhub configuration file (/etc/jupyterhub/jupyterhub_config.py by default). So in my case this looks like

c.GridengineSpawner.batch_script = '''#!/bin/bash
#$ -N jupyterhub-spawner
#$ -j yes
#$ -q {queue}
#$ -l h_rt={runtime}
#$ -pe smp {nprocs}
#$ -v {keepvars}

export PATH="/opt/anaconda/bin:$PATH"
{cmd}
'''

Of course, like Min said, the jupyterhub user should also be a member of the shadow group in order to access the authentication information.

Because the jupyterhub/anaconda install is not in the default path for root, I have a small wrapper script like

$ cat /usr/local/sbin/wrap-jupyterhub
#!/bin/sh
PATH="/opt/anaconda/bin:$PATH"
export PATH
exec jupyterhub "$@"

so that I can run it with the command "sudo -u jupyterhub /usr/local/sbin/wrap-jupyterhub".

Cheers,
Frank