Problem spawning jupyterhub-singleuser - library path?

[rob moser]

Hi All,

I've been working on getting jupyterhub up and running, but I've hit a hitch at the point of spawning the single-user server - it seems to not be carrying the library paths over from the environment of the hub?  Here's the error message I'm getting in the logs:

En[D 2015-12-16 14:01:47.041 JupyterHub app:817] Proxy started and appears to be up
[I 2015-12-16 14:01:47.043 JupyterHub app:1054] JupyterHub is now running at http://134.114.138.133:8887/
14:01:54.832 - debug: [ConfigProxy] PROXY WEB /favicon.ico to http://localhost:8081
[D 2015-12-16 14:01:54.869 JupyterHub log:100] 304 GET /favicon.ico (@134.114.70.85) 14.33ms
14:02:02.096 - debug: [ConfigProxy] PROXY WEB /hub/login?next= to http://localhost:8081
[I 2015-12-16 14:02:02.200 JupyterHub spawner:363] Spawning jupyterhub-singleuser --user=rbm43 --port=37843 --cookie-name=jupyter-hub-token-rbm43 --base-url=/user/rbm43 --hub-prefix=/hub/ --hub-api-url=http://localhost:8081/hub/api --ip=localhost '--notebook-dir=~/.notebook' --debug
/opt/rh/python33/root/usr/bin/python3: error while loading shared libraries: libpython3.3m.so.1.0: cannot open shared object file: No such file or directory
[D 2015-12-16 14:02:02.247 JupyterHub spawner:219] Polling subprocess every 30s
[W 2015-12-16 14:02:12.283 JupyterHub base:290] User rbm43 server is slow to start
[I 2015-12-16 14:02:12.304 JupyterHub log:100] 302 POST /hub/login?next= (rbm43@134.114.70.85) 10189.37ms
[I 2015-12-16 14:02:12.305 JupyterHub login:79] User logged in: rbm43

I'm using the scl Software Collection to run python3 out of an alternate directory (because redhat's default python is older than the hills).  In the process from which I initially ran the hub:

# env
...
LD_LIBRARY_PATH=/opt/rh/python33/root/usr/lib64:...

# which python3
/opt/rh/python33/root/usr/bin/python3

# ls -sal /opt/rh/python33/root/usr/lib64/libpy*
   0 lrwxrwxrwx 1 root root      20 Mar 26  2015 /opt/rh/python33/root/usr/lib64/libpython3.3m.so -> libpython3.3m.so.1.0
2380 -rwxr-xr-x 1 root root 2433200 Dec  3  2013 /opt/rh/python33/root/usr/lib64/libpython3.3m.so.1.0
   4 -rwxr-xr-x 1 root root    3664 Dec  3  2013 /opt/rh/python33/root/usr/lib64/libpython3.so

...so the library is clearly present, and in the library path.  I can basically cut-and-paste the command out of the logs (ditching the troublesome quotes around the notebook-dir parameter) and get a different error that has nothing to do with missing libraries at least.  The only thing I can think of is that that library path isn't getting passed through to the single-user server when it spawns?  Is there a config setting I can use to pass it a library path, perhaps?  Or some other way around the problem?

Thanks for any advice,

     - rob.

For reference: I'm running RHEL6, and my jupyterhub_config.py file contains:

c.Authenticator.admin_users = {'rbm43', 'jgc53', 'jc33'}
c.JupyterHub.cookie_secret_file = '/home/ipython-p/jupyterhub_cookie_secret'
c.JupyterHub.debug_proxy = True
c.JupyterHub.extra_log_file = '/home/ipython-p/ju2.log'
c.JupyterHub.log_file = '/home/ipython-p/jupyterhub.log'
c.JupyterHub.log_level = 'DEBUG'
c.JupyterHub.pid_file = '/home/ipython-p/jupyterhub.pid'
c.JupyterHub.ssl_cert = '/etc/httpd/conf/ssl/hostname.crt'
c.JupyterHub.ssl_key = '/etc/httpd/conf/ssl/hostname.key'
c.LocalProcessSpawner.debug = True
c.PAMAuthenticator.debug = True
c.PAMAuthenticator.encoding = 'utf8'
c.PAMAuthenticator.service = 'system-auth.works.no.home_deploy'
c.Spawner.debug = True
c.Spawner.notebook_dir = '~/.notebook'

[MinRK]

JupyterHub only passes a whitelist of environment variables to the
single-user server subprocesses. LD_LIBRARY_PATH isn’t on that list,
so you may want to add it to env_keep:

c.LocalProcessSpawner.env_keep.append('LD_LIBRARY_PATH')

It may make sense to do this by default.

-MinRK

> (rb...@134.114.70.85) 10189.37ms

> --
> You received this message because you are subscribed to the Google Groups
> "Project Jupyter" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jupyter+u...@googlegroups.com.
> To post to this group, send email to jup...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jupyter/e8d9f364-de26-44ac-8531-f144ad97d8ee%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[rob moser]

On Thursday, December 17, 2015 at 6:40:57 AM UTC-7, Min RK wrote:

JupyterHub only passes a whitelist of environment variables to the
single-user server subprocesses. LD_LIBRARY_PATH isn’t on that list,
so you may want to add it to env_keep:

c.LocalProcessSpawner.env_keep.append('LD_LIBRARY_PATH')

Perfect; exactly the solution I needed.  (My server is now successfully progressing to the next error on the list, but I'll see what I can do with it myself before I come running for help.)

Thanks for your help!

     - rob.

[rob moser]

For what it's worth: the remaining problem I had was that I'd set:

c.Spawner.notebook_dir = '~/.notebook'

and that directory did not exist for all of my users.  I guess I expected it to be created automatically if it didn't exist, but that doesn't seem to be the case.  Don't know if that is behaving as intended, but if so then a config option that allowed it to be created would be nice.

Thanks again for your help,

     - rob.

[MinRK]

On Thu, Dec 17, 2015 at 6:41 PM, rob moser <rbm...@gmail.com> wrote:
> For what it's worth: the remaining problem I had was that I'd set:
>
> c.Spawner.notebook_dir = '~/.notebook'
>
> and that directory did not exist for all of my users. I guess I expected it
> to be created automatically if it didn't exist, but that doesn't seem to be
> the case. Don't know if that is behaving as intended, but if so then a
> config option that allowed it to be created would be nice.

I've gone back and forth on this. In fact, the single-user server used
to create its notebook directory if it didn't already exist. We
removed this because it hid bugs - if you misspell the notebook
directory, it will happily create a new directory in the wrong path,
rather than telling you that you have asked it to run notebooks in a
directory that doesn't exist.

Adding an option is certainly a possibility.

-MinRK

>
> Thanks again for your help,
>
> - rob.
>
>
> On Thursday, December 17, 2015 at 8:51:25 AM UTC-7, rob moser wrote:
>>
>>
>>
>> On Thursday, December 17, 2015 at 6:40:57 AM UTC-7, Min RK wrote:
>>>
>>> JupyterHub only passes a whitelist of environment variables to the
>>> single-user server subprocesses. LD_LIBRARY_PATH isn’t on that list,
>>> so you may want to add it to env_keep:
>>>
>>> c.LocalProcessSpawner.env_keep.append('LD_LIBRARY_PATH')
>>>
>>
>> Perfect; exactly the solution I needed. (My server is now successfully
>> progressing to the next error on the list, but I'll see what I can do with
>> it myself before I come running for help.)
>>
>> Thanks for your help!
>>
>> - rob.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Project Jupyter" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jupyter+u...@googlegroups.com.
> To post to this group, send email to jup...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jupyter/99c9f7bc-96d6-4bc2-9946-acd4a69f8343%40googlegroups.com.

[Fernando Perez]

On Fri, Dec 18, 2015 at 2:47 AM, MinRK <benja...@gmail.com> wrote:

On Thu, Dec 17, 2015 at 6:41 PM, rob moser <rbm...@gmail.com> wrote:
> For what it's worth: the remaining problem I had was that I'd set:
>
> c.Spawner.notebook_dir = '~/.notebook'
>
> and that directory did not exist for all of my users.  I guess I expected it
> to be created automatically if it didn't exist, but that doesn't seem to be
> the case.  Don't know if that is behaving as intended, but if so then a
> config option that allowed it to be created would be nice.

I've gone back and forth on this. In fact, the single-user server used
to create its notebook directory if it didn't already exist. We
removed this because it hid bugs - if you misspell the notebook
directory, it will happily create a new directory in the wrong path,
rather than telling you that you have asked it to run notebooks in a
directory that doesn't exist.

Adding an option is certainly a possibility.

 

In the spirit of separation of concerns, I'd suggest this should be the business of the OS deployment, not ours... This is what the user creation tool on the system should do, populating the home directories according to the local requirements.

I'd instead suggest adding a very visible error dialog or other behavior, such that if it's started with a non-existing directory, the problem is immediately obvious and can be corrected right away without lost time.  But the silent auto-creation seems a worse solution than the original problem to me...

Cheers

f