jupyter lab on docker to connect to host (matlab) kernel

[Karsten Wiesner]

Hi all

I'm about to setup a docker image that runs
jupyterlab. I've found docker-stacks which made it super easy. My
Dockerfile looks like this:

FROM jupyter/datascience-notebook
USER jovyan
RUN jupyter labextension install @jupyterlab/google-drive

So everything is fine. The only missing thing is to have the host matlab
kernel available in jupyter lab on this docker container. Due to license
issues the matlab engine must run on the host system. On my host system
I've installed matlab engine for python and py-mat-bridge from callisto
so when I start jupyterlab or notebook on my host it finds the
matlabkernel. How can I route the matlabkernel to the jupyter lab on my
docker container so that it appears additional to the python,R and Julia
kernel on the docker container? This might be also useful for other connecting 

to other than matlab kernels on the host

thanks in advance,

Karsten

[MinRK]

That's a very interesting problem! One way that *might* work is to mount the matlab installation as a volume in the container. That would assume that the docker image is sufficiently compatible with the host system.

To work on this, I would skip JupyterHub and try to get it running with a single notebook container:

    docker run -v /path/to/matlab:/path/to/matlab -it my-image-name

If you get it working that way, you can add the volume in your `c.DockerSpawner.volumes` config.

Otherwise you are going to have to provide a way for containers to launch processes outside themselves on the host. This is a bit antithetical to containers, so may require a bit of work. You might need to run a service on the host that allows requesting kernels via HTTP, which you can then hook up in your container.

-Min

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+unsubscribe@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/5aba776b-dc1c-462d-b08a-bc1a8513257d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Karsten Wiesner]

Hi Min

thanks for the post. I think mounting the matlab installation would not work because the host is a mac and jupiter docker-stacks is based on linux.. Anyway we are searching for a solution where the host platform can be any OS. Your 2nd idea is what I've been thinking about too. But yes that is tricky. 1st: How to get  the IP Address of the host from within the docker container (running in bridge mode). 2nd.: setting up a remote connection to the kernel. I have found projects that are dealing with this: jupiter_kernel_gateway, rk from korniichuk and I have also tried to manually route ports to the container with docker run -p ... I'm a bit lost though. Isn't there a standard way to or best practise to let jupiter lab/notebook/ipython route to remote kernels?

Cheers,

Karsten

To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.

[Karsten Wiesner]

BTW.:

get the host ip into the docker container is easy: 

1.) add:   --add-host="host-machine:$(ipconfig getifaddr en0)"  to the docker run command

2.) at the container e.g. in a jupiter lab terminal > ssh user@host-machine

...  

[Adam Thornton]

This is the "ignore the problem and solve an easier one" version:

Are you doing stuff with Matlab that needs honest-to-goodness Matlab, or is GNU Octave good enough?  If Octave will do, then you no longer have licensing worries.

[Roland Weber]

Hello Karsten,

one problem you will encounter when using remote kernels is that the kernel doesn't have access to the file system you're seeing in Jupyter Lab. That's almost sure to confuse users. They can edit files in their browser, but then the kernel doesn't see them. The notebook files will work fine, because they're interpreted by Jupyter and only code snippets are sent for execution to the kernel. But if there are data files, or libraries you'd like to import from the kernel, they cannot be read.

Another potential problem are libraries that come with a notebook extension, like ipywidgets with widgetsnbextension. One piece has to be installed in the remote kernel, the other in Jupyter. As long as you control both environments, you can keep versions in sync. When you control only the Jupyter side, you have to rely on someone else on the remote side to sync with what you're doing. 

So no, there isn't a standard way or best practice for using remote kernels. Afaict, the default UI for Jupyter Lab and Notebooks implicitly assumes that kernels are local, and that's true in most cases. Installation instructions for extension packages make the same assumption. You can get remote kernels to work, but the user experience will suffer.

Jupyter Kernel Gateway is one way to provide remote kernels. There's a demo called "nb2kg" that shows how to run a notebook server with all kernels being remote on one gateway. I'm not aware of a KG demo that would mix local and remote kernels, nor of an adaptation for Jupyter Lab.

https://github.com/jupyter/kernel_gateway_demos/tree/master/nb2kg

cheers,

  Roland

[Karsten Wiesner]

Hi Adam

this is not about ignoring the problem ... in fact we want to establish a common platform of open source data science tools to a collaboration of people (jupiter, python, R, etc).   But we have a very huge amount of legacy code (matlab). In the future we tend to the open source languages but at least some of us need to run the legacy code for reference, verification etc. So it makes perfectly sense for us to enable tunnelling to a licensed matlab installation on the host - which I just got running ;-)

Cheers,

Karsten

[Karsten Wiesner]

Thanks all for the hints and insights. I just got it up an running (basically). 

@Roland: Your concerns are right but we just use a docker run -v mounted host filesystem/directory (otherwise one could try to work with sshfs but I have not tested this).

So my way of doing (using https://pypi.python.org/pypi/remote_ikernel) looks like:

Dockerfile:

FROM jupyter/datascience-notebook

USER root

RUN apt-get -y update

RUN apt-get -y install openssh-client

USER jovyan

RUN jupyter labextension install @jupyterlab/google-drive

RUN pip install remote_ikernel

On Host:

docker run -it --rm \

-p 8888:8888 \

--add-host="host-machine:$(ipconfig getifaddr en0)" \

-v $(pwd):/home/jovyan/work \

image-name \

start.sh /bin/bash

establish password less ssh tunnelling to host:

jovyan@46dcdfa9fd9b:

ssh-keygen -t rsa

<ret -> no password>

cat .ssh/id_rsa.pub | ssh xxxx@host-machine 'cat >> .ssh/authorized_keys'

<password>

on hostmachine obtain info about the kernel with jupyter kernelspec list --json and put it to:

jovyan@46dcdfa9fd9b:~$ remote_ikernel manage --add --kernel_cmd="xxxxxxxxxxxxxxx/matlab-jupyter/bin/python -m matlab_kernel -f {connection_file}" --name="matlab-on-host" --interface=ssh --host=xxxxx@host-machine --workdir='xxxxxxxxxxxxx-environment' --language=matlab

jovyan@46dcdfa9fd9b:~$ jupyter lab --ip='*' --port=8888 --no-browser

copy the url from the output to your host browser matlab kernel appears in jupiter lab - tested OK :-)

ToDo:

ssh and remote_ikernel manage stuff can't be done in the Dockerfile because the Host IP is known at build time. I must find a way to automatise this. 

A jupiter lab extension would be great that scans remote hosts for kernels. Then connect to kernels on demand. @ MinRK: Is something like this planned or ongoing?

Cheers,

Karsten