Register external IPython kernel with jupyter notebook?

[Edward Banner]

Hi,

I am attempting to associate a jupyter notebook with a running IPython kernel. This kernel is running outside of jupyter (accomplished by running the following code in a python process):

import IPython

IPython.embed_kernel()

which produces the following output

To connect another client to this kernel, use:

    --existing kernel-9692.json

I can attach to this kernel through jupyter console and jupyter qtconsole, but not with jupyter notebook. As referenced in this issue this is not because of any inherent technical limitation, but rather because it would be confusing from a UI perspective.

I believe the first step would be registering the running kernel with the jupyer notebook server so that the kernel is returned on GET localhost:8888/api/kernels. The hope being then that I could point an existing jupyter notebook to use that kernel (I haven't figured out how to do that as it doesn't appear to be documented on this page).

How can I achieve this first step of registering this IPython kernel with a running jupyter notebook server?

Thanks in advance.

[Edward Banner]

[Edward Banner]

From what I can gather, it seems the way to accomplish this would be through either making a custom server extension or using the jupyter kernel gateway.

Regarding the latter approach I think I would need to make a custom manager which overrides start_kernel() (here's an example I found) to connect to an existing IPython kernel rather than starting a branch new one.

Can anyone comment on which approach would be the better one?

Thanks.

[Roland Weber]

Hello Edward,

through either making a custom server extension or using the jupyter kernel gateway.

The kernel gateway in its websocket personality doesn't work with notebooks at all.

In its notebook personality, it runs a configured notebook with one kernel to serve HTTP requests.

Since your declared intention is to associate a notebook with a kernel you started, I don't think that KG is of any use in your scenario.

cheers,

  Roland

[Edward Banner]

Hi Roland,

Thanks for your response. Your suggestion makes sense. But does that mean going with a custom server extension is the way to go?

I spent yesterday attempting to accomplish this solely though jupyter configuration options. For a running kernel with information

{
  "shell_port": 63391,
  "iopub_port": 63394,
  "stdin_port": 63392,
  "control_port": 63393,
  "hb_port": 63396,
  "ip": "127.0.0.1",
  "key": "fff108f8-b1bf9b17aa337ce7d3d14565",
  "transport": "tcp",
  "signature_scheme": "hmac-sha256",
  "kernel_name": ""
}

I set the following variables in jupyter_notebook_config.py.

c.ConnectionFileMixin.connection_file = '/Users/ebanner/Library/Jupyter/runtime/kernel-10962.json'
c.ConnectionFileMixin.control_port = 63393
c.ConnectionFileMixin.iopub_port = 63394
c.ConnectionFileMixin.stdin_port = 63392
c.ConnectionFileMixin.hb_port = 63396
c.ConnectionFileMixin.ip = '127.0.0.1'
c.ConnectionFileMixin.shell_port = 63391
c.ConnectionFileMixin.transport = 'tcp'

However when I created a new notebook I get errors like

[I 12:53:44.981 NotebookApp] KernelRestarter: restarting kernel (1/5)
[D 12:53:44.982 NotebookApp] Starting kernel: ['/Users/ebanner/.anaconda/envs/py36/bin/python', '-m', 'ipykernel_launcher', '-f', '/Users/ebanner/Library/Jupyter/runtime/kernel-f746726f-0d27-4d63-b520-2f1c84bf5196.json']
[D 12:53:44.986 NotebookApp] Connecting to: tcp://127.0.0.1:63393
Traceback (most recent call last):
  File "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-123>", line 2, in initialize
  File "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 448, in initialize
    self.init_sockets()
  File "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 238, in init_sockets
    self.shell_port = self._bind_socket(self.shell_socket, self.shell_port)
  File "/Users/ebanner/.anaconda/envs/py36/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 180, in _bind_socket
    s.bind("tcp://%s:%i" % (self.ip, port))
  File "zmq/backend/cython/socket.pyx", line 495, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:5653)
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:10014)
zmq.error.ZMQError: Address already in use

until it hits the maximum number of retries. Presumably this is because the ipykernel command attempts to create a *new* socket but since it already exists it fails to do so.

Thinking that I could "trick" jupyter into not using ipykernel and just use the existing sockets I set

c.KernelManager.kernel_cmd = ['/Users/ebanner/.anaconda/envs/py36/bin/python', '/Users/ebanner/bar.py']

where bar.py contains

while True:
    pass

After creating another new notebook I get

[D 12:59:37.283 NotebookApp] Opening websocket /api/kernels/9a6c1e7d-12c4-4d98-a0c6-1f227627f784/channels
[D 12:59:37.283 NotebookApp] Connecting to: tcp://127.0.0.1:63391
[D 12:59:37.284 NotebookApp] Connecting to: tcp://127.0.0.1:63394
[D 12:59:37.284 NotebookApp] Connecting to: tcp://127.0.0.1:63392

which gave me hope! But then when I tried to execute a cell it just hangs. I did notice that http://localhost:8888/api/kernels has the following information.

[
    {
        id: "9b260abf-072f-48e2-b5f0-213d934418a2",
        name: "python3",
        last_activity: "2018-02-13T21:10:41.460182Z",
        execution_state: "starting",
        connections: 0
    }
]

The execution_state being "starting" and connections being at 0 makes me think that bar.py isn't quite doing enough.

Digging into the code I discovered that the object returned from the call to ipykernel is a subprocess.Popen() object but all I could find is that Popen.poll() is called on it but I couldn't find any other interfaces it has to support.

I also tried setting c.NotebookApp.kernel_manager_class to my own custom kernel manager class

from tornado import gen, web
from jupyter_client import KernelManager
from notebook.services.kernels.kernelmanager import MappingKernelManager

class ExistingMappingKernelManager(MappingKernelManager):
    """A KernelManager that just connects to an existing kernel."""

    @gen.coroutine
    def start_kernel(self, kernel_id=None, path=None, **kwargs):
        kernel_id = 1
        km = KernelManager(kernel_name='python3')
        kc = km.client()
        kc.load_connection_file('/Users/ebanner/Library/Jupyter/runtime/kernel-10962.json')
        kc.start_channels()
        try:
            kc.wait_for_ready()
        except RuntimeError:
            kc.stop_channels()
            raise
        raise gen.Return(kernel_id)

but when I create a new notebook I get

[E 13:03:58.692 NotebookApp] Unhandled error in API request
    Traceback (most recent call last):
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/base/handlers.py", line 516, in wrapper
        result = yield gen.maybe_future(method(self, *args, **kwargs))

      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
        value = future.result()
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 4, in raise_exc_info
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run
        yielded = self.gen.throw(*exc_info)
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/services/sessions/handlers.py", line 75, in post
        type=mtype))
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
        value = future.result()
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 4, in raise_exc_info
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1069, in run
        yielded = self.gen.send(value)
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py", line 81, in create_session
        self.save_session(session_id, path=path, name=name, type=type, kernel_id=kernel_id)
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py", line 125, in save_session
        return self.get_session(session_id=session_id)
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py", line 170, in get_session
        return self.row_to_model(row)
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py", line 209, in row_to_model
        raise KeyError
    KeyError
[E 13:03:58.708 NotebookApp] {
      "Host": "localhost:8888",
      "Connection": "keep-alive",
      "Content-Length": "93",
      "Accept": "application/json, text/javascript, */*; q=0.01",
      "Origin": "http://localhost:8888",
      "X-Requested-With": "XMLHttpRequest",
      "X-Xsrftoken": "2|e5e3448b|a67102a2af982e0bece9d040d46b1c87|1516914005",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36",
      "Content-Type": "application/json",
      "Referer": "http://localhost:8888/notebooks/Untitled74.ipynb?kernel_name=python3",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept-Language": "en-US,en;q=0.9",
      "Cookie": "_xsrf=2|e5e3448b|a67102a2af982e0bece9d040d46b1c87|1516914005; username-localhost-8889=\"2|1:0|10:1517089627|23:username-localhost-8889|44:M2NiNzEzOGRjOGVhNDhlMzk0YjRlZGI2OWQ4M2RkZjk=|fcb4100a390a3d9608150659dc573ec135609a1e778f386e293b029e1fe016e3\"; username-localhost-8890=\"2|1:0|10:1518302103|23:username-localhost-8890|44:OTdjNGRhZGRjYTRlNGMxZjg0YzY4ZDIzZjU1NjEzYTQ=|6c9c7647d95641d6f2d7375d9d45130a96e9ed9ec690b2bca01b6652c23ccd05\"; username-localhost-8888=\"2|1:0|10:1518555822|23:username-localhost-8888|44:M2M3OTdhYThiNjQ2NDE3ZWEyZjA3Mjk2MzdjMjZlMmI=|e2c3fe789c16638c5fa4573a9d7626b87a00cb4d33ed1488122c6543b8832398\""
    }
[E 13:03:58.710 NotebookApp] 500 POST /api/sessions (::1) 289.25ms referer=http://localhost:8888/notebooks/Untitled74.ipynb?kernel_name=python3

Presumably this is because I failed to call super(MappingKernelManager, self).start_kernel(**kwargs) which presumably does some necessary setup work that I skipped.

At the end of the day I *did* get a very hacky solution working though with the following code snippet in a jupyter notebook

from jupyter_client import BlockingKernelClient

client = BlockingKernelClient()
client.load_connection_file('/Users/ebanner/Library/Jupyter/runtime/kernel-10962.json')

Then I can do client.execute_interactive(code) as many times as I want!

One way to go is to make a magic command out of this which wraps cell code and passes it to client.execute_interactive() so I don't have to type client.execute_interactive() each time. However I still need to insert the magic into every single code cell, which I would rather not do.

So my question is can I attach a new jupyter notebook to an external IPython kernel solely though configurables? Am I close? If not what would be the best way to go? The closest example I can find to study is nb2kg which allows one to attach a jupyter notebook to a remote kernel. However the kernels were still created by a jupyter notebook server so I am afraid it is still not close enough to help me solve my use case.

I truly appreciate any help as I think I am running out of ideas to try.

[Edward Banner]

Arg! I can't seem to get formatting down on google forums. I apologize. Here is my response in full.

However the execution_state being stuck at "starting" and connections being at 0 make me think that bar.py isn't quite doing enough.

Digging into the code a bit I found the object returned from the call to ipykernel is a subprocess.Popen() object. But I couldn't find anywhere else in the code except KernelManager.is_alive() where Popen.poll() is called on it.

I also tried creating a custom kernel manager class and set

c.NotebookApp.kernel_manager_class = 'kernelmanager.ExistingMappingKernelManager'

where kernelmanager.ExistingMappingKernelManager is the class

from tornado import gen, web
from jupyter_client import KernelManager
from notebook.services.kernels.kernelmanager import MappingKernelManager

class ExistingMappingKernelManager(MappingKernelManager):
    """A KernelManager that just connects to an existing kernel."""

    @gen.coroutine
    def start_kernel(self, kernel_id=None, path=None, **kwargs):
        kernel_id = 1
        km = KernelManager(kernel_name='python3')
        kc = km.client()
        kc.load_connection_file('/Users/ebanner/Library/Jupyter/runtime/kernel-10962.json')
        kc.start_channels()
        try:
            kc.wait_for_ready()
        except RuntimeError:
            kc.stop_channels()
            raise
        raise gen.Return(kernel_id)

but when I create a new notebook I get the error

[E 13:08:17.169 NotebookApp] Unhandled error in API request                                                                                                                                            
    Traceback (most recent call last):                                                                                                                    
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/base/handlers.py", line 516, in wrapper                                                                            
        result = yield gen.maybe_future(method(self, *args, **kwargs))                                                                                                  
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run                                                                          
        value = future.result()                                                                                                                                                                        
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result                                                                                          
        raise_exc_info(self._exc_info)                                                                                                                      
      File "<string>", line 4, in raise_exc_info                                                                                                          
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run                            
        yielded = self.gen.throw(*exc_info)                                                                                                    
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/services/sessions/handlers.py", line 75, in post                                                                            
        type=mtype))                                                                                                                                                                                
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run                                                                  
        value = future.result()                                                                                                                                                                        
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result                                                      
        raise_exc_info(self._exc_info)                                                            
      File "<string>", line 4, in raise_exc_info                                                                                                                              
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/tornado/gen.py", line 1069, in run                                                        
        yielded = self.gen.send(value)                                                            
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py", line 81, in create_session                                                            
        self.save_session(session_id, path=path, name=name, type=type, kernel_id=kernel_id)                                
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py", line 125, in save_session                                                              
        return self.get_session(session_id=session_id)                                              
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py", line 170, in get_session
        return self.row_to_model(row)                                                    
      File "/Users/ebanner/.anaconda/lib/python3.6/site-packages/notebook/services/sessions/sessionmanager.py", line 209, in row_to_model                                                              
        raise KeyError                                                                                                    
    KeyError                                                                                                                                                    
[E 13:08:17.171 NotebookApp] {                                                                              
      "Host": "localhost:8888",                                                                                          
      "Connection": "keep-alive",                                                                  
      "Content-Length": "96",                                                            
      "Accept": "application/json, text/javascript, */*; q=0.01",                                                
      "Origin": "http://localhost:8888",                                                          
      "X-Requested-With": "XMLHttpRequest",                                                                                                                      
      "X-Xsrftoken": "2|e5e3448b|a67102a2af982e0bece9d040d46b1c87|1516914005",                                                                                                                                                      
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36",                                                                                      
      "Content-Type": "application/json",                                                                                                                
      "Referer": "http://localhost:8888/notebooks/Kernel%20Client.ipynb",                                                                                                                    
      "Accept-Encoding": "gzip, deflate, br",                                                                                                                          
      "Accept-Language": "en-US,en;q=0.9",                                                                                                                                    
      "Cookie": "_xsrf=2|e5e3448b|a67102a2af982e0bece9d040d46b1c87|1516914005; username-localhost-8889=\"2|1:0|10:1517089627|23:username-localhost-8889|44:M2NiNzEzOGRjOGVhNDhlMzk0YjRlZGI2OWQ4M2RkZjk=|fcb4100a390a3d9608150659dc573ec135609a1e778f386e293b029e1fe016e3\"; user

name-localhost-8890=\"2|1:0|10:1518302103|23:username-localhost-8890|44:OTdjNGRhZGRjYTRlNGMxZjg0YzY4ZDIzZjU1NjEzYTQ=|6c9c7647d95641d6f2d7375d9d45130a96e9ed9ec690b2bca01b6652c23ccd05\"; username-localhost-8888=\"2|1:0|10:1518555822|23:username-localhost-8888|44:M2M3OTdhYTh
iNjQ2NDE3ZWEyZjA3Mjk2MzdjMjZlMmI=|e2c3fe789c16638c5fa4573a9d7626b87a00cb4d33ed1488122c6543b8832398\""

                                                     
    }                                  

presumably because I did not call super(MappingKernelManager, self).start_kernel(**kwargs) which presumably does some important setup work.

I *did* get a hacky solution "working" by the end of the day, though! I put the following code snippet in a jupyter notebook.

from jupyter_client import BlockingKernelClient

client = BlockingKernelClient()
client.load_connection_file('/Users/ebanner/Library/Jupyter/runtime/kernel-10962.json')

client.start_channels()

Then I am able to do client.execute_interactive(code) as many times as I want!

Using this approach one way forward would be to create a magic command that wraps code in a call to client.execute_interactive(). However for a single notebook that would require having to put the magic inside every code cell, which I would rather avoid if possible.

So my question is can I start a new jupyter notebook and attach it to a running IPython kernel with configurables alone? If so am I close? If not what would be the most promising direction to proceed? The closest example I can think to study is nb2kg which allows a jupyter notebook to attach to a remote kernel. However that kernel was started by a jupyter notebook so it still might not be close enough for my use case to solve my problem

I truly appreciate any help I can get as I am running out of ideas.

<span

[Edward Banner]

Looking through the code a bit more I found this code snippet in jupyter_client/manager.py:

def _launch_kernel(self, kernel_cmd, **kw):
    """actually launch the kernel

    override in a subclass to launch kernel subprocesses differently
    """
    return launch_kernel(kernel_cmd, **kw)

I know the PID of the running kernel process. If I could somehow do a subprocess.Popen() on that existing process and return a reference to that, I think that would work.

Unfortunately I can't anything related to doing a subprocess.Popen() on an *existing* process...

[Roland Weber]

Hi Edward,

I'm not going to read up on everything you posted here :-)

I know the PID of the running kernel process. If I could somehow do a subprocess.Popen() on that existing process and return a reference to that, I think that would work.

Try starting a subprocess that joins or polls on the running kernel. You might also have to forward interrupts, unless you can overwrite the methods for interrupting and terminating the kernel process. The pipes are only used to check if the kernel process is still running, afaik.

Good luck,

  Roland

[Edward Banner]

Hi Roland,

Thanks for the suggestion. I wound up creating a hacky kernel manager which seems to do the job.

https://github.com/ebanner/extipy

The only caveat is that you have to disable message authentication to get it to work because jupyter was complaining that the expected message signature didn't match.