Getting values from Javascript to Notebook

[Barry Demchak]

Hi --

I'm trying to get values from Javascript to a running Notebook ... think of the values as telemetry data coming from the client browser/PC.

The idea is for the Notebook to trigger the Javascript in one cell and then chew on the data in Python (3.7) in another cell.

This works when I single step the cells, but not when I run all cells in one swipe.

I think the reason is that the kernel execute/shell queue is serviced only once the kernel goes idle. When I'm single stepping through the cells, kernel idle would occur at the end of each cell. But when I'm running all cells, idle doesn't occur until after I needed the value. Bad news.

Here are my three cells (as a toy example, actual notebook file attached):

#1:

from IPython.display import HTML

javascript = f'''<script type="text/Javascript">
var my_var = 10
</script>
'''

HTML(javascript)

#2:

javascript = f'''<script type="text/Javascript">
IPython.notebook.kernel.execute("py_var = " + my_var)
my_var = my_var + 1
</script>
'''

HTML(javascript)

#3:

print(py_var)

To see this, run #1, followed by #2, and followed by #3. Rightly, the output of #3 would be 10. If I run #2 and then #3 again, I would (rightly) get 11.

Then, if I run "#2 and below", the output of #3 is still 11, even though my_var actually held 12 and there was a pending 'py_var=12' in the execute queue.

This is wrong. I really hope for the execute queue to be serviced before #3 runs ... not OK to single step through cells.

Is there a way to get the execute queue serviced between cells even when running "#2 and below"?

I have gone through kernel services code (https://github.com/jupyter/notebook/blob/master/notebook/static/services/kernels/kernel.js) and see how the execute() works from the client side. I think I'm really asking for explicit flushing of the shell channel at the server side ... not so obvious where to look for this.

Is there any way to finesse this execute queue processing?

Alternatively, is there a custom widget protocol that applies?

Alternatively, should I be creating my own web socket and protocol for this??

(I well understand how the shell protocol servicing strategy may not be suited to this, but I'm not sure where I should be going next.)

Thanks!

[Travis DePrato]

The issue you're running into is that the Jupyter protocol is fully serial. I've opened a GitHub issue/RFC that would make some things not-serial, but alas, the state of the world is that things are serial.

So, essentially, what happens is this:

• You click run all in the UI
• The frontend sends 3 execute_request messages to the Jupyter server (and ultimately the kernel)
• The kernel starts executing request one (the remaining 2 are queued)
• The JavaScript is sent to the frontend which sends another execute_request message.
• This new execute_request is queued behind the other 2 that were pending before
• The kernel finishes request one, and starts request two
• ...then request three...
• Finally, after request three, the kernel handles the execute_request that was initiated by your JavaScript code

From the kernel's perspective, it executes everything in order.

I don't see this being likely to change. The kernel protocol is actually architected in a way to allow concurrent users, so the concept of sending a request to the frontend is ill-defined since there may be zero, one, or more frontends actively connected.

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/8bc7dcae-d6c9-409f-a3b5-457d9e815554o%40googlegroups.com.

--

Travis DePrato (he/him/his)

Experiential Learning Software Architect

University of Michigan '19

Black Lives Matter

[Barry Demchak]

Hi, Travis –

 

Ohhhhhh …. This larger picture makes perfect sense. I had supposed that this all had to do with atomicity of a cell, but your explanation is much better.

 

So, to fix my situation, there would need to be two queues. One for executing cells and another for the rest (including my requests). I wouldn’t want to say that’s easy, as the devil is always in the details.

 

Clearly, that’s not happening in my timeframe. There may be other ways.

 

1. At both the Javascript and Python level, attach to the existing ZeroMQ and add another channel that I can stuff/harvest at will. This would require figuring out how to access the ZeroMQ in both domains, and then deal with the asynchronous code I’d have to layer on top. This might be promising … for the Python side, I see a bunch of IPython.kernel.zmq calls that I might track down: https://ipython.org/ipython-doc/3/py-modindex.html, though I note they are not here: https://ipython.readthedocs.io/en/stable/api/index.html. I can see I might need to investigate jupyter_client.session.Session.

 

2. If I could open up a private TCP port on the server, I could create a private Redis or ZeroMQ and then do the asynchronous Javascript and Python. I have my doubts about being able to do that, as I’ll bet that the underlying Linux firewall would block that. Might be interesting to double check that.

 

3. I could create a Redis on an external (Amazon) server and then do #1. It wouldn’t cost much, but it would add complexity.

 

BTW, it does no good to try executing my Toy Example from an outer notebook (i.e., %run ToyExample.ipynb) … it seems to run in the caller’s context, and likely queues those called cells in the way you describe.

 

Do you have any reaction to any of these? (A scan of PyPI tells me that #2 doesn’t exist for Jupyter Notebook, though I have more poking to do.)

 

Thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CANWNHrY0UV--Jb-wZnHzQX9acFGH4vEa5hXqzxrrC2Vab6jehg%40mail.gmail.com.

[Travis DePrato]

I'd recommend installing a server extension that sets up a websocket (you'd have to plug into the notebook server's tornado code) and then having the JavaScript communicating over the websocket. You're less likely to run afoul of firewall issues that way. Best of luck!

Best luck!

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/0b0001d64be0%24bc2c5db0%2434851910%24%40tpsoft.com.

[Barry Demchak]

Thanks, Travis …

 

I’m trying to avoid server extensions or firewall changes. I don’t mean to restrict execution of this package to just Jupyters that I control (e.g., Colab).

 

So, I have investigated setting up separate channels to/from Amazon Elasticache (Memcached and Redis), but Amazon limits them to internal VPN usage only. Probably because they’re impossible to secure.

 

I’m investigating piggy backing onto Jupyter’s ZMQ. We’ll see if that’s an option. (BTW, the ZMQ implementation for Javascript is very odd because Javascript doesn’t naturally do sockets. They’re piggybacking onto Flash’s sockets, of all things.)

 

And the other choice is to create a pipe service on my own server, and have both Python and Javascript call it … firewalls notwithstanding.

 

All of this is giving me good material for a conversation with the Jupyter people, should I ever get the chance.

 

Learning a lot …

 

(Comments welcome!)

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CANWNHraS-bc46cFw%3D2%2B0NcqWYJ2smUiaRAjYuHWERRrO4c-RuA%40mail.gmail.com.