How do communications beetween server and client work in Jupyter?

[Bruce Sherwood]

I've previously written here about the vpython module that makes it easy to display program-driven 3D animations in a Jupyter notebook.

I'm now trying to extend the vpython module to function if the user is not running in a notebook, in response to requests from users. I detect whether you're running in a notebook and do two different things depending on the answer. If not in a notebook, I still use a browser for display, using the GlowScript library that calls on WebGL.

I use webbrowser.open('http://localhost:{}'.format(HTTP_PORT)) to open a browser window and start an HTTPServer, which sends to the browser an html file containing the JavaScript program that will receive instructions from the server to create or modify VPython objects and use those instructions to tell the GlowScript library what to do. The communcations between server and browser in the notebook case uses a websocket, and in the new option I create an autobahn.asyncio.websocket in the server. What I run into is that code that works in the notebook case fails in the new case, due to various complications. I can solve a subset of these problems at the cost of the others failing, no matter what subset I choose. It's a Whack-a-mole game.

To take just one example: In order that a user's simple one-line program "box()" display a 3D box, and since I have no programmatic indication that the end of the user's program has been reached, I have to run the websocket in a thread; otherwise there is nothing to signal to the vpython module that output needs to be sent to the browser. But if the websocket is in a thread, there are problems of handling event-driven function calls, because they count as being inside the thread, whereas user-program-driven function calls are not in a thread. Etc.

So my ill-formed question is this: Can someone summarize the Jupyter communications machinery in the context of my difficulty in doing something similar? For example, in the notebook case I don't see any difference in the behavior of user-driven function behavior and event-driven function behavior.

Bruce

[Thomas Kluyver]

Hi Bruce,

On 26 April 2017 at 21:30, Bruce Sherwood <bruce.s...@gmail.com> wrote:

So my ill-formed question is this: Can someone summarize the Jupyter communications machinery in the context of my difficulty in doing something similar? For example, in the notebook case I don't see any difference in the behavior of user-driven function behavior and event-driven function behavior.

The kernel (the part where the user's code runs) runs an event loop. This handles both the user sending code to be executed, and other events such as widget callbacks. However, it doesn't use separate threads for this, so if the user executes a cell containing 'time.sleep(10)', the event loop can't handle anything else until it's finished.

I assume you want it to handle events while the user's code is still running? I think the only options for that are to use some kind of async framework (which may require user's code to be aware of it - i.e. no calling time.sleep()), or to use threads.

Thomas

[Bruce Sherwood]

Thanks.

Bruce

[Lawrence D’Oliveiro]

On Friday, April 28, 2017 at 12:34:56 AM UTC+12, takowl wrote:

The kernel (the part where the user's code runs) runs an event loop. This handles both the user sending code to be executed, and other events such as widget callbacks. However, it doesn't use separate threads for this, so if the user executes a cell containing 'time.sleep(10)', the event loop can't handle anything else until it's finished.

Have you looked at asyncio in Python 3.5 <https://docs.python.org/3/library/asyncio.html>? That would allow the user to execute coroutines that do things like

    await asyncio.sleep(«interval»)

without holding up the event loop.

[Bruce Sherwood]

The solution did indeed involve asyncio, thanks to detailed advice from John Coady.

The vpython module now (released two days ago) detects whether it is running in a Jupyter notebook or not. When running outside the Jupyter notebook environment it sets up an http server to serve image files (for textures on 3D objects) and font files (for extruding 3D text), and it sets up a websocket server for rapid two-way communications between the Python program and a JavaScript program in the browser that calls upon the GlowScript library for WebGL rendering. The websocket code uses asyncio. The most challenging issue was dealing with possible conflicts between the websocket thread and the main program, and the solution involved making sure the main program added data to be transmitted by the websocket only with "atomic" statements such as appending to a list or adding to a set. In some cases the websocket approach seems to run significantly faster than the Jupyter communication mechanism.