A dedicated channel for debugging

[Vladimir Lagunov]

Hello,

I’m looking for a reliable way of tunneling TCP sockets of pydevd through notebook’s websocket. General tunnel can be easily implemented using Comm, but debugger requires a special approach. Debugger is a tool that stops all threads of the process to let you debug them, including the main event loop thread. And, unfortunately, Comm and shell requests are handled in the same thread.

Actually I’ve succeeded in creating a working tunnel for pydevd. A special function moves handling of shell requests in a separate thread. All shell requests are either being processed in-place if they’re related to my protocol extension or being re-queued in a main IOLoop. Pydevd knows about that thread and never stops it. All that features are attained by monkey-patching different protected methods of kernel.shell_socket, kernel.shell_channel, kernel.session, etc. So that black magic is unsteady.

After exploring a code of ipykernel and notebook, I considered that the easiest way to make a tunnel for the debugger is to create a separate channel. Let’s name that channel «management». It should look like an analog of IOPubThread with CommManager and could be used for debugging, profiling, collecting metrics at runtime, etc.

What do you think about that idea? Do you see any obvious pitfalls around implementing the new channel? Will you accept a patch if me with my colleagues will implement it for notebook, ipykernel and probably for a gateway?

Thanks,
Vladimir

[Afshin T. Darian]

Dear Vladimir,

Thank you for writing about what you've been working! This is an exciting area to think about and it's gratifying seeing movement on it.

You are thinking along very similar lines to what other debugging efforts in Jupyter have considered. The emerging consensus is:

• Like you've discovered, the `comm` channel is not the most appropriate channel for this activity, but indeed the messages for debugging should happen on a ØMQ channel. The channel that current debugging efforts have been using for this is the `control` channel.
• Because a standard has come out of Visual Studio Code's work on this, adopting it increases the available ecosystem of both actual tools we could use and also just familiarity with the problem space: https://microsoft.github.io/debug-adapter-protocol/
• Here is the current state of affairs in supporting the Debug Adapter Protocol on the `control` channel: https://github.com/jupyter/jupyter_client/issues/446
• Here is the initial conversation about implementing a front-end for a debugger in JupyterLab: https://github.com/jupyterlab/team-compass/issues/5
• Here is a Python kernel, built by QuantStack, that has an initial implementation of Python debugging: https://github.com/QuantStack/xeus-python
• Here is work-in-progress JupyterLab extension that is not ready for public consumption, but is targeting the JupyterLab 2.0 release to support a first pass on debugging functionality: https://github.com/jupyterlab/debugger/

I hope some of these efforts are helpful to your work, or even inspire you to jump in and work on them. Some of the other folks who have been working on these things may have more to add.

Cheers!

-Darian

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/9a5fd364-6583-40a3-9796-01e92db2cab5%40googlegroups.com.

[Vladimir Lagunov]

Dear Darian,

Thank you for your links, they were extremely useful for me!

I’ve investigated all of them and found that Xeus kernel already moved handling of the control channel to a separate thread. That works fine in Xeus, so I propose to move the control channel to a separate thread in ipykernel too. What do you think?

Thanks,

Vladimir

To unsubscribe from this group and stop receiving emails from it, send an email to jup...@googlegroups.com.

[Sylvain Corlay]

Hi Vladimir,

Xeus developer here. I think that you understood really well the constraints with respect to concurrency for the debugger.

The choice of the Control channel

- the Shell channel is out of question because debug messages would end up being queued with the rest of the shell messages (such as execution requests and comm messages).

- while the Control channel does not have that requirement, which is actually the main reason for its existence as the goal was to be able to send shutdown or interrupt messages while code is queued for execution.

This is the reason why we decided to use Control as the main channel for debug messages.

The Concurrency Model and Xeus

An issue with the current state of ipykernel is that the concurrency model is that of an event loop, meaning that any blocking code execution triggered by a shell message would prevent control messages to be processed. What we want is to be able to add a breakpoint while code is running, for example in a long-running loop, and see the debugger stop at the next iteration of that loop. For this reason, we need to use a separate thread to process control messages.

Xeus is a C++ implementation of the Jupyter protocol. It is not a kernel but a library facilitating the implementation of kernels. Kernels built with xeus include xeus-cling (a C++ kernel), xeus-python (A Python kernel), and more. It has a pluggable concurrency model (which can be changed by providing one's own implementation of the "xserver" base class). For example, Kitware built a custom python kernel with xeus that integrates into the Qt event loop for their Slicer kernel...

Xeus-python, the xeus-based python kernel supports most of the features of ipykernel (rich display, widgets, auto-complete, rich tracebacks, quick-help etc), and provides a debugging backend for the messages of DAP (Debug Adaptor Protocol) over control messages as explained by Afshin earlier.

In order to route these control messages from the backend to the web front-end, several pull requests have been opened, and merged in jupyter_client, notebook, jupyter_server, jupyterlab. A really good summary of that work was done by Johan Mabille here: https://github.com/jupyter/jupyter_client/issues/446.

Another advantage of xeus-python is that it is a much smaller codebase to iterate upon. The entire kernel is about 3k lines of code with a good coverage of the features (rich display, widgets, auto-complete, rich tracebacks, quick-help), and we could iterate quickly. We are looking at moving xeus, xeus-python, and xeus-cling in a different github organization under the Jupyter governance.

The Ongoing Frontend Work

We are now working on the front-end component to that stack. This part is mostly done by Afshin Darian, Jeremy Tuloup, Borys Palka, and Johan Mabille.

In the screencast below, you can see the state that was demoed by Jeremy at EuroScipy a couple of weeks ago. 

There is still a lot to figure out with respect to the UX of the debugger, the lifecycle of debugging sessions etc, which may require more backend work as well (such as storing the debugger state etc). And we take help!

PyDev

PTVSD actually is a layer on the top of pydevd, which is the debugger also used by pycharm and eclipse. It seems that layers are only ever added... In fact, I think that in the end, once the whole front-end is complete, we could rework some aspects of that and directly use pydev (even though we make use of the debug adaptor specification for the frontend messaging).

Looking forward to hearing back from you!

Sylvain Corlay

Founder & CEO @ QuantStack

Scientific Software Developer @ Project Jupyter

sylvain...@quantstack.net +33 7 82 97 90 44

To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/ef50e13f-3ee4-4f52-94e3-fca1ed061280%40googlegroups.com.

[Sylvain Corlay]

You will find the screencast at the following url:

[Sylvain Corlay]

https://files.gitter.im/QuantStack/core_development/LeXW/demo.gif