Leo and Jupyter

[Thomas Passin]

Users of Leo who also know about Jupyter and Jupyter notebooks sometimes see a resemblance.  Now that the Viewrendered3 plugin is working (though still a beta version), it is possible to render code and the resulting text and plots in the same node, which looks even more like Jupyter.  Leo already has a limited ability to import and export in Juypter Notebook format.

Jupyter Notebooks have become a widespread way to share scientific work in a way that allows for immediate duplication of the calculations.  The Jupyter system is especially strong at creating graphical, even interactive plots that embed themselves right into the notebook.

This thread is for discussion about how or whether Leo might be able to play with Jupyter.

I think there are basically four general ways that Leo could interact with Leo.  I'll put them in my next post. See what you think.

[Thomas Passin]

On Monday, March 30, 2020 at 12:06:47 PM UTC-4, Thomas Passin wrote:

This thread is for discussion about how or whether Leo might be able to play with Jupyter.

I think there are basically four general ways that Leo could interact with Leo.  I'll put them in my next post. See what you think.

 There are at least four ways that Leo could be used with Jupyter:

1. **Import a Jupyter notebook into Leo**.  A notebook could be viewed and rendered in Leo.  It is not clear why this would be better than using a notebook viewer program, although Leo might make navigation between cells easier than with most other programs.  Rendering these notebooks in Leo would be desirable for good readability.  It could be done with a modification to Viewrendered 3, or something similar.

2. **Export a Leo tree to a Jupyter notebook**.  If Leo could export a tree to a Jupyter-compatible notebook, then work that was done in Leo could be shared with others, and the well-developed software that already exists to view and author notebooks could be used with these Leo-originated notebooks.  Exporting would play a role similar to creating Sphinx documents from a Leo tree - it would be a one-way sharing of the rendering and execution results with others.

3. **Combine importing and exporting**.

4. **Build a Jupyter front end client for Leo**.  Presumably this would be in the form of a Leo plugin.  This would have the advantage that the plugin could make use of Jupyter kernels developed for other languages.  Leo would not need to re-invent that wheel. A Jupyter client would not necessarily duplicate everything provided by other clients, such as JupyterLab or the QTConsole.

[Thomas Passin]

On Monday, March 30, 2020 at 12:08:15 PM UTC-4, Thomas Passin wrote:

On Monday, March 30, 2020 at 12:06:47 PM UTC-4, Thomas Passin wrote:

This thread is for discussion about how or whether Leo might be able to play with Jupyter.

I think there are basically four general ways that Leo could interact with Leo.  I'll put them in my next post. See what you think.

 There are at least four ways that Leo could be used with Jupyter:

[snip]

4. **Build a Jupyter front end client for Leo**.  Presumably this would be in the form of a Leo plugin.  This would have the advantage that the plugin could make use of Jupyter kernels developed for other languages.  Leo would not need to re-invent that wheel. A Jupyter client would not necessarily duplicate everything provided by other clients, such as JupyterLab or the QTConsole.

A Jupyter server handles just one kind of cell - those that want code executed.  The server executes that code and sends it back to the client.  The client figures out how to display and persist it.  A server runs a "kernel", and different kernels can handle different languages.  If you want Python, you use a Python kernel.  If you want R, you use an R kernel.

[Thomas Passin]

Leo Notebooks are easy to imagine, but they would never gain traction because Jupyter is too widespread already.

On Monday, March 30, 2020 at 12:13:04 PM UTC-4, Thomas Passin wrote:

On Monday, March 30, 2020 at 12:08:15 PM UTC-4, Thomas Passin wrote:

On Monday, March 30, 2020 at 12:06:47 PM UTC-4, Thomas Passin wrote:

This thread is for discussion about how or whether Leo might be able to play with Jupyter.

I think there are basically four general ways that Leo could interact with Leo.  I'll put them in my next post. See what you think.

 There are at least four ways that Leo could be used with Jupyter:

[snip]

Since Leo outlines consist of a graph of nodes, it seems likely that the nodes could be mapped to Jupyter cells in some manner.  The nodes in a subtree could be rendered, for example by Viewrendered3 or a similar but  specialized plugin.

There is no reason to try to duplicate Jupyter Notebook capabilities as Leo outlines, because Leo-format notebooks would never gain traction.  Jupyter notebooks are already too widespread.

The questions are: what benefit could be gotten by interfacing Leo and Jupyter, and how complicated would it be?

[Thomas Passin]

On Monday, March 30, 2020 at 12:15:44 PM UTC-4, Thomas Passin wrote:

The questions are: what benefit could be gotten by interfacing Leo and Jupyter, and how complicated would it be?

It seems to me that it would be essential to be able to embed images, interactive plots, and the like that are the result of executing the code in code cells.  For working with notebooks from Jupyter users, and especially if they are to be executed inside Leo, the most apparent matters that would need to be dealt with are the following.

1. **Correct handling of Jupyter's "Magic" %-expressions.** Notebooks that were started in Leo would not necessarily need to include them, but imported notebooks are likely to use them.  This would matter if the notebook were re-executed in Leo.

2. **Bundling of execution output data**. Images, interactive graphs, etc.  These would need to be put into a location, and into a form that Leo's renderer can work with.
 

[Edward K. Ream]

On Mon, Mar 30, 2020 at 11:08 AM Thomas Passin wrote:

> There are at least four ways that Leo could be used with Jupyter:

All four look like the hard way. I tried the hard way several times last year and I'm not eager for more :-)

I am interested in a Jupyter plugin for Leo only if it leverages another project's bridge code. Bokeh might provide the necessary leverage.  See this page. I'll be investigating this topic next.

Edward

[Thomas Passin]

Well, here's the thing.  How would someone use a Jupyter Notebook with Leo?

1. To look at someone else's notebook.  A notebook viewer would do this job at least as well as Leo.

2. To create a notebook in Leo for someone else to look at.  It they don't need to be able to execute the code, we could probably serialize VR3's output to a notebook as things stand.  No server needed.

3.  To let someone else view and execute the code.  We could probably still serialize VR3 subtrees to do this, but it would be harder.  Note that if we create the notebook, we don't have to understand or execute the %-magic expressions.  They are a convenience, not an essential.  Once it's been executed and the notebook saved, it's a tricky question whether or not we could use it again in Leo.  For that, we would need a much better importer than we have now.

4. To be able to execute code in some language we don't support, like R or Stata, but Jupyter has a kernel for.  It would probably be easier to get VR3 working with this other language instead.

5.  To be able to execute the code in someone else's notebook.  To do this, we would need to be able to handle the %-magic expressions.  Obviously that could be done, but I fancy it will take a lot of doing.

6. To work collaboratively with others.  This is one benefit of using a Jupyter server, but it would be the hardest to achieve in Leo.  And I don't think many of us need to do in very often.

Feel free to add to this list, but so far I don't see a strong reason why better import/export wouldn't be enough for almost all needs.  No client-server needed.  And for client-server to be useful, we would need better import-export capabilities anyway. 

The current Leo export/import machinery basically only handles text nodes,as far as I can see.  There's really not much point in a text-only notebook.  You need the code and results to make Jupyter-style notebooks make sense.

[Edward K. Ream]

On Mon, Mar 30, 2020 at 12:33 PM Thomas Passin <tbp1...@gmail.com> wrote:

>...I don't see a strong reason why better import/export wouldn't be enough for almost all needs.

This is the kind of thing I'd like to discuss on zoom.

Edward

[tfer]

There is another option we could consider, rather than full juypter stuff, just add an Ipython console tab to the log pane.  Though not as sexy as Juypter, it has most of its guts, is embed-able, and has a qt-console version.

[Thomas Passin]

On Tuesday, March 31, 2020 at 11:44:23 AM UTC-4, tfer wrote:

There is another option we could consider, rather than full juypter stuff, just add an Ipython console tab to the log pane.  Though not as sexy as Juypter, it has most of its guts, is embed-able, and has a qt-console version.

The thing is, I don't see the point of working with Jupyter/ipython in Leo unless we can actually make use of Leo's strengths.  Otherwise, Jupyter notebooks are very good, highly developed, and have many features (like their kernels for other languages and widgets) that we will never be able to duplicate in Leo.  Leo is especially good at at navigation, viewing, and and modification of the outline, and I don't want to give that up.

The whole point of thinking about getting Leo to play with Jupyter would be to be potentially share notebooks with Jupyter users.  Otherwise, we can just make Leo notebooks using VR3 or some similar plugin - I think I'd be happy with that - and we get all the advantages of Leo and the notebook-style approach.  And it would be a lot easier in terms of development effort.

That's my thinking, anyway.

[Edward K. Ream]

On Tue, Mar 31, 2020 at 10:44 AM 'tfer' via leo-editor <leo-e...@googlegroups.com> wrote:

There is another option we could consider, rather than full juypter stuff, just add an Ipython console tab to the log pane.  Though not as sexy as Juypter, it has most of its guts, is embed-able, and has a qt-console version.

Iirc, doing this is non-trivial, for reasons that I don't remember ;-)

Edward

[Edward K. Ream]

On Tue, Mar 31, 2020 at 1:03 PM Thomas Passin <tbp1...@gmail.com> wrote:

I don't see the point of working with Jupyter/ipython in Leo unless we can actually make use of Leo's strengths. 

I agree.

Last year we reached consensus that embedding Leo into emacs or vim is a bad idea. Similar logic applies here.

Otherwise, Jupyter notebooks are very good, highly developed, and have many features (like their kernels for other languages and widgets) that we will never be able to duplicate in Leo.  Leo is especially good at at navigation, viewing, and and modification of the outline, and I don't want to give that up.

Yes.

You and I have been discussing vr3 architecture privately recently. Let's make sure future discussions are here on leo-editor. vr3 looks to be just what is needed.

The whole point of thinking about getting Leo to play with Jupyter would be to be potentially share notebooks with Jupyter users.

I agree.

Otherwise, we can just make Leo notebooks using VR3 or some similar plugin - I think I'd be happy with that - and we get all the advantages of Leo and the notebook-style approach.  And it would be a lot easier in terms of development effort.

Imo, vr3 will offer very substantial advantages over jupyter. So now people will have a happy choice: use Jupyter or Leo, whichever best suits their needs.

Edward