ENB: Thought experiments re IPython

[Edward K. Ream]

This Engineering Notebook post discusses integrating Leo with IPython. Similar remarks apply to Jupyter and SageMath.

To summarize: thought experiments doom all such projects.

Bypass IPython's event loop?

Suppose Leo avoids creating IPython's event loop. Doing so would avoid hanging Leo when accessing IPython's features. This idea implies bypassing IPython's repl.

Bypass IPython's repl?

Suppose Leo provides IPython's features by accessing them via IPython's API. However, experiments (and a review of IPython's code) show that IPython's features require IPython's repl!

Indeed, IPython's BaseIPythonApplication class contains this description:

    "IPython: an enhanced interactive Python shell."

Create a bespoke client-server architecture?

Suppose Leo accesses IPython's features by accessing a running instance of IPython via a client-server architecture. Leo would be the client. It would be straightforward to start this server when IPython starts.

But what, exactly, would Leo ask IPython to do? And why would anyone care? There seems to be no answers to this question!

Conclusion

The thought experiments presented here suggest that Leo gains nothing by attempting to work closely with IPython. There are no scenarios in which Leo gains from accessing IPython's features from within Leo.

This surprising conclusion is good news. It's always good to abandon mistaken ideas!

After clearing our heads, we can see that this conclusion makes sense. Leo needs no help from IPython and vice versa. Why did we ever think otherwise?

Summary

Leo's IPython integration does not work and probably has not worked for years. Yet nobody seems to have noticed!

Integrating Leo and IPython presents insurmountable problems. All attempts would likely be complex and fragile, as indeed leoIPython.py has proven to be.

Leo gains nothing by attempting to work closely with IPython. All recent comments from Leonistas support this conclusion.

Unless I hear convincing counter-arguments, a new PR (scheduled for 6.8.2) will remove leoIPython.py and Leo's valuespace plugin. There will be no need to add IPython or Jupyter to Leo's requirements.

All your comments and questions are welcome. There is plenty of time for further discussion.

Edward

[Thomas Passin]

I think you may be a little hasty here.  Also, IPython is not the same as Sage even if Sage uses IPython under the cover (I'm not completely sure it does), and even if the standard Sage REPL looks like a console. So I've been think about Sage. Some of this may also apply to IPython, but maybe not because, as Edward said, it's basically a shell. Of course, this will leave out Windows users; I don't count WSL as being Windows for most users.

Sage, now, it may look like a shell but it can benefit a lot from notebook-like features:

1. After our Sage user goes through various math steps and maybe generates some graphics, now what?  He wants to save that work and be able to look at it and pick it up again later!  And in the saved work, who wants to save the false starts and the syntax errors? A Leo node differs from a console shell in these ways: you can put the cursor out of sequence and do editing, you can highlight a block and do something with it, you can do something (like recalculate) with the entire node, and you can save and restore it.

So here's the user wanting to do Sage. Type @language sage at the top. Just like CTRL-B sends the node or a marked part of it to be run as a script, the user can do something similar with lines of Sage code. The node acts much like, but not exactly the same, as a sage console, or the Jupyter Notebook version. Static graphics will be easy to deal with - get Sage to save the graphic to a file and Qt can insert that graphic with a cursor. The graphic can be persisted at that location by having the Sage handler insert an @image <path> line; during reload or re-display, the node's code re-inserts the graphic the same way it was originally inserted.  Of course, the user doesn't need to know about that.   A bonus is that the user could insert any image by manually typing in an @image line. Another bonus is that VR3 already understands the @image directive.

This scenario uses Sage to produce a persistable, editable, rerunable version of the Sage console.

2. Our user wants to do all of the above but add text commentary to explain context or to clarify the work, much as can be done with a Jupyter notebook.  So he puts all the Sage work in a section or sections starting with an @language sage directive and ending with an bare @ character. Leo already has all the machinery for doing this; VR3 works much like this to mix code and non-code.  Sage code execution could include either single blocks or all Sage blocks in the node.  With this model, we're basically doing literate programming - but live - so we're going back to Leo's roots.

3. As soon as we've gotten Scenario 2 working, the next step would be supporting a whole tree of Sage nodes. Some things need to be worked out, like whether math execution should be restricted to a single node or cover the whole tree. But those are details. Just like the rst3 command processes an entire tree, a run-sage command could process an entire Sage tree.  I don't think there is any need to support @others since progressive indentation doesn't make much sense for a Sage notebook.  VR3 doesn't support it for a similar reason.  And I don't thing that <<named sections>> should be supported because they would allow out-of-order execution and rendering, and that could be actively bad.

Scenario 3 gets us some of the most used parts of Jupyter notebooks with far lower complexity. It improves on VR3 is some ways (for notebooks, I mean) since the VR3 display is a static, non-editable view. VR3 has the advantage of being able to render an entire tree, while working with Leo nodes as suggested above only lets one look at one node at a time.  I think we can live with that.

[HaveF HaveF]

Leo gains nothing by attempting to work closely with IPython. 

I agree with Edward that complex integration does not bring any benefits.

If I imagine, the combination of Leo and Jupyter may only be a combination at the file level. Maybe Leo will parse the ipynb file and turn it into blocks, and then we can use Leo's outline feature to deconstruct/organize/clone blocks at will.
After the new outline is formed and saved, then in JupyterLab, `reload from disk` reads the new ipynb file.

We all know that Leo's outline is unmatched✨✨✨

As I said before, for now, there is no such function, I use jupytext to implement the above idea.

ipynb files ----(jupytext)--> py files ----(import and organize by leo) --> (modify py files from leo or any other editors) --> (reload from disk) in jupyterlab --> (reset kernel and run all cells)

 

[Thomas Passin]

That's very interesting.  The Jupyter ecosystem is expanding so fast that there is always something new.  I don't actually use Jupyter-* so I'm not in touch with it at all. I always think I should be using Jupyter but I don't do much work these days that would make much use of it.

[Edward K. Ream]

On Sun, Oct 20, 2024 at 8:08 AM Thomas Passin <tbp1...@gmail.com> wrote:

I think you may be a little hasty here.

...

Sage, now, it may look like a shell but it can benefit a lot from notebook-like features:

A quick look at IPython's code shows that trying to extract features from the IPython codebase is a truly terrible idea.

Every aspect of IPython is bound to the repl. For example, IPython uses live objects throughout, which means that IPython can offer superior code completion. But it's not possible to extract IPython's code completion without extracting the repl! And even if it were possible, the extracted code could change at the next IPython/Jupyter release.

It's time to reject this fantasy!

1. After our Sage user goes through various math steps and maybe generates some graphics, now what?  He wants to save that work and be able to look at it and pick it up again later!  And in the saved work, who wants to save the false starts and the syntax errors?

The workaround is to redo the valid steps in another notebook. There may be automatic ways of doing that.

A Leo node differs from a console shell in these ways: you can put the cursor out of sequence and do editing, you can highlight a block and do something with it, you can do something (like recalculate) with the entire node, and you can save and restore it.

That's the Siren song.

So here's the user wanting to do Sage. Type @language sage at the top. Just like CTRL-B sends the node or a marked part of it to be run as a script, the user can do something similar with lines of Sage code. The node acts much like, but not exactly the same, as a sage console, or the Jupyter Notebook version. Static graphics will be easy to deal with - get Sage to save the graphic to a file and Qt can insert that graphic with a cursor. The graphic can be persisted at that location by having the Sage handler insert an @image <path> line; during reload or re-display, the node's code re-inserts the graphic the same way it was originally inserted.  Of course, the user doesn't need to know about that.   A bonus is that the user could insert any image by manually typing in an @image line. Another bonus is that VR3 already understands the @image directive.

This scenario uses Sage to produce a persistable, editable, rerunable version of the Sage console.

Imo, this scenario is infeasible and unwise. I am definitely no longer tempted by Sirens.

Edward

[Thomas Passin]

On Sunday, October 20, 2024 at 2:17:13 PM UTC-4 Edward K. Ream wrote:

Imo, this scenario is infeasible and unwise. I am definitely no longer tempted by Sirens.

There you go, the value of creating scenarios! 

[Edward K. Ream]

:-) That heuristic has been invaluable recently. Thanks for suggesting it.

Edward

[Ville M. Vainio]

Chipping in as someone that wrote the (first version) of ipython plugin - there is no way Leo could compete with the current state of Jupyter notebooks. There is deep integration with VS Code, so you get the "full" editing experience from there. There would be tons of work, and probably little benefit compared to "main" Jupyter notebook experience

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/CAMF8tS0ayX9nR7WnwhFZk76J2o%3D%3D-boE7ZBjrthH5TK2JqPG-A%40mail.gmail.com.

[Edward K. Ream]

On Sun, Oct 20, 2024 at 2:56 PM Ville M. Vainio wrote:

Chipping in as someone that wrote the (first version) of ipython plugin - there is no way Leo could compete with the current state of Jupyter notebooks.

Good to hear from you, Ville! It's good to have your confirmation of recent conclusions.

There is deep integration with VS Code, so you get the "full" editing experience from there. There would be tons of work, and probably little benefit compared to "main" Jupyter notebook experience

You're reminded me to look at vs-code plugins supporting Jupyter notebooks.

Heh, the "Jupyter" plugin from Microsoft has 83 million users! I'm happy not to try to compete.

Edward

[HaveF HaveF]

On Sun, Oct 20, 2024 at 9:59 PM Thomas Passin <tbp1...@gmail.com> wrote:

That's very interesting.  The Jupyter ecosystem is expanding so fast that there is always something new.  I don't actually use Jupyter-* so I'm not in touch with it at all. I always think I should be using Jupyter but I don't do much work these days that would make much use of it.

Yes, this ecosystem is expanding so fast. The reason I am using jupytext is just because it can convert ipynb to plain source code. As we all know, AI is very convenient for source code assisted editing.

There are lots of `kernal`s for jupyterlab, this is the most interesting part, for example, you can use go kernel or deno kernel in jupyter.