ENB: @jupytext will simplify everything
Edward K. Ream
This Engineering Notebook post describes a breakthrough that will radically simply #4117 support jupytext. As a result of this Aha, I've just renamed this issue. This ENB will be pre-writing for a thorough revision of that issue's first comment.
Background: syncing files
Work on #4117 has gone smoothly. Nevertheless, one detail keeps complicating the code. How should Leo keep a notebook file, say x.ipynb, in sync with its paired jupytext file, x.py?
Aha: Don't sync files!!!
Leo should support @jupytext nodes. The following will be equivalent:
@jupytext x.py
@jupytext x.ipynb
@jupytext x
The file's extension doesn't matter because:
- When Leo reads an @jupytext x node, Leo will use jupytext to re-create that node and all its descendants from x.ipynb.
- When Leo writes the @jupytext x node, Leo will use jupytext to re-create x.ipynb from the entire @jupytext tree.
- When Leo checks for an externally changed .ipynb file, Leo will re-create the corresponding @jupytext x node, assuming such a node exists.
@jupytext collapses the complexity of all the jupytext-related code:
- Leo will only read and write .ipynb files.
- Within Leo, jupytext will never create .py files!
- As a result, Jupyter users won't need to use jupytext!
Use cases within Leo
Users can edit @jupytext nodes as usual. They can insert, delete and rearrange nodes.
A Leo option might specify whether to treat @jupytext nodes as @clean or @file. However, it's an open question whether Leo's sentinels are tolerable within Jupyter itself. I'll experiment soon.
Summary
@jupytext simplifies everything:
- Only @jupytext nodes update .ipynb files automatically.
- Leo will only read and write .ipynb files.
- There are no paired files to sync!
- There are no syntactically incorrect paired .py files!
- Jupyter users don't need to use jupytext!
All your questions and comments are welcome.
Edward
P.S. Writing this ENB has convinced me that @jupytext is the way forward. I would be shocked if @jupytext isn't the best solution for both Leo's users and maintainers.
EKR
Edward K. Ream
HaveF HaveF
> This Engineering Notebook post describes a breakthrough that will radically simply #4117 support jupytext.I elided some important details:> When Leo reads an @jupytext x node, Leo will use jupytext to re-create that node and all its descendants from x.ipynb.Recreating the node implies using jupytext to create a temporary .py file. But this file will exist only as a string. It will never become a real external file:
> Have you always wished Jupyter notebooks were plain text documents? Wished you could edit them in your favorite IDE? And get clear and meaningful diffs when doing version control? Then, Jupytext may well be the tool you’re looking for!
If we just save it in Leo as string, we may lose this advantage.
Of course, it seems that the logic is more complicated. In this case,
@jupytext x.py
@jupytext x.ipynb
@jupytext x
I don’t have a clear preference for this idea, just an idea.
HaveF HaveF
I have to say @jupytext is an amazing idea. I wonder if it would be better to let the user choose whether to write to an external file or not?
- jupytext also supports placing paired files in other folders. If we want to support writing to external files, it will be more troublesome.
Edward K. Ream
Today has been a golden day of programming. Pr #4119, renamed "Support @jupytext," now demonstrates all facets of the project.
Here are the highlights:
Dead easy collaboration
The present code demonstrates round-tripping of .ipynb files between Leo and Jupyter.
Everything "just works" without using jupytext in Jupyter!
Dead easy class design
- leoJupytext.py defines the JupytextManager (jtm) class. The LeoApp class instantiates a singleton instance of the jtm in the usual way.
- The AtFile class supports reading and writing @jupytext nodes with prosaic code, delegating all significant operations to the jtm.
- The ExternalFilesController class supports updating @jupytext nodes when external editors (including Jupyter) write .ipynb files.
Dead easy code
The JupytextManager class is remarkably concise. For example, here is jtm.read:
def read(self, c: Cmdr, p: Position) -> str: # pragma: no cover
"""
p must be an @jupytext node describing an .ipynb file.
Convert x.ipynb to a string and return that string.
"""
path = self.full_path(c, p)
if not path:
return '' # full_path gives any errors.
# Read .ipynb file into contents.
notebook = jupytext.read(path, fmt='py:percent')
contents = jupytext.writes(notebook, fmt="py:percent")
return contents
Nothing could be simpler! And here is the client, AtFile.readOneAtJupytextNode:
def readOneAtJupytextNode(self, p: Position) -> None:
"""
p must be an @jupytext node.
- Convert the .ipynb file to a string s.
- Update p.b's tree using s.
"""
c = self.c
contents = g.app.jupytextManager.read(c, p)
if contents:
p.b = '@language json\n\n' + contents
This code is a prototype. Instead of setting p.b, this method should use the @clean update code to set p's entire tree.
Summary
The present code demonstrates the round-tripping of .ipynb files between Leo and Jupyter. Everything "just works" without using jupytext in Jupyter!
The JupytextManager and AtFile classes work together to support reading and writing @jupytext nodes. The jtm and the ExternalFilesController classes collaborate to update @jupytext nodes when .ipynb files change externally.
The only remaining task is to have @jupytext nodes work like @clean nodes. Perhaps only a few more lines of code will suffice!
It's probably been five years since I have had such a significant and productive day of programming. None of this progress would have been possible without jupytext's spectacularly simple API!
Edward
Edward K. Ream
Edward K. Ream
Of course, it seems that the logic is more complicated. In this case,
@jupytext x.py
@jupytext x.ipynb
@jupytext xwill make a difference.
Edward K. Ream
HaveF HaveF
It's probably been five years since I have had such a significant and productive day of programming.
Edward K. Ream
It's probably been five years since I have had such a significant and productive day of programming.
I can feel your emotions across the screen and I'm happy for you :-D