Proposal: abandon support for .json files and jupyter notebooks.

[Edward K. Ream]

Don't panic: support for .leojs files (see #442) will remain.

The present importer for (special-format!) .json files is dangerously incorrect. It must either be rewritten or retired. I much prefer the latter option. Reading (and round-tripping) a full file format requires low-level outline operations that could corrupt the original outline. Imo, the risks far outweigh the rewards, which are likely to be nil :-)

At one time, I thought importing jupyter notebooks (.ipynb files) might allow Leo to evolve into an outline-oriented jupyter editor. Nothing ever came of the idea. Testing that round-tripping .ipynb files works seems like a waste of time. Without those tests, the .ipynb writer might corrupt jupyter notebooks!

Summary

The .json and .ipynb importers/writers are dangerous. The .json importer might corrupt Leo's outlines; the .ipynb writer might corrupt jupyter notebooks. Imo, it's time to retire them all.

Your comments please.

Edward

[tbp1...@gmail.com]

I wouldn't object.  You can usually view HTML versions of Jupyter notebooks in VR3 anyway.  Jupyter is so evolved by now that it would be a losing battle to try to support all its capabilities, but outside of Jupyter.  You can make your own (non-Jupyter but something like) notebooks to some extent with VR3, and I have found capability that useful.

[Edward K. Ream]

On Fri, Aug 12, 2022 at 10:33 AM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

I wouldn't object.  You can usually view HTML versions of Jupyter notebooks in VR3 anyway.  Jupyter is so evolved by now that it would be a losing battle to try to support all its capabilities, but outside of Jupyter.  You can make your own (non-Jupyter but something like) notebooks to some extent with VR3, and I have found capability that useful.

Thanks for your comments. I agree with you.

Edward

[Félix]

oh! I remember now noticing about an older json export functionality, and forgot to ask about it... i would go in the same direction as you proposed.

indeed the .leojs file format is somewhat more straightforward and complete as a JSON export format in itself. 

so i support removing the older json exporter (along with the jupyter one too: one less for me to mirror in leointeg/leojs ! ) 

:)

Félix

[Edward K. Ream]

On Fri, Aug 12, 2022 at 11:46 PM Félix <felix...@gmail.com> wrote:

oh! I remember now noticing about an older json export functionality, and forgot to ask about it... i would go in the same direction as you proposed.

indeed the .leojs file format is somewhat more straightforward and complete as a JSON export format in itself. 

so i support removing the older json exporter (along with the jupyter one too: one less for me to mirror in leointeg/leojs ! ) 

Thanks for your comments. Among many other items, PR #2729 removes these two importers.

Edward

[tbp1...@gmail.com]

VR3 will still be able to display jupyter notebooks.  It will convert an @jupyter node that has the URL of a jupyter outline to an HTML file and then render the HTML.  VR3 does not use the Leo importer at all.

[Félix]

good. 

I've just checked the changes, i've just also thought of removing the 'export to jupiter' command  from leoserver on felixserver4 branch too hehe..

Félix

[tbp1...@gmail.com]

There may be a possibility for round-tripping VR3 "notebooks" with Jupyter.  For example, there is Jupytext.  See also discussion on Github nbconvert issue 452.  It might turn out that small enhancements to VR3 could support round-tripping between VR3 and Jupyter notebooks.

That would be cool - if there is enough interest, I could spend some time on it.  Please comment here if you are interested.  No guarantees, though!

[Edward K. Ream]

On Tue, Aug 16, 2022 at 7:54 AM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

There may be a possibility for round-tripping VR3 "notebooks" with Jupyter.  For example, there is Jupytext.  See also discussion on Github nbconvert issue 452.  It might turn out that small enhancements to VR3 could support round-tripping between VR3 and Jupyter notebooks.

Thanks for this heads up. I'm interested.

Edward

[HaveF HaveF]

Of course, .ipynb files are not worth to support.

But it's bit sad to see that the JSON file is not supported, after all, json and yaml are two main configuration file formats.

Sometimes, I have to read hundred lines of JSON file, often tortured by indentation.

I wonder if there are any workarounds or indirect ways to help me improve the readability of JSON in the current situation, thanks in advance

[HaveF HaveF]

Recently I also tried importing Java files using Leo and to my surprise, it can be imported, although there are problems with understanding the nodes. For example, as we can see in my screenshot, java file doesn't have a function called if or for, I think this is mainly because Leo named some nodes based on file indentation.

Creating a file structure based on indentation in the file I think is very, very creative. Indentation is meant to be a way of expressing structure. For all files of unknown formats, can we make it like importing java, creating nodes for different indentations, so, I think, Leo is equivalent to supporting all file format imports with indentation and no explicit support.

[Edward K. Ream]

On Wed, Jul 5, 2023 at 3:40 AM HaveF HaveF <iamap...@gmail.com> wrote:

Of course, .ipynb files are not worth to support.

I don't recall a proposal to abandon .json files. Can you refresh my memory?

Edward

[HaveF HaveF]

I don't recall a proposal to abandon .json files. Can you refresh my memory?

I read it in this thread:

> The present importer for (special-format!) .json files is dangerously incorrect. It must either be rewritten or retired. I much prefer the latter option. Reading (and round-tripping) a full file format requires low-level outline operations that could corrupt the original outline. Imo, the risks far outweigh the rewards, which are likely to be nil :-)

Currently, the JSON support(@auto) is reading all the content in JSON into a single node

--

--
Sincerely,

HaveF

[Edward K. Ream]

On Wed, Jul 5, 2023 at 5:07 AM HaveF HaveF <iamap...@gmail.com> wrote:

I don't recall a proposal to abandon .json files. Can you refresh my memory?

I read it in this thread:

> The present importer for (special-format!) .json files is dangerously incorrect. It must either be rewritten or retired. I much prefer the latter option. Reading (and round-tripping) a full file format requires low-level outline operations that could corrupt the original outline. Imo, the risks far outweigh the rewards, which are likely to be nil :-)

Thanks for the reminder. Iirc Félix agreed with the proposal to retire the json importer. That was good enough for me :-)

In most cases reading a .json file into a single file is a good enough start. You can then convert the node to @clean and reorganize as you like.

Edward

[HaveF HaveF]

In most cases reading a .json file into a single file is a good enough start. You can then convert the node to @clean and reorganize as you like.

OK, it seems like a reasonable solution. Thanks Edward

--

--
Sincerely,

HaveF

[Edward K. Ream]

On Wednesday, July 5, 2023 at 7:00:47 AM UTC-5 iamap...@gmail.com wrote:

In most cases reading a .json file into a single file is a good enough start. You can then convert the node to @clean and reorganize as you like.

OK, it seems like a reasonable solution. Thanks Edward

You're welcome.

Edward

[Thomas Passin]

On Wednesday, July 5, 2023 at 8:00:47 AM UTC-4 iamap...@gmail.com wrote:

In most cases reading a .json file into a single file is a good enough start. You can then convert the node to @clean and reorganize as you like.

OK, it seems like a reasonable solution. Thanks Edward

As you re-organize a file like this, remember that the extract minibuffer command makes the process much easier.  Add a line to the top of a block you want to extract, select the whole block, and the extract command will pull all those lines into a new node whose headline will be the line you added. The new node will be indented in the outline if the selected block was indented.

[HaveF HaveF]

As you re-organize a file like this, remember that the extract minibuffer command makes the process much easier.  Add a line to the top of a block you want to extract, select the whole block, and the extract command will pull all those lines into a new node whose headline will be the line you added. The new node will be indented in the outline if the selected block was indented.

Amazing !!!

I was planning to use `w.getSelectedText().strip()` to reinvent my wheel...

Once again, you saved me time, at least half an hour... Thanks!

Hi, Thomas, I believe your daily workflow is an invaluable asset, as well as Edward....

 

--

--
Sincerely,

HaveF

[Thomas Passin]

On Wednesday, July 5, 2023 at 12:15:59 PM UTC-4 iamap...@gmail.com wrote:

As you re-organize a file like this, remember that the extract minibuffer command makes the process much easier.  Add a line to the top of a block you want to extract, select the whole block, and the extract command will pull all those lines into a new node whose headline will be the line you added. The new node will be indented in the outline if the selected block was indented.

Amazing !!!

I was planning to use `w.getSelectedText().strip()` to reinvent my wheel...

Leo is filled with wonderful conveniences like this.  The only problem is to discover them ...

 

Once again, you saved me time, at least half an hour... Thanks!

Hi, Thomas, I believe your daily workflow is an invaluable asset, as well as Edward....

There's a large difference between Edward's work flow and mine - not that either of us has a single "work flow", I'm sure.  He likes to use clones extensively - often extracting them with the cff command, I think.  I tend to search using the Nav tab and mark nodes of interest - I have my own key bindings to do that, since the bindings of the EditPlus editor were so familiar to me from years of pre-Leo use (of course, once you have marked some nodes you can clone them all with the clone-marked-nodes command).  Instead of cloning I jump between them with the forward-back arrows and the goto-next-marked command, which I have bound to F4.  When I want to keep looking back at the content of a node, I will usually keep it visible using the Freewin plugin (that ability is the main reason I wrote the plugin).

Edward has done a lot more study of other code bases than I and that has probably got a lot to do with his extensive use of clones compared to me.  In my case, I'm more likely to be chasing through Leo's code base looking for the key method that supports some command or method.  Once I find it, I mark it.  I don't want to end up with a lot of cloned nodes just because they fit some search pattern but aren't actually what I want.

[Edward K. Ream]

On Wed, Jul 5, 2023 at 12:06 PM Thomas Passin wrote:

> There's a large difference between Edward's work flow and mine - not that either of us has a single "work flow", I'm sure.  He likes to use clones extensively - often extracting them with the cff command, I think. 

Correct.

> Edward has done a lot more study of other code bases than I and that has probably got a lot to do with his extensive use of clones compared to me.

I use clones because they help focus my attention on the task at hand.

This FAQ entry describes in detail how I use clones. I don't understand why more Leonistas don't follow my example :-)

Info issue #1177 describes another aspect of my workflow.

Edward

[HaveF HaveF]

This FAQ entry describes in detail how I use clones. I don't understand why more Leonistas don't follow my example :-)

Perhaps in some cases, words are still a little difficult to convey knowledge adequately :D

 

Info issue #1177 describes another aspect of my workflow.

Thanks, Edward. Btw, I found the g.getGitIssues function yesterday, it seems very useful. 

I'm trying to add support to handle private repo issues.

And also thank you, Thomas. Especially your explanation of your workflow. In addition, 

I found the freewin plugin, although it is in the codebase, but it is not written in the documentation, 

I will take a look later.

--

--
Sincerely,

HaveF

[HaveF HaveF]

There's a large difference between Edward's work flow and mine - not that either of us has a single "work flow", I'm sure.  He likes to use clones extensively - often extracting them with the cff command, I think.  I tend to search using the Nav tab and mark nodes of interest - I have my own key bindings to do that, since the bindings of the EditPlus editor were so familiar to me from years of pre-Leo use (of course, once you have marked some nodes you can clone them all with the clone-marked-nodes command).  Instead of cloning I jump between them with the forward-back arrows and the goto-next-marked command, which I have bound to F4.  When I want to keep looking back at the content of a node, I will usually keep it visible using the Freewin plugin (that ability is the main reason I wrote the plugin).

Hi, Thomas, If I'm right, i.e., you mainly use mark, clone to understand the problem, 

and Edward mainly aggregates all relevant content through cff and then filters it, right?

Also I looked at the freewin plugin and I don't quite understand why you need this? 

Does this plugin look like a panel + viewrender combination? 

Its functionality could be replaced by the powerful 'free layout', wouldn't it?

Or is there anything else I missed?

 

--

--
Sincerely,

HaveF

[Thomas Passin]

On Wednesday, July 5, 2023 at 10:30:39 PM UTC-4 iamap...@gmail.com wrote:

There's a large difference between Edward's work flow and mine - not that either of us has a single "work flow", I'm sure.  He likes to use clones extensively - often extracting them with the cff command, I think.  I tend to search using the Nav tab and mark nodes of interest - I have my own key bindings to do that, since the bindings of the EditPlus editor were so familiar to me from years of pre-Leo use (of course, once you have marked some nodes you can clone them all with the clone-marked-nodes command).  Instead of cloning I jump between them with the forward-back arrows and the goto-next-marked command, which I have bound to F4.  When I want to keep looking back at the content of a node, I will usually keep it visible using the Freewin plugin (that ability is the main reason I wrote the plugin).

Hi, Thomas, If I'm right, i.e., you mainly use mark, clone to understand the problem, 

and Edward mainly aggregates all relevant content through cff and then filters it, right?

I haven't used clones very much.  I usually skip around from marked node to marked node, plus using the forward/back buttons.

 

Also I looked at the freewin plugin and I don't quite understand why you need this? 

Does this plugin look like a panel + viewrender combination? 

Its functionality could be replaced by the powerful 'free layout', wouldn't it?

Or is there anything else I missed?

Free layout does not do the same job as  the Freewin plugin.  FW gives you a view of a particular node.  No matter where you navigate in the tree, a given FW window always shows you the same node (though you can have several FW windows open at the same time, each for a different node).  The alternate view (the viewrendered-type display) is there mostly so you can get a nicely readable rendering of a body written in ReStructuredText, or check that your edits to such a node render the way you wanted.

Sometimes I think of a FW window as a kind of index card open next to the work in progress.  Also, a FW window is a basic editor that is live on line with the underlying node.  So even if you have selected some other node in the tree, if you see something that you want to change (in the FW's host node), you can just edit it in the FW window and save with CTRL-s as usual.  No need to navigate back to that original node to make the change and then navigate back.

Suppose you are working with three or four nodes, and you have cloned them under an organizing node somewhere,  Since they are close together, you can easily move from one to another.  But you can't see more than one at a time.  Or instead of (or in addition to) cloning the nodes, you could open several of them in FW windows, and arrange those next to Leo's window.  Now you could look from one to another without losing sight of the ones you navigated away from.  This image shows a comparison between some of the prototype FW code to the final plugin code.  If you can't see them at the same time it's harder to understand the differences. (BTW, FW windows now do syntax coloring; this wasn't in place at the time of the screenshot).

Here is the thread in which I introduced the FW plugin.  One of my posts in the thread has some screenshots where I tried to show ways I used it.

[HaveF HaveF]

Sometimes I think of a FW window as a kind of index card open next to the work in progress.  Also, a FW window is a basic editor that is live on line with the underlying node.  So even if you have selected some other node in the tree, if you see something that you want to change (in the FW's host node), you can just edit it in the FW window and save with CTRL-s as usual.  No need to navigate back to that original node to make the change and then navigate back.

Suppose you are working with three or four nodes, and you have cloned them under an organizing node somewhere,  Since they are close together, you can easily move from one to another.  But you can't see more than one at a time.  Or instead of (or in addition to) cloning the nodes, you could open several of them in FW windows, and arrange those next to Leo's window.  Now you could look from one to another without losing sight of the ones you navigated away from.  This image shows a comparison between some of the prototype FW code to the final plugin code.  If you can't see them at the same time it's harder to understand the differences. (BTW, FW windows now do syntax coloring; this wasn't in place at the time of the screenshot).

Here is the thread in which I introduced the FW plugin.  One of my posts in the thread has some screenshots where I tried to show ways I used it.

Thank you for providing a detailed explanation. By the way, I discovered the `add-editor` command, although it is not as versatile as the FW.

--

--
Sincerely,

HaveF

[Thomas Passin]

On Wednesday, July 5, 2023 at 11:45:48 PM UTC-4 iamap...@gmail.com wrote:

Thank you for providing a detailed explanation. By the way, I discovered the `add-editor` command, although it is not as versatile as the FW.

I always had trouble using an added editor. I was forever switching one to the wrong node because I didn't notice it was focused before navigating in the tree.  Then I'd be typing into the wrong node.  IIRC, that was one of the drivers for me to write the FW plugin.  Also, the added editors only apply within one outline, whereas a FW window continues to be visible even after you switch to another outline.  Another impetus was that VR3 could be used in a free layout separate window but it can't edit its host node, and it's not node-locked. 

[HaveF HaveF]

I always had trouble using an added editor. I was forever switching one to the wrong node because I didn't notice it was focused before navigating in the tree.  Then I'd be typing into the wrong node.  IIRC, that was one of the drivers for me to write the FW plugin.  Also, the added editors only apply within one outline, whereas a FW window continues to be visible even after you switch to another outline.  Another impetus was that VR3 could be used in a free layout separate window but it can't edit its host node, and it's not node-locked. 

That makes FW more sense to me. These words should be in the doc of plugin.

Thanks! Thomas!

 

--

--
Sincerely,

HaveF

[HaveF HaveF]

Btw, I believe making some FW windows always stay on the top and set opacity of windows are also useful.

https://stackoverflow.com/questions/19097323/setwindowflagsqtwindowstaysontophint-hides-qt-window

https://doc.qt.io/qt-6/qwidget.html#windowOpacity-prop

--

--
Sincerely,

HaveF

[Thomas Passin]

I might be more receptive to this idea if we can come up with a way for a user to change in and out of the stay-on-top property and the opacity.  It won't work as one-size-fits-all, in my experience.

[Edward K. Ream]

On Wed, Jul 5, 2023 at 8:32 PM HaveF HaveF <iamap...@gmail.com> wrote:

I found the g.getGitIssues function yesterday, it seems very useful. 

I'm trying to add support to handle private repo issues.

I'm using another function, g.execGitCommand, as the basis for #3423.

Edward

[HaveF HaveF]

I might be more receptive to this idea if we can come up with a way for a user to change in and out of the stay-on-top property and the opacity.  It won't work as one-size-fits-all, in my experience.

A checkbox and a spinbox along with the current "Rendered < -- > Plain" button for every separate FW window?

 

--

--
Sincerely,

HaveF

[HaveF HaveF]

I'm using another function, g.execGitCommand, as the basis for #3423.

Thanks, Edward, I'll check it later 

--

--
Sincerely,

HaveF

[Edward K. Ream]

On Thu, Jul 6, 2023 at 7:20 AM HaveF HaveF <iamap...@gmail.com> wrote:

I'm using another function, g.execGitCommand, as the basis for #3423.

Thanks, Edward, I'll check it later 

You're welcome.  Happy coding :-)

Edward