Rst3 vs. rst4?

[Viktor Ransmayr]

Hello Edward,

When I was reading your latest post on "Discuss: Radically simplify the rst3 command?", one additional un-related question came to my mind:

Can you explain what measures you use to decide, when you rework an available command / extension & accept (the possibility of) breaking backwards compatibility  - and - when you create a new major version of a command / extension & keep the existing version available in order to guarantee backwards compatibility for Leo users?

Sorry for the possible distraction - but - I'm really interested in your experience & view!

With kind regards,

Viktor

[Edward K. Ream]

On Sat, Mar 27, 2021 at 6:06 AM Viktor Ransmayr <viktor....@gmail.com> wrote:

Can you explain what measures you use to decide, when you rework an available command / extension & accept (the possibility of) breaking backwards compatibility  - and - when you create a new major version of a command / extension & keep the existing version available in order to guarantee backwards compatibility for Leo users?

An excellent question. It's not a distraction.

It took me a moment just now to remember why I started on #1867 a few days ago.

I was working on #1843, re @rst-insert-body & @rst-insert-tree. I thought it would take a few hours, and then I would restart the sabbatical. But then I saw that #1868 (clickable links based on gnx's) would make it easy to specify nodes to be copied by hand into an @rst tree.

All this led to the four Aha's in the first post. I saw that rst3 was trying to do something quite silly, namely supporting Leo-style literate programming with a gazillion features.

I also saw that the supposed problems with creating documentation are really not Leo's concern. Documentation is hard. Writers may need different words (on the same topic) depending on the exact context. So the "include text by cloning" model is misguided.

All these Aha's pointed to an inescapable conclusion: rst3 should do nothing except:

1. Generate rST section references automatically.

2. Handle the details of driving docutils.

I stated this conclusion at the start of the first post, and restated it at the end of the first post. I'm happy to continue to restate it, because it is an earthquake in my thinking. Leo should leave writing to writers!

Now then, what should I do with these Ahas? Do I leave rst3 in place and add an rst4 command, as your title suggests? It's a fair question, and my answer is a judgment call.

Having said that, I am convinced that simplifying rst3 is best in the long run:

1. Most importantly, the new rst3 command creates clearer thinking about how writers should use Leo. Imo, the old way encouraged people to waste their time. Now, the only way is the simple way.

2. I am thrilled to be able to delete most of (all of?) Leo's rst3 reference. The path to power lies in simplicity and generality, not endless features.  The old way was poor design.  Nobody needs it.

3. The new code has collapsed in simplicity. In future, Leo's devs will be able to understand leoRst.py. Right now, the simpler code is much easier to test. Eventually I'll aim for 100% code coverage.

Summary

There may be one or two Leonistas who may have to revise their documentation. I am convinced that even they will benefit by doing so.

Edward

[tbp1...@gmail.com]

However, the title of Viktor's post implies a valid suggestion.  Everything you say you want to achieve could be done by leaving rst3 as is - never touching it again - and creating a new rst4.  It would be no more work, and there would be no latent backwards compatibility issues.

In fact, it would make more sense because the goals of the new version are different from the old.  A new name would be appropriate for that.

[Edward K. Ream]

On Saturday, March 27, 2021 at 11:52:00 AM UTC-5 tbp1...@gmail.com wrote:

Everything you say you want to achieve could be done by leaving rst3 as is - never touching it again - and creating a new rst4.  It would be no more work, and there would be no latent backwards compatibility issues.

 There are costs associated with doing nothing, some of which I've mentioned: learning costs for newbies and maintenance costs.

There are several other costs of doing nothing. Call them "opportunity costs" if you like:

- The design and code of rst3 are poor adverts for Leo.

- rst3 leads our thoughts astray. The rst3 command should handle content, not manufacture content.

- A clearer view of rst3 will encourage those with advanced needs to write simple scripts to create the content they want.

Summary

We must not rationalize past mistakes. Progress happens by learning from those mistakes and moving on.

I've ripped out major projects before. Leo is stronger without Qt docks, and Leo will be stronger with a simplified rst3 command.

Imo, an rst4 command would increase confusion. The rst3 command (and its horrendous docs) would be an ongoing distraction.

I'd like to hear now from those who actually use the "advanced" features of rst3.

Edward

[Edward K. Ream]

On Saturday, March 27, 2021 at 4:37:39 PM UTC-5 Edward K. Ream wrote:

> There are several other costs of doing nothing. Call them "opportunity costs" if you like:

In my mind, this isn't a difficult judgment call. Let me explain:

Leo has had, from very early days, superb capabilities for creating and changing outlines: think, p.b, p.h, p.gnx, Leo's iterators, and yes, clones.

So if people want automated ways of creating content, it's dead easy to do so with scripts and plugins. Instead, I created a horrendous botch of half-baked options for the rst3 command. That botch must go. It's not a tough call.

Edward

[tbp1...@gmail.com]

I'm not saying anything against what you say, not at all.  I'm just suggesting that the end result might better be named by a different name, presumably "rst4".

[Edward K. Ream]

On Sat, Mar 27, 2021 at 9:11 PM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

I'm not saying anything against what you say, not at all.

Glad to hear it :-)

I'm just suggesting that the end result might better be named by a different name, presumably "rst4".

Renaming the command would be confusing make work:

- The rst3 command uses settings whose names start with 'rst3-'.  Why require people to change 'rst3-' to 'rst4-' in their myLeoSettings.leo file?

- The command works on `@rst` nodes. A new rst4 command should still work on `@rst` nodes. Why force people to change `@rst` to `@rst4` in their outlines?

Renaming the command would misstate the coming changes:

- Removing features that almost nobody uses.

- Removing 13 of the original 26 rst3-related settings in leoSettings.leo.

If I had not announced my intentions, probably nobody would have noticed these changes! I did announce my intentions because:

- It's never good to hide potentially controversial decisions.

- I wanted to subject my thinking to public scrutiny, and get the benefits from doing so.

- I wanted to keep everyone up to date.

Summary

I repeat.  I want now to hear only from those who would be seriously inconvenienced by simplifying Leo's rst3 command.

Edward

P. S. If there are those who use the "advanced" features of the rst3 command, they have a simple workaround: open their documentation files with Leo 6.3 while they transition their documentation.

To help them make that transition, I'll rewrite Leo's "Creating Documents from Outlines" Chapter. The new chapter will discuss how to write scripts to compose `@rst` outlines from data.

Those scripts are easy! For example, instead of adding the @rst-insert-tree command, as #1843 suggests, the following tested script will show root's tree:

result = []
for p in root.self_and_subtree():
    level = p.level() - root.level()

    indent = ' ' * 2 * level  # Two-space indents.

    result.append(indent + '- ' + p.h + '\n')
s = ''.join(result)

More compactly, using f-strings and generator expressions:

s = ''.join(

    f"{' '*2*(z.level()-root.level())}- {z.h}\n"

        for z in root.self_and_subtree())

This script shows why we should be putting our attention on Leonine scripting, not faux-enhancements to Leo's rst3 command. Why settle for what `@rst-insert-tree` would give us? A simple script gives us complete flexibility to format the tree as we like.

EKR

[Edward K. Ream]

On Saturday, March 27, 2021 at 4:37:39 PM UTC-5 Edward K. Ream wrote:

- rst3 leads our thoughts astray. The rst3 command should handle content, not manufacture content.

- A clearer view of rst3 will encourage those with advanced needs to write simple scripts to create the content they want.

When we shift our focus away from rst3 features, we will find scripting tools such as sphinx-gallery:

    "Embed rST in your example Python files, allowing you to interweave narrative-like content with code...

    Sphinx-Gallery also automatically generates a Jupyter Notebook for each your example page."

Edward

[Edward K. Ream]

On Sunday, March 28, 2021 at 8:57:46 AM UTC-5 Edward K. Ream wrote:

When we shift our focus away from rst3 features, we will find scripting tools such as sphinx-gallery:

And sphinx itself.

The new chapter about rst3 will discuss tools such as sphinx and sphinx gallery. That's going to be a lot more useful that discussing endless options.

Speaking of options, the ekr-new-rst branch now documents all the remaining rst3 options in leoSettings.leo where they belong.

Edward

[Edward K. Ream]

On Sunday, March 28, 2021 at 9:09:50 AM UTC-5 Edward K. Ream wrote:

>> When we shift our focus away from rst3 features, we will find scripting tools such as sphinx-gallery:

> And sphinx itself.

And Jinja2. Excellent tools abound.

Edward

[tbp1...@gmail.com]

My Sphinx documents use a Sphinx-type table of contents at the top of the tree,  a hierarchy of rst files, and inter-file links.  Nothing more complicated.  The content includes text, code snippets,  entire small programs, output captures formatted as plain mono-spaced text, and images.  I don't use anything more complex, and until a few days ago  was not aware that you could do anything more complicated with rst3.

So when you are done with the revamped rst3, I want to be able to run it on my tree and get Sphinx-ready output, just like you can get currently.  If that will work, I will be happy.  If it won't, I will probably have to do a lot of work to be able to generate my documentation, and I won't like that.

[Edward K. Ream]

On Sun, Mar 28, 2021 at 10:46 AM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

My Sphinx documents use a Sphinx-type table of contents at the top of the tree,  a hierarchy of rst files, and inter-file links.  Nothing more complicated.  The content includes text, code snippets,  entire small programs, output captures formatted as plain mono-spaced text, and images.  I don't use anything more complex, and until a few days ago  was not aware that you could do anything more complicated with rst3.

So when you are done with the revamped rst3, I want to be able to run it on my tree and get Sphinx-ready output, just like you can get currently.  If that will work, I will be happy.  If it won't, I will probably have to do a lot of work to be able to generate my documentation, and I won't like that.

Everything should work exactly as before when you run the "new/simplified" rst3 command.

Presumably you are using @bool rst3_call_docutils = False, as in LeoDocs.leo. That option creates intermediate (.html.txt) files for use by sphinx, without calling docutils.

As of rev c0fff62, git diff confirms that the new version of rst3 produces .html.txt files for all `@rst` trees in LeoDocs.leo that are binary equal to those produced by the legacy version of rst3. This is a major milestone.

So it looks like #1867 is complete, except for cleanups and documentation. However, I am going to wait to merge the ekr-new-rst branch into devel.  Here's why:

Despite what I have been saying for a few days, a new script might be worthwhile. This script would simulate some of the old features of the rst3 command, but in a straightforward manner.

In hindsight, the bad old features were intended to abstract or filter nodes/text.  Well, we now have c.cloneFindByPredicate.  We can imagine an initial filtering/gathering of nodes using c.cloneFindByPredicate, followed by an optional additional filtering of text within nodes. Instead of cloning, it would be easy to create c.copyFindByPredicate, to allow unrestricted munging of filtered text without affecting the original outline.

I have no idea how this will turn out. I am sure only that the new rst3 code will be a much better base for experimentation than the old. There is a real possibility that sphinx, sphinx-gallery, and jinja already do everything people will need, so before charging ahead it would be good to study the sphinx world.

Summary

The ekr-new-rst branch looks to be code complete, but I won't rush to merge it into devel.

Today's work marks the end of the "pause" in my sabbatical.  I'll take a break for a day or two, then resume my sabbatical. Studying the sphinx world and its relationship to rst3-oriented tasks seems like a good next sabbatical project.

As always, I'll be happy to answer questions and fix urgent bugs during the sabbatical.

Edward

[tbp1...@gmail.com]

On Sunday, March 28, 2021 at 2:25:22 PM UTC-4 Edward K. Ream wrote:

Presumably you are using @bool rst3_call_docutils = False, as in LeoDocs.leo. That option creates intermediate (.html.txt) files for use by sphinx, without calling docutils.

Actually, I have it set to True.  I'll try it with False, but everything has worked correctly so far.

BTW, lately I have been running Sphinx without bothering to create a Sphinx config file.  I don't usually need too many settings, and I can just put them as -D options on the command line.  Then I run the command with subprocess, which I can launch with either CNTR-B or with a button.

[Edward K. Ream]

On Sun, Mar 28, 2021 at 3:35 PM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

Presumably you are using @bool rst3_call_docutils = False, as in LeoDocs.leo. That option creates intermediate (.html.txt) files for use by sphinx, without calling docutils.

Actually, I have it set to True.  I'll try it with False, but everything has worked correctly so far.

My apologies for the confusion.

The @bool rst3-write-intermediate-file setting controls whether rst3 writes the sources (results of processing the @rst tree) to an intermediate file.

The @bool rst3_call_docutils setting controls whether rst3 writes external docutils files. These files contain the result of passing the sources to docutils.

Let me emphasize that everything (including settings) should work exactly as before for those, like you, who don't rst's advanced features (code mode, doc-only mode, etc). If something changes for you, it's a bug. Please report it.

Edward

P.S. Yesterday I fixed a recent blunder that prevented intermediate files from being written when @bool rst3_call_docutils is False. So please use the latest rev in ekr-new-rst.

To clarify what rst3 is doing, the rst3 command now reports both the number of intermediate files and docutils files rst3 writes.

EKR