Progress report (revised): the new simpler rst3 command

[Edward K. Ream]

I posted an incomplete version of this post by mistake. I have deleted the incomplete post.

#1867 suggests retaining only the basic features of the rst3 command used in LeoDocs.leo.

Overview

This project has been a great success:

- The rst3 command should work exactly as before for those who use only the basic features. If not, please report a bug.

- Regenerating all of Leo's web site appears to work.

- The rst3 sources, in leoRst.py, are now straightforward. All evil kwargs are gone, the call chains are much simpler, and much other cruft is gone.

- The simplified code in leoRst.py allows a new, simple, general way for Leonistas with scripting skills to create bespoke content from existing nodes anywhere in an outline. I'll discuss this new scheme below.

Aha: Writers alone are responsible for content

The supposed limitations of Leo's rst3 command have been the subject of discussion for about two decades. For example, there have been many requests for "half clones," clones that don't contain their descendants. The faux-advanced rst3 features were a feeble attempt at a workaround.

The advanced features are misguided because writers alone are responsible for the content of @rst trees. rst3 merely supports organizing the content with outlines.

Writers are free to use clones within @rst trees if that works. If not, writers must create bespoke content, possibly by cutting and pasting text.

This principle cuts the Gordian Knot. But do not despair. I will soon implement a sensational new way for Leonistas with scripting skills to create bespoke content from existing nodes anywhere in an outline. Let me explain.

Custom content filters

The rst3 command creates sources from @rst trees. These sources can be written to intermediate files or passed to docutils and written to so-called docutils files.

In the new code, rst.writeNode creates all the sources.

writeNode writes headlines (p.h) as follows:

self.result_list.append(self.underline(p.h))

writeNode writes bodies (p.b) this way:

self.result_list.append(p.b)

Just two lines of code do all the work. Well, suppose we allow Leonistas to define headline and body filters, and use those filters as follows:

self.result_list.append(self.underline(self.filter_h(p)))

and

self.result_list.append(self.filter_b(p))

@data headline-filter and @data body-filter will define the filters.  The default filters will return p.h and p.b.

On startup, rst.reloadSettings will compile the scripts in the two @data nodes. If successful, the filter_b and filter_h methods will wrap the user-defined filters. reloadSettings will use the default filters if the user-defined filters fail to compile.

Given position p, filters have complete access  (via c = p.v.context) to the entire outline! Filters can return any string they like, including '' or None.

It gets better. Writers can store their "intentions" anywhere they like, including:

- In headlines of certain kinds, including, say, headlines matching regular expressions.

- In bodies of certain kinds, including comments, doc parts, or plain text matching patterns defined, say, by regular expressions.

- In uA's (p.v.u)

- In caches.

In short, a filter can use p.b, p.h, p.u, or p.gnx for p itself, but the filter can use p (and c = p.v.context) to access any ancestor or descendant node, or indeed any other node in the outline.  Nothing could be simpler or more general.

Aha! Icons can (optionally) mark nodes containing uA's. So writers can, if they want, see the nodes that will result in special handling by the rst3 command! This Aha clarifies why I won't do #588.

Summary

The simplified rst3 command will work exactly as before for most Leonistas.

Aha: Writers are solely responsible for the contents of @rst trees. This principle should end decades of discussions.

Aha: Content filters will modify how rst3 handles @rst trees in a completely general way. Event filters have (using p) full access to the entire outline, including all ancestors and descendants of p.

Aha: Leonistas can use icons (defined in uA's) to mark the presence of any other uAs, including uA's that contain the writer's "intentions," that is, uA's that control the new filters.

In short, simplifying the rst3 command has created three huge Aha's.

Edward

P. S. Python's text tools are evolving

Tools such as sphinx and docutils continue to become more capable. For example, sphinx now uses pygments for all syntax coloring. There is no need to support SilverCity or provide settings for code blocks.

As a result of these improvements, I am simplifying the @rst trees in LeoDocs.leo.

EKR

[Edward K. Ream]

On Monday, March 29, 2021 at 6:22:34 AM UTC-5 Edward K. Ream wrote:

Aha: Content filters will modify how rst3 handles @rst trees in a completely general way. Event filters have (using p) full access to the entire outline, including all ancestors and descendants of p.

Some further comments:

Instead of using @data nodes to define filters, it makes more sense to use scripts or plugins. I'll create an example plugin soon.

rst3 neither knows nor cares how filters work. Anything goes. For example, a filter could run p.b through a templating engine like jinja2.

Once can imagine filters giving the effect of the legacy rst3 command, but I'll create such filters only if someone desperately needs them. I'll probably will include a few simplified utilities from the old code, like the code that finds and parses option @doc parts.

Edward