Leo's new beautifier: progress report

[Edward K. Ream]

Leo's beautify-files command now uses the Orange class in leoAst.py. This class is a proof of concept for #1440, unify the ast and token worlds. I've been working on this project for almost four months. It is nearing completion. Further testing remains.

Yesterday the beautify-files command passed a major milestone: all of Leo's core files compile without syntax errors after being beautified :-) No, I haven't committed any of the changes: that will require careful perusal of diffs.

Commands

The beautify-files commands beautifies all @<file> nodes in the selected outline.

The diff-beautify-files command (aka beautify-files-diff) reports the diffs (to the console) that would result from running the beautify-files command.

Settings

These commands take the following settings, with the indicated defaults:

@bool beautify-allow-joined-strings = False

By default, the beautifier will refrain from joining lines if they contain strings. This is my strong preference, because it suppresses rearranging of code such as:

message = (
    f"\n"
    f"  contents: {contents!r}\n"
    f"  expected: {expected!r}\n"
    f"       got: {results!r}")

I use this pattern all the time now. I would be most annoyed if some "helpful" beautifier collapsed it.

@bool beautify-keep-blank-lines = True

As I write this, I'm not sure this option works, or is even in effect. It's on the list of things to investigate.

@int beautify-max-join-line-length = 80
@int beautify-max-split-line-length = 90

Notice: these two settings have different defaults. The beautifier will join lines only if they won't soon be candidates for being split. Zero suppresses splitting or joining.The beautifier ensures that the join length is no larger than the split length.

In any case, the beautifier will join lines only if they have not just been split ;-)

Enabling and disabling beautification

Within Leo, @nobeautify suppresses all beautification until the end of the node. It's probably best to put this directive only a the start of a node. See below for an important warning.

The beautifier supports two special comments:

# pragma: no beautify
# pragma: beautify

These will allow a stand-alone version of the beautifier to suppress beautification in a range of lines.

Warning: Beautification catastrophes could occur if 'indent' and 'dedent' tokens are not matched within the "verbatim" range. It's probably best to use @nobeautify only at the start of a node. I might add support for @beautify, but that increases the danger. More noodling is required.

To do

1. 100% coverage of all non-debugging code.

Full coverage guarantees important properties. It will be an important advertising point when I announce this project.

2. Merge fstrings branch into devel.

This will be done as soon as full coverage is achieved.

3. Beautify all of Leo's core files.

This will be done in a separate branch. Imo, this work should be done before closing #1440.

4. Announce #1440 to the world.

The core python devs should definitely know about this work. What they do with it is up to them.

Summary

Leo's new beautify-files command is now a worthy competitor to black. It contains important settings and features not found in black.

Please feel free to play with the beautify-files command and report any problems. Make sure you have backups of to-be-beautified files. In particular, take care with @nobeautify.

Unit tests guarantee that Leo's beautify works exactly like black for all of the pet peeves listed in pep 8. To do this, the beautifier must know which parse tree node corresponds to each ":" token. This single datum justifies using the entire TokenOrderGenerator class!

Completing a major project such as #1440 is itself a significant project. Surely #1440 is much nearer completion than it was a few days ago ;-)

Edward

[Edward K. Ream]

On Tue, Feb 11, 2020 at 7:02 AM Edward K. Ream <edre...@gmail.com> wrote:

Yesterday the beautify-files command passed a major milestone: all of Leo's core files compile without syntax errors after being beautified :-)

That's no longer true, if it ever was. I'll let you know when I can recommend using beautify-files.

Edward

[Edward K. Ream]

On Tuesday, February 11, 2020 at 7:02:02 AM UTC-6, Edward K. Ream wrote:

[Monday] the beautify-files command passed a major milestone: all of Leo's core files compile without syntax errors after being beautified :-)

As of rev f8fc59 it should be pretty safe to try out the new beautify-files command. Again, make sure you have backups of any to-be-beautified file.

Yesterday's problems arose from the code that supports @nobeautify.  Rev f8fc59 fixes those problems, with just 6 lines of code. All files in LeoPyRef.leo beautify without (apparent) problems, and all unit tests pass on the beautified files.

I won't ever blindly check in beautified files. Unit tests do not suffice to give sufficient confidence. Instead, I'll carefully examine the diffs. I'll also ensure that beautifying each file a second time introduces no new changes.

Summary

The beautify-files and diff-beautify-files are ready for careful testing. Back up your files first.

These commands know about Leo sentinels, that is, Leo's outline structure and directives.

Support for @beautify/@nobeautify (#pragma: beautify / #pragma: nobeautify) is a feature that black lacks. It would likely be difficult for the black devs to add similar features.

Edward

[Edward K. Ream]

On Wednesday, February 12, 2020 at 6:23:45 AM UTC-6, Edward K. Ream wrote:

As of rev f8fc59 it should be pretty safe to try out the new beautify-files command. Again, make sure you have backups of any to-be-beautified file.

Still true.

I have beautified all of Leo's files in the fstrings branch. This revealed several minor bugs which will be fixed soon.

More importantly, I saw that splitting and joining lines involve subtle matters of policy that I had not considered at all. For now, my preference is never to join lines.

For example, when I have taken the trouble to call or def arguments on separate lines, I see no reason to undo that just because they actually could fit on a single line. It's not as if anyone is going to print the code, so "saving" lines makes no sense.

Summary

I'll check in the changes to devel when I have beautified and fstringified all of Leo's files to my satisfaction. This will likely take a day or two.

Edward

[Edward K. Ream]

On Thu, Feb 13, 2020 at 4:02 AM Edward K. Ream <edre...@gmail.com> wrote:

> ...it should be pretty safe to try out the new beautify-files command.

There is a problem with @doc and @ sections.

Leo uses a too-clever-by-half convention for determining whether blank lines in such sections are "real". At present, beautifying a file destroys that info. It will probably be possible to save some new state to deal with this, but I haven't done so yet.

Another known bug: -1 sometimes gets formatted as - 1. This will likely be fixed by consulting the ast node corresponding to one or more tokens. Failing unit tests cover this bug.

A recent rev fixed a nasty bug: beautifying files would insert too many blank lines at the end of the file. The result, within Leo, were unwanted @last directives.

Edward