Huge improvements to @clean are in devel

[Edward K. Ream]

PR #4386 is now in the "devel" branch.

This PR makes huge improvements to @clean. We have seen nothing like it since @clean first appeared in April 2015. We can now use @clean nodes on huge repos!

This PR fixes bugs and hangnails we've been living with since day one:

- Leo will detect changes to external @clean files, whether or not Leo is running.

- Leo leaves whitespace unchanged when importing to @clean or @auto. Why has nobody complained before?

- When importing or updating @clean trees, Leo will move leading blank lines to the start of the previous node. These moves happen only in newly-updated nodes, so you can "undo" these moves if you like. But such moves are almost always what we want.

A spectacular new feature: the update review tree

Whenever Leo detects updated nodes in @clean trees, Leo will create a new top-level node called Updated @clean/@auto nodes. Let's call this node and its descendants the update report tree:

- This top-level node contains update organizer nodes  for each updated file. The name of these nodes has the form: Updated from: whatever.py.

- Leo creates clones of each updated node as children of these organizer nodes.

- Leo puts the diffs of all the updated nodes from each file in the appropriate update organizer node.

The update review tree ranks with Leo's killer git-diff and cff commands! You can retain, change, or delete any part of the update report tree as you like:

- Deleting any part of this tree has affect any external file.

- As with the cff and git-diff commands, changing cloned nodes will change the corresponding external file.

Conclusion

PR #4386 is a huge step forward for Leo. Please report any problems immediately.

Edward

[Edward K. Ream]

On Tuesday, July 15, 2025 at 6:55:02 AM UTC-5 Edward K. Ream wrote:

> Deleting any part of [the update report tree] has affect any external file.

Oops: I should have said:

Deleting any part of this tree has no affect on any external file.

Edward

[Thomas Passin]

I've had a few errors already but they have only been partly repeatable. The context is that I have an outline that contains an @clean README.txt node. Currently its text displays with  blue color which is not what I want.  So I opened the node in an external editor with Leo also open and the outline selected.

I added a new line to the top of the file in the external editor: @nocolor. Leo picked up the change, added the changed notification and clones, and displayed the text in white.  So far so good.  Then I did the inverse - removed the @nocolor line using the external editor. Here's where I started to have problems.

I've tried cycling through this change several times.  Once Leo crashed when I tried to navigate between the clone and the original node position using the forward-back arrows.  Once the change didn't appear in the original node. There are a lot of mix-and-match combinations, and I haven't tried them all systematically yet: Save the outline between operations, or not; delete the clone nodes or not; edit the @nocolor line in or out in Leo and then do the opposite in the external editor; ...

I can't be more definite yet because I haven't been systematic enough but I have definitely had erroneous operations.

[Edward K. Ream]

On Tuesday, July 15, 2025 at 8:01:45 AM UTC-5 tbp1...@gmail.com wrote:

I've had a few errors already but they have only been partly repeatable. The context is that I have an outline that contains an @clean README.txt node. Currently its text displays with  blue color which is not what I want.  So I opened the node in an external editor with Leo also open and the outline selected.

I added a new line to the top of the file in the external editor: @nocolor. Leo picked up the change, added the changed notification and clones, and displayed the text in white.  So far so good.  Then I did the inverse - removed the @nocolor line using the external editor. Here's where I started to have problems.

Many thanks for your testing. I'll take a look today.

Edward

[Edward K. Ream]

On Tuesday, July 15, 2025 at 9:52:29 AM UTC-5 Edward K. Ream wrote:

Many thanks for your testing. I'll take a look today.

See #4394. 

Edward

[Edward K. Ream]

On Tuesday, July 15, 2025 at 6:55:02 AM UTC-5 Edward K. Ream wrote:

PR #4386 is now in the "devel" branch.

This PR makes huge improvements to @clean.

Félix asks whether the generating the update review tree is optional, implying that perhaps it should be optional.

However, let's think about the typical use case for updating @clean trees. We do this ourselves during testing, but in practice we expect to have Leo update our @clean trees when someone else changes the code, say when we pull new code into our branch.

In such cases the update review trees (including the diffs) should be exactly what we want to come up to speed with the changes.

Edward

[Edward K. Ream]

On Tuesday, July 15, 2025 at 8:01:45 AM UTC-5 tbp1...@gmail.com wrote:

I've had a few errors already but they have only been partly repeatable.

I added a new line to the top of the file in the external editor: @nocolor.

Leo directives are a special case that I have not fully considered before. Here's what I know:

First, Leo never writes Leo directives (or sentinels) when writing @clean trees. That's fine, because we assume that the external files should (and do) know nothing about Leo!

Second, the @clean update algorithm will read any line (including Leo directives) because the algorithm knows nothing about Leo or importers. It's driven purely by diffs. But that shouldn't matter either.

In other words, there is nothing very special about @color or @nocolor. If you add them to an external file, Leo will keep on deleting them (as it should) when rewriting the file.

The crashes are another matter. I suspect a missing call to c.recolor may be the culprit. My first thought is to call c.redraw_later() at the end of at.clone_all_changed_vnodes. Traces confirm that the desired tree redraw eventually happens. I don't want to call c.redraw because that would force an immediate redraw. In turn, that might cause unwanted flash.

Edward

[Félix]

About generating top level nodes "update review tree" automatically upon @clean refreshed from external file.

Please let me expose the reasons I strongly think this specific feature (popping up new nodes in the outline upon refreshing/reading @clean files) should be an opt-in via a setting.

Right out of the gate, I'll address what to me was the most important thing: Leonistas use Leo in very different ways! (I learned that repeatedly when coding leointeg because people were asking for features related to a Leo usage that never crossed my mind!)

The reasons for my insistance of an opt-in setting for generating 'update review tree' nodes automatically on @clean refreshes:

I personally, and also all the people I know in my area that use Leo for coding, (mostly for markdown, JSON, SVG graphic files, html and other file types) often use @clean instead of @file because we cannot have sentinels in those files. (as opposed to the way you've described where you thought that it was for when someone else changes the code, or when we pull new code) 

...and we edit our code freely in Leo AND in external editors all the time ourselves without even thinking about it, switching back and forth rapidly and saving every few seconds here and there.
 
But what I want to underline, is that the ability of Leo that we use all the time (well actually 'half' of the time) is the flexibility of editing either in Leo, or in an external editor, 50/50 without even having to think about it, and that the other (leo vs external-editor) updates automatically.

For example, we use Leo to visually structure our modules, subsections, classes and methods, or documentation in a Leo outline: the various chapters and articles in a manner that makes the most sense when looked at as an outline-tree. (which is Leo's main selling point) Then we alternate between Leo and some external editor: for example, inkscape to edit visually an SVG drawing, or some program in an external editor with some bells & whistles that help in coding. (e.g. vscode via LeoInteg to get all the goodies of coding suggestions-as-you-type, A.I. documentation on hover on any word, etc... which are of great help in speeding up coding) and thus, many @clean in our Leo outline get refreshed very often, but never because of "when someone else changes the code, say when we pull new code into our branch." to quote your above message.

Another example: I have a text editor for JSON files (for configuration files) which suggests and autocompletes the various parts of the JSON structures that are allowed in that specific config file. Leo is great for organising and presenting everything, but having 'diff/change-details' nodes appear by default in the outline every time I edit some file in my project externally from Leo would be annoying to say the least.

(by the way, that specific feature of not caring to edit in either leo or an external editor and have the other update instantly was the major selling point for the adoption of Leo in my experience, so telling someone "oh you have to know how to get to the myleosettings and find a specific setting and change its headline to 'False'" would be awkward somehow if there was nodes appearing in the outline after refreshing @clean nodes)

An additional and important point about not disturbing the outline unexpectedly: when working on documentation, (usually markdown or plain text) I will have a leo-script that loops all top-level node that produces a table of contents with back-links to those chapters. Having a new and unexpected node appear in the outline at any moment that some @clean in my project updated itself from file would be problematic.

(as it would for any other leo-script that loops the outline to do stuff or generate something, would be thrown off by having top-level nodes that would appear unexpectedly because of a refresh from file of any @clean present in the outline prior to running that script.)

So my point after all this preamble is that we are hitting CTRL+S at every 30 seconds in some editor and expect Leo to refresh (SVG, JSON, md... @clean nodes)  as it has always done without popping up unexpected nodes.

Although Leo is great for importing and inspecting+learning a codebase, We dont use it to do that most of the time.

Conclusion: I really appreciate and admire the improvements you have made to the refreshing of @clean nodes from file, (and I'm about to incorporate them in LeoJS) but most people do not use Leo like you do, and because of that, I think a setting (maybe "at-clean-detailed-diff" ) should be turned on to get this specific feature of having nodes appear in the outline to detail the changes that were detected.

Thanks for taking the time to read this, Edward. I know how much thought and care you put into Leo. I offer this suggestion with the utmost respect for your vision, but also out of a deep sense of how essential this behavior is for workflows like mine, and many others I’ve come across. I truly hope we can find a way to make that specific part (the generating the update review tree) an opt-in via a setting - or even a menu toggle, so that Leo continues to feel seamless and nonintrusive for all types of users.

With appreciation,
Félix

[Edward K. Ream]

On Tue, Jul 15, 2025 at 7:18 PM Félix <felix...@gmail.com> wrote:

Please let me expose the reasons I strongly think this specific feature (popping up new nodes in the outline upon refreshing/reading @clean files) should be an opt-in via a setting.

I'm convinced. I'll add a setting soon.

Edward