Giant Aha: @shadow *is* safe

[Edward K. Ream]

The following post is long, but imo it is one of the most important in
Leo's long history of long posts :-)

This post finally, indisputably, proves that @shadow is safe, and
proposes a consolation prize for the long delay in implementing
@shadow, namely @auto-shadow.

This is a continuation of the thread, leoShadow.py: first light. In
the last post to that thread I said:

QQQ
The proximate cause of the failed unit test is an overly-strict test
in test_propagate_changes. A lesser test at the end of
propagate_changed_lines succeeds for both unit tests. This test
ensures that the sentinel lines and non-sentinel lines *separately*
match the expected sentinel and non-sentinel lines. This test is
clearly too lax because it *completely* ignores the interleaving
between sentinel and non-sentinel lines.
QQQ

Last night in bed, thinking about what I had just written, I realized
that the failed unit test simply is not a problem. There are two
reasons for this statement:

1. The public file produced by the "failed" unit test will be
identical with the original (that is, changed) public file. Thus,
regardless of where @shadow puts the inserted lines, there will be no
difference that can be seen outside of Leo. But this 'round trip of
public files' is the essential requirement!

2. Furthermore, once the Leo user saves the @shadow file, the location
of all newly inserted lines will be "fixed" (determined). That is,
the next time Leo opens the @shadow file, the @shadow code will "know
for sure" where the line should go. The reason is simple: the line
will be placed by the "equal" SequenceMatcher opcode, rather than the
"insert" opcode. Thus, once the Leo user chooses the location of
newly inserted lines, the @shadow code will know (or remember) the
location of those lines, because they will be part of the original,
unchanged, *private* file. The sentinels in the private file will
tell where to put the line unambiguously.

In short:

1. **Boundary conditions do not affect public files**, the files seen
by people who do not use Leo.

2. Once an inserted line has been saved (thereby generating a new
public file), Leo will forever afterward remember the location of the
file.

Thus, the *only* remaining detail, a minor one, is to alert the user
when changes to a public file results in change to the Leo outline.
But Leo already warns when such external changes are made. If the
user does nothing, the corresponding public file will remain as it
was.

Imo, this discussion clearly demonstrates that @shadow is safe. This
is a huge step forward.

This Aha came, in no small part, from the change in terminology from
"files with/without sentinels" to "private/public files". When we
focus on what the public sees, we see that @shadow is safe. Is just
about that simple.

So the executive summary is:

Boundary problems while scanning changed public files can affect only
*private* view of the Leo outline. The public file will remain
unchanged regardless of how boundary conditions are resolved. The user
can change her private view as she likes, and once she has done so and
saved the @shadow file, the change will become permanent: it is
embodied in her private files.

Hurray! Clarity! Safety!

So what to do about the failed unit test? Let's reconsider the
opening quote:

QQQ
The proximate cause of the failed unit test is an overly- strict test
in test_propagate_changes. A lesser test at the end of
propagate_changed_lines succeeds for both unit tests. This test
ensures that the sentinel lines and non-sentinel lines *separately*
match the expected sentinel and non-sentinel lines. This test is
clearly too lax because it *completely* ignores the interleaving
between sentinel and non-sentinel lines.
QQQ

Now it should be clear that the essential test is the test that non-
sentinel (public!) lines are preserved. Thus, the test at the end of
propagate_changed_lines is *not* too lax: the test that sentinel
(private!) lines are preserved is nice to have, but preserving public
lines is the acid test.

So the solution to unit tests that fail at the boundary of nodes is
simply to relax the test. I'll do this by marking such tests with
@shadow-test-lax, which will cause the actual UnitTest test case not
to perform the stricter test that fails. How easy is that?
Naturally, I'll put comments in each @shadow-test-lax node explaining
why a looser test is required.

The Aha just discussed will make Leo famous.

I have been lamenting to myself recently that I didn't see all of this
when Bernhard Mulder first proposed shadow files. I'm not sure how
clearly he understood all the issues I have just discussed, but in any
case, the responsibility for dismissing shadow files is mine and mine
alone.

However, in the spirit of 'make lemonade from lemons', it may be that
the long delay in recognizing @shadow's true importance has had some
benefits. In particular, @auto might not have happened had @shadow
been adopted earlier.

What good is @auto now that we have @shadow, you ask? I reply in two
ways:

1. @auto is useful in its own right, for study, etc, quite
independently of whether people use @shadow.

2. We can eliminate some potential problem with both @auto and @shadow
by using @auto-shadow. Yes, I'll provide abbreviations for this.

@auto-shadow does what the name implies: It uses @auto logic to import
the file if no private (shadow) files exist for the file; otherwise it
uses the @shadow logic. This will smooth the process of creating and
using other people's outlines. @auto-shadow will ensure that
everything "just works"

A great day for Leo.

Edward

[Edward K. Ream]

On Jul 10, 8:45 am, "Edward K. Ream" <edream...@gmail.com> wrote:

> 2. We can eliminate some potential problem with both @auto and @shadow
> by using @auto-shadow. Yes, I'll provide abbreviations for this.

Doh! The abbreviation should be @shadow :-) That is, @shadow should
work as I proposed @auto-shadow to work.

Edward

[Edward K. Ream]

On Jul 10, 8:45 am, "Edward K. Ream" <edream...@gmail.com> wrote:

> This Aha came, in no small part, from the change in terminology from
> "files with/without sentinels" to "private/public files".

For the record, another important simplification was Bernhard's use of
reader and writer classes. In an algorithm as visually complex as the
main opcode loop, any visual simplification can be significant.
Furthermore, these classes were perfect for adding debugging and
tracing info. I wouldn't willing do without them.

Edward