A long discussion on Clones
Gil Shwartz
discussion I will try to analyze the use cases for clones, define some
terminology, possible problems when using clones, and some suggestions
to solve these problems as a basis for open discussion. I split this
_very_ long post to several sub-posts in order for each section to be
clearer. Please feel free to comment on any part.
Gil Shwartz
Actually I am familiar with only two use cases. I'll call the first
one “clones for convenience”. This use case involves cloning for the
sake of easier coding, flow understanding, etc. By his description,
Edward is using this use case a lot. The second use case is “clones
for synchronization”. I think someone described this use case as using
Leo as a macro processor. Indeed I find myself using Leo to sync
between files or portions thereof. It is especially useful for
languages that do not have functions, e.g. HTML. The expectations and
behavior of the users of these two use cases are quite different, but
I believe it is agreed by all the Leo must reflect the actual state of
external files it reads and not arbitrarily alter the content that it
presents to the user.
Gil Shwartz
When talking about a clone we need to differentiate between the nodes
in Leo's tree that share the same textual content and sub-tree, I'll
call them “clone nodes”, and the logical entity which is the clone
itself, so to speak, and is represented in Leo by a GNX (a unique
global identifier), I'll call it “clone entity”. Furthermore, there
are 3 different clone types to consider. I call the first “zero source
clone”. This is the type of clone in which all clone nodes are managed
exclusively by Leo (“internal nodes”), i.e. none of the clone nodes
are a child of a file that is read back by Leo. The second is “single
source clone”. For this clone type, only one clone node (the “external
node”) is a descendant of a file that is read back by Leo, i.e. this
type of clone has only one clone node who's content may be changed
outside of Leo. The third and last clone type is “multi source clone”.
This is the most problem prune clone type and it has two or more clone
nodes who's content may be changed outside of Leo's control (“external
nodes”). These may even be two clone nodes in the same derived file.
Gil Shwartz
Considering “zero source clone” we do not have any problem at all.
Since Leo is the only one that manages the content of the clone nodes,
it makes sure that all clone nodes are in sync all the time (bugs
permitted). Attempting to modify the Leo XML file out side of Leo is
always a high risk operation, but actually, Leo's XML data structure
protects us from causing clone problems by keeping only a single copy
of the clone entity's content. Thus, if this content in edited
directly outside of Leo, it will change the content of all
corresponding clone node. I.E. no problems at all.
Now we are ready to consider “single source clone”. This type of clone
primarily corresponds with the first use case “clones for
convenience” (as does “zero source clones”). For this type of clone it
might be the case that the content of the “external node” becomes out
of sync with the rest of the clone nodes that Leo manages. Since the
other clone nodes are typically copies of the “external node” for
convenience, it is quite rational to update them when Leo detects a
change in the content of the “external node”. This is what Leo
implements today with its strategy of “last node wins”. Since Leo file
reads the .leo file (with all the clone nodes in it) first and only
then starts reading derived files, the “external node” is read last
and the rest of the clone node are updated to its content. Easy,
simple and efficient. (By the way, in a future implementation, if one
wants to prevent this auto-update of the “internal nodes”, it can be
done simply by keeping a node of the clone in a “junk” file, forcing
the clone to be “multi source clone”, see next.)
Gil Shwartz
This is the difficult case, which is typically associated with the
second use case “clones for synchronization”. This is a powerful clone
type, which a user chooses to use and needs to be able to manage
properly and responsibly. The problem with “multi source clone” is
obvious – multiple clone nodes may be changed outside of Leo, causing
clone sync to break without an automagical method to fit and fix all
scenarios.
Let's start with what will probably be the most common case: all clone
nodes are in sync but one external node. This out-of-sync clone node
may not be the last clone node to be read, and Leo _must not_ change
its content or the user will loose touch with the real code. We are in
a situation in which there are two versions of clone content for the
same clone entity. Let's call the version of the out-of-sync clone
node version 1 and the version of the rest of the clone nodes version
2.
Assuming that this situation is properly flagged by Leo (see below)
and given some Leo tools to resolve this situation (see below as
well), the user might want to: i. de-clone the out-of-sync clone node
thereby converting it to a regular node, ii. change the content of the
out-of-sync clone node to content version 2, or iii. make the content
of the out-of-sync clone node (version 1) the content of all clone
nodes thereby discarding version 2. The user will choose i. if indeed
the content of the out-of-sync node should no longer be the same as
the other clone nodes (i.e. the second use case is no longer relevant
for this node). II. will be selected if the user determines the change
of the out-of-sync node to be accidental (i.e. the user reverts the
content to enforce the synchronization constraint). III. is the right
choice if the user determines that the change in the out-of-sync clone
node is appropriate and due to lack of use of Leo the person making
that change neglected to appropriately propagate it to all the right
places.
To make things even worse, this simple scenario may be further
complicated by option iv. change the content of some of the other
clone nodes to that of the out-of-sync node (version 1). This might
happen when some of the clone nodes are actually “clones for
convenience” of the out-of-sync clone node, while other clone nodes
are “clones for convenience” of at least another external node
(otherwise this would not be a “multi source clone” type).
To summarize, the following are useful options for the user to have:
a. De-clone this clone node (making it a regular node and keeping its
content version x).
b. Split-clone all clone content version x nodes (keeping them as
clones with content version x).
c. Re-sync this clone node with clone content version y (replacing
this node x by y).
d. Re-sync all clone content version x nodes with clone content
version y (replacing all x by y).
e. Re-sync all clone content version y nodes with this clone content
(replacing all y by x).
It may sound confusing at first, but simply put it is: turn A to B and
get rid of A (or vice versa), keep A and B separate, or just handle
this particular node.
Gil Shwartz
I think it is important to differentiate between “zero” and “single”
source clone types and “multi source clone” type. This can be done
easily by using double clone arrow for the latter clone type. Every
time a clone node is moved, Leo can evaluate if it becomes an external
clone node and if it does and the clone entity has more than one
external clone nodes, it will change the single clone arrow to a
double one (of course, doing vice versa when a clone node becomes an
internal clone node and there are only 0 or 1 external clone nodes).
It is worth noting that typically at the time of creation, a clone
will be “multi source clone” (if the cloned node is external,
immediately there are two clones in the same file). Later when one
clone node is moved elsewhere, many clone entities will become “single
source clone” types.
To assist in resolving clone conflicts (only existing for “multi
source clone” type), the version of each clone node should be known to
the user. The easiest way is to visualize it, e.g. by adding the
version number to the node's box. I do not anticipate most conflicts
to include many versions, but the solution should be reasonably
general. I believe it should be accompanied with some helpful goto
node functions:
1. Goto to next conflicting clone (moves between clone nodes of
different clone entities which have conflicts).
2. Goto the next clone node (moves between clone nodes of the same
clone entity).
3. Goto the next same version node (moves between clone nodes of the
same clone content version).
4. Goto the next other version node (moves between clone nodes of the
different clone content versions).
For a healthy clone, 2 & 3 are the same and 4 does nothing. I think 2
is useful in general for any type of clone.
I further find a suggestion previously made by others – creating a
“clone for convenience” of the conflicting clone nodes – useful, and
would like to expand on it. In case of conflict, it might be useful to
create a tree with the following structure (maybe, as suggested, at
the top of Leo's tree):
- Conflicting clone 1
-- Clone version 1.1
--- File abc.c
---- <name of clone node> - this body contains clone content version
1.1
--- File def.c
---- <name of clone node> - this body contains clone content version
1.1
-- Clone version 1.2
--- File ghi.c
---- <name of clone node> - this body contains clone content version
1.2
- Conflicting clone 2
-- Clone version 2.1
etc.
I suppose that in most cases there will only be 1 conflicting clone
with 2 content versions. There might be a lot of conflicts if the code
went through a major restructure, but indeed this should probably
break the existing “clone for synchronization” constraint.
TL
> define some terminology, possible problems when using clones,
Nice overview. Thanks for taking the time to create it.
TL
Edward K. Ream
> In this
> discussion I will try to analyze the use cases for clones, define some
> terminology, possible problems when using clones, and some suggestions
> to solve these problems as a basis for open discussion.
I'll get this the attention it deserves after fixing urgent bugs in
Leo. Maybe today :-)
Edward
TL
> of the same clone entity).
This is already available. You can issue the goto-next-clone command
on the command line or bind it to a key.
TL
Ville M. Vainio
zpcspm
> Or right click a cloned node and choose "go to clone".
Any additional requirements to have this working? In the qt GUI right
clicking gives me the menu provided by todo.py (qt branch of cleo.py
plugin), because I have todo.py enabled. In the tk GUI right clicking
on the node icon brings up the cleo.py menu, right clicking on the
node headline gets
exception handling headrclick event
AttributeError: 'position' object has no attribute 'isAtNorefFileNode'
printed in the log pane.
It's not like I'm using the mouse in leo a lot, I'm getting used more
and more to exclusive keyboard driving, because it's very handy. But I
occasionally try simple recipes like this for testing purposes, in
hope to catch eventual bugs.
Ville M. Vainio
> Any additional requirements to have this working? In the qt GUI right
> clicking gives me the menu provided by todo.py (qt branch of cleo.py
> plugin), because I have todo.py enabled. In the tk GUI right clicking
With qt, enable contextmenu.py, it has all the "default" rclick items.
-
zpcspm
> On Tue, Jan 26, 2010 at 8:07 PM, zpcspm <zpc...@gmail.com> wrote:
> > Any additional requirements to have this working? In the qt GUI right
> > clicking gives me the menu provided by todo.py (qt branch of cleo.py
> > plugin), because I have todo.py enabled. In the tk GUI right clicking
>
> With qt, enable contextmenu.py, it has all the "default" rclick items.
It works. I had an old version of ~/.leo/myLeoSettings.leo which
didn't contain contextmenu.py at all.
Since I pull the current trunk everyday, now here comes another
question: what would be a good way to keep in sync at least the
@enabled-plugins node in ~/.leo/myLeoSettings.leo with the one from
the global leo settings file that is part of the trunk an eventually
gets changed after pulling the last revision? Doing this visually
everyday would be annoying.
Concerning the tk GUI, the question remains. I don't recall doing
anything special (like manually enabling some plugins) in order to get
the popup menu when right clicking, but I might be wrong, because it's
been a while since I mostly use the qt GUI. Is this a bug? If so, is
there a need to fill a bug report at launchpad about this?
Gil Shwartz
Yes, you're right. I was a bit confused with find-next-clone, but goto-
next-clone is definitely it (although in the above context it should
not differentiate between content versions of the clone). Anyway, it
is good to have a complete description and encouraging to have some of
it already done.
Gil Shwartz
>
> TL
Thanks :)
Edward K. Ream
> Actually I am familiar with only two use cases. I'll call the first
> one “clones for convenience”. This use case involves cloning for the
> sake of easier coding, flow understanding, etc. By his description,
> Edward is using this use case a lot.
I call this creating views. And yes, this turns out to be extremely
convenient. I don't know how I would make complex changes to Leo
without the ability to focus my attention on the relevant code,
thereby ignoring everything else.
> The second use case is “clones
> for synchronization”. I think someone described this use case as using
> Leo as a macro processor. Indeed I find myself using Leo to sync
> between files or portions thereof. It is especially useful for
> languages that do not have functions, e.g. HTML.
I would say there are two separate use cases here:
A. Using clones *within* a file, as macros. This does indeed
compensate for HTML's (and XML's) lack of functions.
B. Using clones *across* files, to synchronize files. I regard this
case as dangerous. I use this capability in a weak way, to save
projects and to-do items in leoProjects.txt, leoToDo.txt, etc. These
files have @all directives, and due to the recent changes to Leo's
file code, these "@all-files" can never override clone definitions in
.leo files or other external files.
In other words, the point of the recent critical bug fix was to make
sure this use case is safe. Relying on Leo to keep non-@all external
files in sync using clones is not recommended, by me at least :-) You
are asking for pain due to a classic multiple-update problem.
Edward
Edward K. Ream
>
> When talking about a clone we need to differentiate between the nodes
> in Leo's tree that share the same textual content and sub-tree, I'll
> call them “clone nodes”, and the logical entity which is the clone
> itself, so to speak, and is represented in Leo by a GNX (a unique
> global identifier), I'll call it “clone entity”.
This is non-standard terminology. I'm not sure I approve.
In the one-node world, all joined nodes are the *same* node. This
includes clones, but it also includes all descendant nodes of clones.
For example,
- A'
- B
- A'
- B
It's important to understand that while the two instances of A and B
may look like different nodes on the screen, in fact, both "copies" of
A are the same node, as are both copies of B. Internally, the graph
looks like this:
Hidden Root Node [A,A]
A [B]
B [ ]
Here the [ ] notation denotes the children array of the node. So the
hidden root node contains two "pointers" to A. That's what makes A a
clone, in essence.
> Furthermore, there
> are 3 different clone types to consider. I call the first “zero source
> clone”. This is the type of clone in which all clone nodes are managed
> exclusively by Leo (“internal nodes”), i.e. none of the clone nodes
> are a child of a file that is read back by Leo. The second is “single
> source clone”. For this clone type, only one clone node (the “external
> node”) is a descendant of a file that is read back by Leo, i.e. this
> type of clone has only one clone node who's content may be changed
> outside of Leo. The third and last clone type is “multi source clone”.
> This is the most problem prune clone type and it has two or more clone
> nodes who's content may be changed outside of Leo's control (“external
> nodes”). These may even be two clone nodes in the same derived file.
Again, this terminology is non-standard. Leo doesn't have anything
internally that corresponds to this. The distinction, if one can call
it that, is enforced by Leo's read code, which in several places
decides what to do if two definitions of a node clash.
I agree that it may be good to be aware of this distinction when using
Leo. But Leo must work well even if users have no idea about this
distinction :-)
Edward
Edward K. Ream
> Zero and Single Source Clones
>
> Considering “zero source clone” we do not have any problem at all.
> Since Leo is the only one that manages the content of the clone nodes,
> it makes sure that all clone nodes are in sync all the time (bugs
> permitted). Attempting to modify the Leo XML file out side of Leo is
> always a high risk operation, but actually, Leo's XML data structure
> protects us from causing clone problems by keeping only a single copy
> of the clone entity's content. Thus, if this content in edited
> directly outside of Leo, it will change the content of all
> corresponding clone node. I.E. no problems at all.
True, but perhaps a bit confusing. The reason what you say is true is
that Leo reads .leo files completely before reading external files.
Furthermore, within a .leo file, all clones are represented by a
*single* <t> element. So there is essentially no way to modify one
clone and not the other. Attempting to "split" a node by modifying
<v> elements is not going to happen in practice, and if one did try
that the result would likely be a broken .leo file.
> Now we are ready to consider “single source clone”. This type of clone
> primarily corresponds with the first use case “clones for
> convenience” (as does “zero source clones”). For this type of clone it
> might be the case that the content of the “external node” becomes out
> of sync with the rest of the clone nodes that Leo manages. Since the
> other clone nodes are typically copies of the “external node” for
> convenience, it is quite rational to update them when Leo detects a
> change in the content of the “external node”. This is what Leo
> implements today with its strategy of “last node wins”.
That's probably true, but I don't recommend saving clones in multiple
non-@all external files.
In other words, Imo, the most important distinction is between
external files containing @all and external files that don't.
My suggested rule is this: when putting clones in external files, make
sure the clone appears in at most one external file that does not
contain an @all directive.
Leo's new read code gives top priority to clones in external files not
containing @all. The new read code gives lowest priority to clones in
external files that do contain @all.
> Since Leo file
> reads the .leo file (with all the clone nodes in it) first and only
> then starts reading derived files, the “external node” is read last
> and the rest of the clone node are updated to its content. Easy,
> simple and efficient. (By the way, in a future implementation, if one
> wants to prevent this auto-update of the “internal nodes”, it can be
> done simply by keeping a node of the clone in a “junk” file, forcing
> the clone to be “multi source clone”, see next.)
True, with the provisos mentioned above.
Edward
Ville M. Vainio
> My suggested rule is this: when putting clones in external files, make
> sure the clone appears in at most one external file that does not
> contain an @all directive.
Perhaps leo could report violations of this practice by a friendly g.es....
--
Edward K. Ream
> Multi Source Clones
>
> This is the difficult case, which is typically associated with the
> second use case “clones for synchronization”.
And this is the case I would like to discourage.
I have no commitment whatever to making this case work reliably, now or ever.
> The problem with “multi source clone” is
> obvious – multiple clone nodes may be changed outside of Leo, causing
> clone sync to break without an automagical method to fit and fix all
> scenarios.
Exactly. And if the "last clone wins" rule doesn't work, you are in
trouble of your own making. Leo can not and will not help you out,
other than to say that a node has unexpected changed. At present,
this warning is given in red.
> To summarize, the following are useful options for the user to have:
> a. De-clone this clone node (making it a regular node and keeping its
> content version x).
> b. Split-clone all clone content version x nodes (keeping them as
> clones with content version x).
> c. Re-sync this clone node with clone content version y (replacing
> this node x by y).
> d. Re-sync all clone content version x nodes with clone content
> version y (replacing all x by y).
> e. Re-sync all clone content version y nodes with this clone content
> (replacing all y by x).
>
> It may sound confusing at first, but simply put it is: turn A to B and
> get rid of A (or vice versa), keep A and B separate, or just handle
> this particular node.
Imo, these kinds of complexities are an indication that one is trying
to solve the wrong problem.
Edward
Edward K. Ream
> Gui Suggestions
>
> I think it is important to differentiate between “zero” and “single”
> source clone types and “multi source clone” type. This can be done
> easily by using double clone arrow for the latter clone type.
Easily? That's not proven. We could add a "multi-clone" bit to
vnodes, but now every single outline operation could affect that bit.
And how would we set the bit? We somehow would have to coordinate the
sax-read code with the atFile.read code.
> Every time a clone node is moved, Leo can evaluate if it becomes an external
> clone node and if it does and the clone entity has more than one
> external clone nodes,
I absolutely refuse to do this, and I will not accept a plugin that
attempts to do this. What you are suggesting would require a complete
scan of the entire .leo file for every move operation. It's a huge
performance hit.
More importantly, you can not do this to Leo and expect Leo to remain
robust. It may not be apparent, but Leo works as well as it does
because I have made it my top priority to avoid these kinds of global
interactions. Putting such code into Leo is the best way I know to
destroying its integrity.
As we have just seen, data problems are the most critical problems in
Leo. We simply can not add this kind of complexity to Leo.
Instead, I suggest you find ways to work around xml's limitations.
You might start with xslt.
Edward
Edward K. Ream
>> Every time a clone node is moved, Leo can evaluate if it becomes an external
>> clone node and if it does and the clone entity has more than one
>> external clone nodes,
>
> I absolutely refuse to do this, and I will not accept a plugin that
> attempts to do this.
Still true, but this remark should be taken with a grain of salt :-)
I'm not going to change Leo's data structures, but we can attempt to
solve the problem in much simpler ways.
After some thought, I realize the essence of the problem involves only
Leo's read code. There are several possible approaches.
1. The simplest thing that could possibly work. When a conflict
happens, presumably in the atFile.read code, Leo will remember the
completing versions in, say, c.conflictingNodesList. Entries will
record the vnode and all the conflicting versions of the text. A
plugin could use an open2 hook to attempt to choose the proper
version, possibly by putting up a dialog.
2. More complicated, but perhaps more useful. Mark some trees with an
@master directive. Presumably, @master and @all would work together
to provide a *single* external file that contains master copies of
data. It's vital to understand that being a "master" node in this
sense affects only Leo's read logic, and *nothing* else.
3. Scripting solutions, within Leo or without (xslt). Always open
ended, but I have no great ideas at present.
Just the start of some ideas. I wanted to throw this out quickly so
people will realize I'm not as negative on this subject as my initial
words may have implied.
Edward
Edward K. Ream
>> Every time a clone node is moved, Leo can evaluate if it becomes an external
>> clone node and if it does and the clone entity has more than one
>> external clone nodes,
>
> I absolutely refuse to do this, and I will not accept a plugin that
> attempts to do this.
I want to say a few more words about this. The reason I can reject
this idea so quickly is it implies far-reaching changes to Leo that
are not needed, and that are not at all pretty.
One could say that the essence of the proposal is that it would
change v.isCloned. Here it is:
def isCloned (self):
return len(self.parents) > 1
Not only is this extremely simple and fast, more importantly, being a
clone is thus a property *only* of Leo's nodes. We simply can not
allow what is now an intrinsic property of data to become an extrinsic
property of the outlines overall structure.
For example, *every* Leo command would have implications for
cloned-ness. Changing @thin to @@thin would have global implications.
I am not going there.
Happily, we can devise much simpler approaches to dealing with
read-time problems.
Edward
Gil Shwartz
In general, though, I was trying to provoke some thinking on the
greater uses of clones, beyond what is currently supported by Leo, in
the hope that it can be refined through discussion to something which
may expand Leo's functionality and increase its usefulness even more.
> > The second use case is “clones
> > for synchronization”. I think someone described this use case as using
> > Leo as a macro processor. Indeed I find myself using Leo to sync
> > between files or portions thereof. It is especially useful for
> > languages that do not have functions, e.g. HTML.
>
> I would say there are two separate use cases here:
>
> A. Using clones *within* a file, as macros. This does indeed
> compensate for HTML's (and XML's) lack of functions.
>
> B. Using clones *across* files, to synchronize files. I regard this
> case as dangerous. I use this capability in a weak way, to save
> projects and to-do items in leoProjects.txt, leoToDo.txt, etc. These
> files have @all directives, and due to the recent changes to Leo's
> file code, these "@all-files" can never override clone definitions in
> .leo files or other external files.
>
> In other words, the point of the recent critical bug fix was to make
> sure this use case is safe. Relying on Leo to keep non-@all external
> files in sync using clones is not recommended, by me at least :-) You
> are asking for pain due to a classic multiple-update problem.
>
> Edward
I accept the differentiation between A & B, but I could not point to
any practical difference in handling them. A may still result a clone
conflict (being "multiple source" one), as, of course, B.
I am not sure that @all is the key thing here, while, again, I
recognize the this is the current state of Leo. I think the more basic
distinction should be between @file like directives (@file in the
greater sense of @thin, @shadow, etc.) for which Leo reads the
external content, and which therefore may result with clone conflicts,
and those for which it does not.
I think that any case where Leo reads an external file and does not
reliably represents its content should be eliminated. Otherwise, Leo
presents a false picture to the user of the content of the
environment. If I analyze correctly, this may currently happen in the
following situations:
1. On a clone conflict, where currently the last node wins and this is
problematic for "multiple source".
2. Using @all, which is something a bit new, described above. Is that
good? It is a difficult logic (if I captured it right) - Leo reads the
external @file, but runs down any clone within it, which Leo already
knows about.
Gil
Gil Shwartz
> > in Leo's tree that share the same textual content and sub-tree, I'll
> > call them “clone nodes”, and the logical entity which is the clone
> > itself, so to speak, and is represented in Leo by a GNX (a unique
> > global identifier), I'll call it “clone entity”.
>
> This is non-standard terminology. I'm not sure I approve.
Sorry about that - I just needed to invent some terms to help
differentiate between different cases and to explicitly name thing. I
felt that "clone" is used to too many parts of the concept. I am
willing to adopt any terminology you feel comfortable with, as long as
it has distinctions for the various parts that play in a clone to we
can discuss them clearly.
> In the one-node world, all joined nodes are the *same* node. This
> includes clones, but it also includes all descendant nodes of clones.
I completely agree. This is what I meant by "share the same textual
content and sub-tree", but you probably describe it better.
> For example,
>
> - A'
> - B
> - A'
> - B
>
> It's important to understand that while the two instances of A and B
> may look like different nodes on the screen, in fact, both "copies" of
> A are the same node, as are both copies of B. Internally, the graph
> looks like this:
>
> Hidden Root Node [A,A]
> A [B]
> B [ ]
>
> Here the [ ] notation denotes the children array of the node. So the
> hidden root node contains two "pointers" to A. That's what makes A a
> clone, in essence.
The Hidden Root Node is what I called the "clone entity". The "clone
nodes" are the two occurrences A', which are different nodes in the
tree. The content of A, including B and its content, are the "clone
content" (or later "content version").
> > Furthermore, there
> > are 3 different clone types to consider. I call the first “zero source
> > clone”. This is the type of clone in which all clone nodes are managed
> > exclusively by Leo (“internal nodes”), i.e. none of the clone nodes
> > are a child of a file that is read back by Leo. The second is “single
> > source clone”. For this clone type, only one clone node (the “external
> > node”) is a descendant of a file that is read back by Leo, i.e. this
> > type of clone has only one clone node who's content may be changed
> > outside of Leo. The third and last clone type is “multi source clone”.
> > This is the most problem prune clone type and it has two or more clone
> > nodes who's content may be changed outside of Leo's control (“external
> > nodes”). These may even be two clone nodes in the same derived file.
>
> Again, this terminology is non-standard. Leo doesn't have anything
> internally that corresponds to this. The distinction, if one can call
> it that, is enforced by Leo's read code, which in several places
> decides what to do if two definitions of a node clash.
Of course Leo does not represent this internally. I think some of this
is new and represent a chance to further develop Leo.
Indeed the read code is a player in the clone game, but other parts of
Leo should participate as well. The thing on which the read code has a
monopoly on is the detection of clone conflicts and the creation of
whatever is decided that will help in clone resolution. This is yet to
through more discussion.
> I agree that it may be good to be aware of this distinction when using
> Leo. But Leo must work well even if users have no idea about this
> distinction :-)
>
> Edward
I totally agree. Many users will only use Leo clones for views ("clone
for convenience"), if at all. This will also result the current Leo
graphical representation - a single arrow. It will also never result
clone conflicts. However, we do not want these users to trip naively
on a clone trap, and the more advanced users might want to use Leo in
a more advanced way. I think Leo should be able to support them, and
therefore having this distinction explicitly is important (as a clones
user I was not aware of this until I wrote this overview). My believe
is that Leo should provide the tools and graphical representation to
differentiate and manage the more complex cases as well.
Gil
Gil Shwartz
> non-@all external files.
Indeed, currently Leo does not support well the case of "multi source
clone". Wouldn't we like to improve this?
> In other words, Imo, the most important distinction is between
> external files containing @all and external files that don't.
>
> My suggested rule is this: when putting clones in external files, make
> sure the clone appears in at most one external file that does not
> contain an @all directive.
>
> Leo's new read code gives top priority to clones in external files not
> containing @all. The new read code gives lowest priority to clones in
> external files that do contain @all.
I have referred to this in a reply above. I find this logic very
confusing and use case specific.
Gil
Gil Shwartz
> > second use case “clones for synchronization”.
>
> And this is the case I would like to discourage.
Why? Granted it is a bit more complex, but it is still a valid (as
well as a very important, in some cases) case. Moreover, I'm quite
certain that this discussion can lead to ideas and solutions that
would make dealing with this case a lot more simple than the first
impression.
> I have no commitment whatever to making this case work reliably, now or ever.
But we can try :)
> > The problem with “multi source clone” is
> > obvious – multiple clone nodes may be changed outside of Leo, causing
> > clone sync to break without an automagical method to fit and fix all
> > scenarios.
>
> Exactly. And if the "last clone wins" rule doesn't work, you are in
> trouble of your own making. Leo can not and will not help you out,
> other than to say that a node has unexpected changed. At present,
> this warning is given in red.
Again, I'm trying to discuss future code, realizing the limitations of
the current one with respect to "multi source" type of clone.
> > To summarize, the following are useful options for the user to have:
> > a. De-clone this clone node (making it a regular node and keeping its
> > content version x).
> > b. Split-clone all clone content version x nodes (keeping them as
> > clones with content version x).
> > c. Re-sync this clone node with clone content version y (replacing
> > this node x by y).
> > d. Re-sync all clone content version x nodes with clone content
> > version y (replacing all x by y).
> > e. Re-sync all clone content version y nodes with this clone content
> > (replacing all y by x).
>
> > It may sound confusing at first, but simply put it is: turn A to B and
> > get rid of A (or vice versa), keep A and B separate, or just handle
> > this particular node.
>
> Imo, these kinds of complexities are an indication that one is trying
> to solve the wrong problem.
>
> Edward
Hopefully I can convince you otherwise, and other opinions are
welcome. I just need to go offline now, so I'll return with more a bit
later.
Gil
Gil Shwartz
> > containing @all. The new read code gives lowest priority to clones in
> > external files that do contain @all.
>
> I have referred to this in a reply above. I find this logic very
> confusing and use case specific.
>
> Gil
I was thinking about it more while offline and I believe I need to
sharp my level of understanding:
What is fundamentally special/different about @all (e.g. wrt @others)
that makes its clones of "lesser value"?
Gil
Gil Shwartz
> > source clone types and “multi source clone” type. This can be done
> > easily by using double clone arrow for the latter clone type.
>
> Easily? That's not proven. We could add a "multi-clone" bit to
> vnodes, but now every single outline operation could affect that bit.
> And how would we set the bit? We somehow would have to coordinate the
> sax-read code with the atFile.read code.
I think that the type of clone is a property of the clone entity (I am
not sure what the standard terminology is, but I am referring to what
is represented by <t> element in the .leo file, and by "Hidden Root
Node" in a previous comment of Edward). Therefore, that same way that
a change of clone content is immediately reflected by the clone node,
so does the bit setting of the clone type.
I do not think that this has anything to do with the sax-read
(assuming here you mean reading the .leo XML file). The bit need not
be stored in .leo. It is computed in run-time as clone nodes are added
to the clone entity. When reading the .leo file we only add internal
nodes to the clone so it is obviously of type "zero source". Later,
when external files are read, the first node that is added to a clone
causes the clone to be of type "single source". Any other node make it
"multi source".
I think this scheme is fairly simple and robust.
As a side comment, I also think that the node name should be an
attribute of the clone entity rather of the clone node (the <v>), but
this is less important for the current discussion (or maybe not, if we
decide on a fundamental treatment to clones).
> > Every time a clone node is moved, Leo can evaluate if it becomes an external
> > clone node and if it does and the clone entity has more than one
> > external clone nodes,
>
> I absolutely refuse to do this, and I will not accept a plugin that
> attempts to do this. What you are suggesting would require a complete
> scan of the entire .leo file for every move operation. It's a huge
> performance hit.
>
> More importantly, you can not do this to Leo and expect Leo to remain
> robust. It may not be apparent, but Leo works as well as it does
> because I have made it my top priority to avoid these kinds of global
> interactions. Putting such code into Leo is the best way I know to
> destroying its integrity.
I do not think there is a significant performance hit or global
interaction involved. The way I see it, a tree node already have run
time information of whether it is a clone node or not. This is
obvious, because when a node body is edited, it is immediately
reflected by its other clone nodes. So, for the above, only movement
of clone node is relevant and not every move operation. The number of
clone node is typically a lot smaller from the number of total nodes.
Moreover, when moving a clone node, we only need to determine if it
has changed state from "internal node" to "external node" and vice
versa. This is not a global operation, rather a simple traverse over
its parents to look for "@file", "@thin", "@shadow", etc. If we keep
the old route to the tree, it typically overlap much of the new route
following a single move, so we only need to evaluate to a common
patent or the root - whichever is simpler/more convenient/faster.
Obviously, this operation is bound by O(node depth), which really
cannot be considered global interaction.
I hope this makes it more realizable.
> As we have just seen, data problems are the most critical problems in
> Leo. We simply can not add this kind of complexity to Leo.
>
> Instead, I suggest you find ways to work around xml's limitations.
> You might start with xslt.
>
> Edward
I'm sure sure I understand the relation to XML here.
Gil
Gil Shwartz
> On Wed, Jan 27, 2010 at 8:56 AM, Edward K. Ream <edream...@gmail.com> wrote:
>
> >> Every time a clone node is moved, Leo can evaluate if it becomes an external
> >> clone node and if it does and the clone entity has more than one
> >> external clone nodes,
>
> > I absolutely refuse to do this, and I will not accept a plugin that
> > attempts to do this.
>
> I want to say a few more words about this. The reason I can reject
> this idea so quickly is it implies far-reaching changes to Leo that
> are not needed, and that are not at all pretty.
>
> One could say that the essence of the proposal is that it would
> change v.isCloned. Here it is:
>
> def isCloned (self):
> return len(self.parents) > 1
>
> Not only is this extremely simple and fast, more importantly, being a
> clone is thus a property *only* of Leo's nodes. We simply can not
> allow what is now an intrinsic property of data to become an extrinsic
> property of the outlines overall structure.
I understand this, and see no reason why it should change.
> For example, *every* Leo command would have implications for
> cloned-ness. Changing @thin to @@thin would have global implications.
> I am not going there.
This is a good point. However, it is relevant only for @file like
commands that affect the internal/external state of a clone node. I do
not think that changes like @this to @@thin (i.e. change from external
to internal) are done often to justify throwing this idea. Also, the
implication of such change (assuming it is not something like @file to
@shadow, which is of no interest from our perspective) is scanning the
sub-tree for clone nodes and incrementing or decrementing their clone
entity's external count. This is probably quite fast (and, I believe,
rarely done).
> Happily, we can devise much simpler approaches to dealing with
> read-time problems.
>
> Edward
Anything simple is good.
Gil
Gil Shwartz
> solve the problem in much simpler ways.
Always good.
> After some thought, I realize the essence of the problem involves only
> Leo's read code. There are several possible approaches.
>
> 1. The simplest thing that could possibly work. When a conflict
> happens, presumably in the atFile.read code, Leo will remember the
> completing versions in, say, c.conflictingNodesList. Entries will
> record the vnode and all the conflicting versions of the text. A
> plugin could use an open2 hook to attempt to choose the proper
> version, possibly by putting up a dialog.
I think this is definitely on the right track (even though I'm not
sure what c.xxx is). While a plugin can certainly be used to make
conflict decisions, maybe like the ones described earlier in this
discussion, Leo would still need to reliably display the exact
information read from the external files, and be able to manage that.
I thought that whenever the read code detects a clone conflict, it
create another clone entity. We can have the clone history if we use
ids (gnx) like this <original clone gnx>:<version info>. <version
info> can be <user id>.<datetime>.<version>, i.e. very much like the
current gnx structure. So, for example, if we started with a single
clone entity having gnx Gil.20100124231208.1, and the read code
detected two versions, the result will be two clone entities, the
first with gnx Gil.20100124231208.1:Gil.20100127231242.1 and the
second with Gil.20100124231208.1:Gil.20100127231242.2. These clones
can be handled by Leo the same way their are today, and a plugin can
offer merge and other functionality. If a version is split from the
clone it can have a new simple (regular) gnx.
> 2. More complicated, but perhaps more useful. Mark some trees with an
> @master directive. Presumably, @master and @all would work together
> to provide a *single* external file that contains master copies of
> data. It's vital to understand that being a "master" node in this
> sense affects only Leo's read logic, and *nothing* else.
I would say I'm against any solution that would make Leo discard of
file data it reads. This comes from my philosophical view that Leo is
a _perosnal_ meta-data structure that is layered over files (typically
source code files), and should therefore accurately represent the
exact file content at read time and later resolve meta-data structure
conflict if there are any. If @master is an explicit instruction by
the user that Leo discard of some read content, this is indeed a
preferred situation in which a informational notification is enough. I
would not force the user to use @master, so it might eliminate some of
the conflict if the user logic is strong (e.g. single user development
environment), but the rest of the conflicts are still in need to be
resolved.
> 3. Scripting solutions, within Leo or without (xslt). Always open
> ended, but I have no great ideas at present.
Not sure about this.
> Just the start of some ideas. I wanted to throw this out quickly so
> people will realize I'm not as negative on this subject as my initial
> words may have implied.
>
> Edward
Gil
Edward K. Ream
> I think that any case where Leo reads an external file and does not
> reliably represents its content should be eliminated.
I agree completely. This is an excellent characterization of the
problem: an external file supposedly represents its own content
completely, but *doesn't* because it loses a clone battle.
It seems to me that we are both saying the same thing. This case
should be avoided.
At present, I recommend avoiding it as follows.
1. Recent changes to Leo ensure that external files containing @all
always lose clone battles. This makes sense, imo, because such files
aren't used to create "real" source files.
2. I strongly discourage the case where two (or more) external files
*not* containing @all share the same clone. In that case, one clone
will win, and all other external files will lose. Not a good
situation.
BTW, Leo reads external files after reading their .leo file, so the
.leo file will always lose to an external file. Again, this seems
quite reasonable to me. We expect users to update external files from
time to time, but never (or hardly ever) to update .leo files in an
external editor.
> 2. Using @all, which is something a bit new, described above. Is that
> good? It is a difficult logic (if I captured it right) - Leo reads the
> external @file, but runs down any clone within it, which Leo already
> knows about.
I think it is the simplest thing that could possibly work. The logic
is simple: just at the point at which a node might win a clone battle,
we disable the update if we are in an external file containing @all.
The reason this is the simplest approach is that it is limited to one
or two well-defined places in Leo's read logic. It has *no* effect
outside of that read logic.
Edward
Edward K. Ream
> Indeed the read code is a player in the clone game, but other parts of
> Leo should participate as well.
This is exactly where we disagree. We can imagine extending the read
logic, say with a postpass, to handle clone battles more
intelligently. That would be a "contained" solution.
But to do more than that would be most unwise, imo. If we can not, at
read time, resolve clone battles, then we are asking for big big
trouble. The ramifications throughout Leo are simply mind boggling.
Leo's read and write code are stable because the are completely
stateless. We can't have unresolved data hanging around. It's just
out of the question.
The read code could certainly flag dubious resolutions and let the
user, perhaps with the aid of a plugin, undo whatever choices the read
code might make. But that's really a nit. The fundamental constraint
is that Leo's data *always* be in a known, stable state.
Edward
Edward K. Ream
> Again, I'm trying to discuss future code, realizing the limitations of
> the current one with respect to "multi source" type of clone.
>
>> > To summarize, the following are useful options for the user to have:
>> > a. De-clone this clone node (making it a regular node and keeping its
>> > content version x).
>> > b. Split-clone all clone content version x nodes (keeping them as
>> > clones with content version x).
>> > c. Re-sync this clone node with clone content version y (replacing
>> > this node x by y).
>> > d. Re-sync all clone content version x nodes with clone content
>> > version y (replacing all x by y).
>> > e. Re-sync all clone content version y nodes with this clone content
>> > (replacing all y by x).
We can imagine doing this, provided all these options do not actually
impact Leo's data structures in any way. The read code could leave
some "error recover" data lying around, for the use, say, of a plugin,
but that's about all that can be done, imo. That may be enough to
provide the kinds of options you suggest above. If it is, all well
and good.
Edward
Edward K. Ream
> What is fundamentally special/different about @all (e.g. wrt @others)
> that makes its clones of "lesser value"?
Good question! @all just stuff *unrelated* nodes into an external
file. It would be illogical to attempt to use the external file as
anything other than a grab-bag of data. Such bags are useful, as in
leoProjects.txt and leoToDo.txt, but bags are useful neither as source
files or as database files. In either of those cases you certainly
*do* want the structure and constraints given by @others.
Edward
Edward K. Ream
> I think that the type of clone is a property of the clone entity (I am
> not sure what the standard terminology is, but I am referring to what
> is represented by <t> element in the .leo file, and by "Hidden Root
> Node" in a previous comment of Edward).
Again, this is a major disagreement between us. What you are calling
the "clone entity" is, it seems to me, the same thing as the entire
Leo outline (the entire .leo file). I don't want any such
outline-wide data dependencies influencing clones!!!!
In contrast, the clone-status of any node is simply the number of
parents it has. This is a local property of data. True, the
clone-status of a node is not an intrinsic property of that node in an
external file--it depends on how many other nodes Leo has read. But
again, all this gets resolved at read time, and gets modified only by
Leo's low-level node methods. This is the way the situation *must*
remain.
> I do not think that this has anything to do with the sax-read
> (assuming here you mean reading the .leo XML file). The bit need not
> be stored in .leo. It is computed in run-time as clone nodes are added
> to the clone entity.
That would be too expensive and too dangerous. The property of being
in an external file depends on a large number of factors. Really.
The very last thing I want to do is do AI on the entire Leo outline
every time any part of the outline changes.
In contrast, something like @master is conceivable, as I said earlier.
It is a localize request that the node be handled with greater
respect by the read code. The write code could check that an @master
node not be contained in more than one external file.
> As a side comment, I also think that the node name should be an
> attribute of the clone entity rather of the clone node (the <v>), but
> this is less important for the current discussion (or maybe not, if we
> decide on a fundamental treatment to clones).
The name doesn't matter much because it can change. But the nodes
identity, its gnx, must be an intrinsic property of the node.
>> > Every time a clone node is moved, Leo can evaluate if it becomes an external
>> > clone node and if it does and the clone entity has more than one
>> > external clone nodes
That's one case. But renaming an @file node to @@file is another
case. Failure to find *all* the cases will create data disasters. It
is impossible *in principle* to find all the cases, because this
global constraint must be checked every time anyone makes any change
to Leo. It's hopeless, and an invitation to having the entire Leo
project fail
Edward
Edward K. Ream
>> After some thought, I realize the essence of the problem involves only
>> Leo's read code. There are several possible approaches.
>>
>> 1. The simplest thing that could possibly work. When a conflict
>> happens, presumably in the atFile.read code, Leo will remember the
>> completing versions in, say, c.conflictingNodesList. Entries will
>> record the vnode and all the conflicting versions of the text. A
>> plugin could use an open2 hook to attempt to choose the proper
>> version, possibly by putting up a dialog.
>
> I think this is definitely on the right track (even though I'm not
> sure what c.xxx is).
c is the commander, the "owner" of all the data in the outline. The
idea is for the read code to make a guess about about how to resolve
data conflicts, but then to save "resolution data" in the commander to
allow the user, with the help of a plugin, to make different
resolution.
> I thought that whenever the read code detects a clone conflict, it
> create another clone entity.
Another hidden root node? Too complicated. All the read code has to
do is save the info about conflicts somewhere. Anywhere will do, but
the commander is the logical place.
We can have the clone history if we use
> ids (gnx) like this <original clone gnx>:<version info>. <version
> info> can be <user id>.<datetime>.<version>, i.e. very much like the
> current gnx structure. So, for example, if we started with a single
> clone entity having gnx Gil.20100124231208.1, and the read code
> detected two versions, the result will be two clone entities, the
> first with gnx Gil.20100124231208.1:Gil.20100127231242.1 and the
> second with Gil.20100124231208.1:Gil.20100127231242.2. These clones
> can be handled by Leo the same way their are today, and a plugin can
> offer merge and other functionality. If a version is split from the
> clone it can have a new simple (regular) gnx.
The resolution data will indeed contain gnx's of conflicting nodes,
along with all the variants of the data in the battling nodes.
> I would say I'm against any solution that would make Leo discard of
> file data it reads.
Saving the conflicting data for later in the commander should satisfy us both.
Edward
Gil Shwartz
> On Fri, Jan 29, 2010 at 3:39 PM, Gil Shwartz <gilshwa...@gmail.com> wrote:
> > I think that the type of clone is a property of the clone entity (I am
> > not sure what the standard terminology is, but I am referring to what
> > is represented by <t> element in the .leo file, and by "Hidden Root
> > Node" in a previous comment of Edward).
>
> Again, this is a major disagreement between us. What you are calling
> the "clone entity" is, it seems to me, the same thing as the entire
> Leo outline (the entire .leo file). I don't want any such
> outline-wide data dependencies influencing clones!!!!
I admit that maybe your terminology confused me. I assumed there are
many "Hidden Root Nodes", and root is used in the sense of clone root,
not the entire Leo tree root.
I am not too familiar with Leo's internal, but I do remember a long
time ago, a separation between the tree structure (tree nodes,
including clone nodes) and the collection of nodes content designated
by GNXs. This suggests that for a clone, multiple clone nodes just
point to a single node content (which is what I refer to a clone
entity, and which may include sub-nodes).