Eurekas! The sea of nodes, half clones & more
Edward K. Ream
standing questions became apparent. The solutions are simple and
unforgettable.
The train of thought that lead to the Ahas has something to do with a
"chance" remark made by somebody recently, I don't know who or in what
thread, about clones without children being related to different views
of the outline. I may have mangled the sentiment completely. If you
know what I'm talking about, please point me at the remark.
===== The problems
For at least 5 years, and probably longer, people have wanted
differing views of nodes:
1. The sea of nodes. The great LeoUser (the progenitor of Leo's
minibuffer) had a dream of nodes being "free form", that is, not
constrained by outline structure. It's a tantalizing vision...
2. Half clones. A frequently requested feature is some kind of clone-
like operation that would *not* clone a node's children.
The obvious solution to both of these problems would seem to involve
eliminating structure links, but...
===== The Aha
Suppose we ignored Leo's structure links. We would immediately need
*another* set of links in order to organize the sea of nodes, or to
organize the half clones. So the Aha is:
The sea of nodes, and half clones, require *additional* links
The idea that we can *eliminate* structure information leads us in
exactly the wrong direction.
You can think of each kind of structure as different colored links.
Each color induces a different view of nodes. We could have a sea-of-
nodes view, a per-person view, a general-graph view, etc. Each view
might have a different *rendering engine*.
Each view has its own rules! The rules for each view (and thus each
rendering engine and each set of links) are completely independent!
Leo's traditional outline view prohibits nodes from being ancestors of
themselves. But that rule would not exist in either a general graph
view or a sea-of-nodes view. Back links might be shown in such views
without the slightest problem.
===== Implementation
At present, all link information resides in vnodes, specifically the
parents and children arrays. The rendering information is the
information not written to the file. Everything else is the "real"
content of the vnode. This content, and *only* this content, is
independent of view and rendering engine.
It is easy to implement multiple links! We simply add a single dict
to each vnode, say v.viewDict. Keys are view keys, values are parents
and children arrays for that view.
Switching (global) views simply involves switching the global view
key. Leo's fundamental code in leoNodes.py will simply access
v.viewDict.get(view).parents and v.viewDict.get(view).children instead
of v.parents and v.children.
Python's properties make this "indirection" transparent to most of the
code in leoNodes.py! We can implement multiple links in less than a
page of code.
===== Conclusions and questions
It's too soon to tell what practical impact these Ahas will have.
They might form the basis of Leo 5.0. They may suggest other new
directions.
Surely, though, this is a fundamental breakthrough.
Questions remains about half clones. Could we show different colored
links within the same outline? The obvious way to do this would be to
show "my-view" links in @view my-view trees. Would this be useful, or
just confusing? It's too early to tell.
Things are clearer, I think, for scripts. Scripts will have no
trouble dealing with links of a different color. For example, Leo's
rst3 command could have an option to specify which *kinds* of
structure to use in organizing documentation.
This is the general answer to Kent's desire to organize documentation
freely. Each way of organizing documentation induces a different
color of links. The content stays the same, but the organization can
vary freely. Actually, scripts could even vary the *content* by
munging the content based on the view-selector. This would be easy to
do using tags in the text, a topic of another post...
Finally, the picture that emerges is a *general graph of views*. Think
of the general graph as a set of overlays. Each overlays shows the
links of a particular color. Each overlay is a *distinct* view, with
its own rules for links.
Yes, there is a general graph in the background: the union of all
links of all colors. But this "joined" view is much less interesting
than the individual views! Each individual view has its *own*
identity and its *own* structure and is *own* rules. For example, a
DAG doesn't become a general graph merely because it is part of the a
larger union.
In short, this Aha creates a new kind of data structure: a merger of
*independent* components.
Edward
P.S. I had some other Ahas yesterday, but they will have to wait for
another post...
EKR
HansBKK
For those without too much time on their hand, skip to the bottom line, just scan quickly if it seems at all of interest, or ignore completely.
Let's say I've got the complete works of Carl Jung in plain text format, and I want to create a self-contained hypertext "study guide" with several "axes of access" for organizing in-depth textual analysis:
- the full text of the original works themselves
- broken down into chunks
- hypertext links for footnotes, the original front-matter ToC and end-matter indices
- so far nothing new, just as a well-curated ebook would present itself
- expanded indices, cross-referencing the various works originally published separately
- more modern/relevant terminology
- separate topic-domain-specific terminology indexing and cross-referencing "axes", e.g.
- clinical psychology and psychoanalysis (his "home domain" in traditional academia)
- social sciences
- transpersonal psychology, dreams and the collective unconscious
- humanities, literature and the arts
- traditional myth, religion and spirituality, including 12-Step recovery, Zen, the I Ching et al
- the "supernatural" - mysticism, alchemy, parapsychology, UFOs, the occult etc
Then, starting to depart from the "pure/original" full text, adding content from third parties:
- summary texts with brief/superficial commentary tied in to the full-text and the above indices
- in-depth "textbooks" and "study guides" which combine quotes from/pointers to the original fulltext
- selecting, re-ordering and combining from the various works freely as the third-party author sees fit
- with extensive third-party analysis, clearly distinguished from the text written by Jung himself
- also for these works, summary texts with brief/superficial commentary
- and of course all cross-referenced by the above indices
Think of what Leo now offers in the navigation outline as **the** "canonical" ToC outline for the original work. Forgetting clones for now, a single dimensioned hierarchy, ordered from top to bottom, click on "The Practice of Psychotherapy", "Chapter 1" etc. and you follow the original sequence, with the text "chunked" into nodes granular enough to be meaningfully "tagged" by indexing topic terms.
Now make that say the first "Tab" called "Fulltext ToC". Have a second tab titled "Summary ToC" - in this case it will look identical, but traverse the summary texts rather than the fulltext (with internal cross-references inline with the content as well). Then a Tab for each work about or commenting on Jung's work, starting with say "Jennings' Passages Beyond the Gate", or the 2007 course syllabus for Harvard Prof John Mack's course on UFO abduction.
UI issues aside, my point is that each of these "alternative tables of contents" can each stand on their own and present their own subset of the content, re-sequenced and interleaved between different works from different authors.
In my implementation of such hypertext "undatabases" using TiddlyWiki, it's ability to give "meta-tags" to topic tags and navigation "meta-nodes" was critical, so that the traditional psychoanalysis domain's use of the term "superconscious" is kept separate from Blavatsky and Gurdjieff's, and the Summary ToC clickable for Chapter 1 is separate from the Fulltext's.
I've kind of run out of steam at this point, as I'm sure has anyone wading through all this along with me - thanks for your patience, and I hope my Aha seems at least tangentially relevant to Ed's, although I'm sure his had a radically different focus.
Bottom line is - empower the user to create multiple "parallel universes" of meta-trees to navigate the same super-set of data.
tfer
of mine to add the ability of interposing test nodes, (that end up in
a test file), with code that ends up in a code file.
Rather than expanding the @file directive to take two file arguments,
have two separate @file statements, each with a subtree that "feeds"
them, yet present them intertwined via a third "view" tree.
Tom
Edward K. Ream
> I was thinking that this might have an application to a recent request
> of mine to add the ability of interposing test nodes, (that end up in
> a test file), with code that ends up in a code file.
Yes. I guess my subconscious turned your idea into mine. Thanks for
priming the pump. I'm happy to give you full credit for the idea.
What matters is the final result.
> Rather than expanding the @file directive to take two file arguments,
> have two separate @file statements, each with a subtree that "feeds"
> them, yet present them intertwined via a third "view" tree.
This sounds like the reverse of what I was thinking of, which is a
"master" tree that people use, and one or more "helper" trees that
actually carry the data.
Edward
Edward K. Ream
> This got me quite excited
Imo, if people aren't excited by this Aha, then they don't get it ;-)
> Let's say I've got the complete works of Carl Jung in plain text format, and
> I want to create a self-contained hypertext "study guide" with several "axes
> of access" for organizing in-depth textual analysis:
[snip]
> Think of what Leo now offers in the navigation outline as **the**
> "canonical" ToC outline for the original work.
[snip]
> UI issues aside, my point is that each of these "alternative tables of
> contents" can each stand on their own and present their own subset of the
> content, re-sequenced and interleaved between different works from different authors.
Exactly. I've often said that there is no one "right" view of data,
but unless we can define *fully co-equal* outline views, the
"canonical" outline view is "more right" than any other outline view.
At present in Leo, each node has only one "right" set of children!
Your example shows why this is not optimal.
> Bottom line is - empower the user to create multiple "parallel universes" of
> meta-trees to navigate the same super-set of data.
Yes. "Parallel universes" refer to different structure (sets of links)
on the same "data", that is, the v.b, v.h and v.ua parts of vnodes.
Edward
Kent Tenney
> On Fri, Mar 2, 2012 at 9:27 AM, HansBKK <han...@gmail.com> wrote:
>> This got me quite excited
>
> Imo, if people aren't excited by this Aha, then they don't get it ;-)
I don't get it. :-]
I have trouble following the explanation: my fault.
I did wonder if there is overlap with past discussions of generalising
Leo structure via 'nodes' and 'edges' where an edge is a first class
participant, instead of a pointer. That is the model in the graph db world.
I have a mental picture of that, but not yet of the current aha.
Thanks,
Kent
>
>> Let's say I've got the complete works of Carl Jung in plain text format, and
>> I want to create a self-contained hypertext "study guide" with several "axes
>> of access" for organizing in-depth textual analysis:
>
> [snip]
>
>> Think of what Leo now offers in the navigation outline as **the**
>> "canonical" ToC outline for the original work.
>
> [snip]
>
>> UI issues aside, my point is that each of these "alternative tables of
>> contents" can each stand on their own and present their own subset of the
>> content, re-sequenced and interleaved between different works from different authors.
>
> Exactly. I've often said that there is no one "right" view of data,
> but unless we can define *fully co-equal* outline views, the
> "canonical" outline view is "more right" than any other outline view.
>
> At present in Leo, each node has only one "right" set of children!
> Your example shows why this is not optimal.
>
>> Bottom line is - empower the user to create multiple "parallel universes" of
>> meta-trees to navigate the same super-set of data.
>
> Yes. "Parallel universes" refer to different structure (sets of links)
> on the same "data", that is, the v.b, v.h and v.ua parts of vnodes.
>
> Edward
>
> --
> You received this message because you are subscribed to the Google Groups "leo-editor" group.
> To post to this group, send email to leo-e...@googlegroups.com.
> To unsubscribe from this group, send email to leo-editor+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en.
>
Ville M. Vainio
I don't get it either :).
I guess the summary is that general graphs (or multple general graphs, composed of same nodes by different set of edges) could be interesting for leo 5.
The most difficult part here is what it would look like in the ui
Edward K. Ream
> On Sat, Mar 3, 2012 at 5:00 AM, Edward K. Ream
>
> I don't get it. :-]
There are several aspects of the Eureka:
1. An answer to a long-standing question, namely, how can we make
sense of the concept of a "half clone", that is, "a clone without
children".
The answer is clear: we are really talking about clones with *other* children.
2. As a spur to new thinking.
Hans has taken the "first next" step. He points out that treating a
*particular* set of children as the "right" children is as limiting as
treating a particular view of data as the "right" view. It's much
more interesting to treat all (or multiple) structures as equally
valid.
For example, I am pretty sure that the present limitations of the rst3
command could be removed if that command were not "tied" to the
"official" outline structure.
You have mentioned the desire to create different kinds of
documentation from the "same" data. We can see now that the Eureka
makes that possible in a new way. The data is simply content, but now
the structure can vary freely.
Edward
Kent Tenney
> On Sat, Mar 3, 2012 at 7:07 AM, Kent Tenney <kte...@gmail.com> wrote:
>> On Sat, Mar 3, 2012 at 5:00 AM, Edward K. Ream
>>> Imo, if people aren't excited by this Aha, then they don't get it ;-)
>>
>> I don't get it. :-]
>
> There are several aspects of the Eureka:
>
> 1. An answer to a long-standing question, namely, how can we make
> sense of the concept of a "half clone", that is, "a clone without
> children".
I've not been interested in clones, so not one of my questions.
>
> The answer is clear: we are really talking about clones with *other* children.
>
> 2. As a spur to new thinking.
>
> Hans has taken the "first next" step. He points out that treating a
> *particular* set of children as the "right" children is as limiting as
> treating a particular view of data as the "right" view. It's much
> more interesting to treat all (or multiple) structures as equally
> valid.
>
> For example, I am pretty sure that the present limitations of the rst3
> command could be removed if that command were not "tied" to the
> "official" outline structure.
>
> You have mentioned the desire to create different kinds of
> documentation from the "same" data.
That has been in the spirit of the db backend discussion, and the
node/edge concept. IE Leo being a data manager, but the data
existing in a standard form. The nodes would exist in the same
space as bookmarks, pdf file contents, history and logging data ...
Leo would be a tool to view and mashup the data. But the core
concept is that the storage and relation mechanism is well known.
Hence: db backend
Often 'db backend' discussions are around collaborative editing
of Leo files, I'm not interested in multiple 'users' changing my Leo files,
I'm interested in multiple 'applications' using my Leo data.
Thanks,
Kent
We can see now that the Eureka
> makes that possible in a new way. The data is simply content, but now
> the structure can vary freely.
>
> Edward
>
Edward K. Ream
> I'm not interested in multiple 'users' changing my Leo files,
> I'm interested in multiple 'applications' using my Leo data.
The client doesn't matter. The Eureka makes possible multiple
*organizations* of nodes. That should be useful for people,
applications, whatever.
EKR
Kent Tenney
OK, I guess I'll have to _see_ it to get it.
Thanks,
Kent
>
> EKR
Seth Johnson
I will be very interested in how this shapes up and how you start
looking at collaboration, distribution and versioning. (BTW, I would
designate those "organizations" by unique keys for their use type,
link type and particular use, as well as designating the states they
reside in.)
I actually did download and take a look at the file load procedures,
with an eye for somehow hooking it to a db backend -- I expect you've
changed that stuff rather markedly recently.
Seth
Edward K. Ream
> I will be very interested in how this shapes up and how you start
> looking at collaboration, distribution and versioning. (BTW, I would
> designate those "organizations" by unique keys for their use type,
> link type and particular use, as well as designating the states they
> reside in.)
Right. These would be the keys to v.viewDict.
Edward
Edward K. Ream
On Mar 3, 12:00 pm, Kent Tenney <kten...@gmail.com> wrote:
> OK, I guess I'll have to _see_ it to get it.
Suppose we implement chapters (or even hoists) in the new scheme.
Different chapters = different links.
So you could, in a chapter, insert or delete nodes, without affecting
structure in other chapters, including the "main" chapter.
Also, there could be two ways to create a chapter: create-chapter-
from-node and create-chapter-from-node-ignoring-children.
This has important uses: you could create a chapter oriented towards
documentation, and use the rst3 command in such a chapter to focus on
only the nodes (and their descendants *in that chapter*.)
Edward
P.S. BTW, c.hiddenRootVnode must also have a v.viewDict, so that
different views (chapters, hoists, whatever) can each have their own
notion of what top-level nodes are.
EKR
Ville M. Vainio
v.viewDict is an implementation detail. An alternative is to introduce 'edges' as separate first class construct, and then bundle edge type in that construct (color, half-clone, backlink, normal tree edge...).
This scheme (of edges overall) was discussed earlier, when talking about database storage for Leo (2 sql tables - Nodes and Edges)
Seth Johnson
representing the use type, "node" type and particular use (also
columns for state) within which nodes are being organized -- that is,
it can store edges with a specification of the context in which they
are being used (Ed's distinct "organizations").
Instead of a separate table for nodes, I store attributes as a general
entity type, and they too have context such that their values are
relevant to particular "organizations" of "nodes" or not. "Nodes" are
virtual in my scheme.
Seth
Edward K. Ream
> The table in which Ville stores edges can be extended with columns
> representing the use type, "node" type and particular use (also
> columns for state) within which nodes are being organized -- that is,
> it can store edges with a specification of the context in which they
> are being used (Ed's distinct "organizations").
>
> Instead of a separate table for nodes, I store attributes as a general
> entity type, and they too have context such that their values are
> relevant to particular "organizations" of "nodes" or not. "Nodes" are
> virtual in my scheme.
@Ville, @Seth:
Thanks for reminding me of your (plural) earlier work. I am prone to
forget such things. It's a fact of life at my age, alas. Or maybe
it's not so much related to age as it is that I'm not really
comfortable with db organizations, and so I have less to "hold on to",
so to speak.
I trust that when the time comes to do something about the Eureka you
will keep reminding me of all the relevant issues.
Edward