Leo as a Database

[tbp1...@gmail.com]

If you look at it right, Leo can be thought of - and used - as a database.  I don't really mean just the tree-and-body organization and data.  I mean a database in a more geneal and organized sense.

A year or two ago, we had in another thread a presentation of how @Edward's brother Speed uses Leo "as a database" in his business.  That was so complicated and so woven into his business practices that I never fully grasped how it works.

And yes, I know that a Leo outline can be written as a SqlLite database file.  I don't mean that, either.  That's structured so that the outline can be reliably extracted from it, but it's not in a form that's suitable for being a more general data store.

In a Leo outline we have basically only a few kinds of data elements:

1. Nodes with their identifiers (gnxes), headlines, and bodies;

2. The arrangement of these nodes into their current trees. 

3.  Possibly we could consider external files as additional data elements.

The node arrangement is ephemeral - most of us keep changing the structure to some degree.  The identifiers are unique to their specific node, much like primary keys.

The rest of the data is essentially in the form of blobs, to use a database term.  It's not differentiated.  To search out particular bits of information would take a natural language processor, and probably would not be very successful anyway.  To be able to do more, we need to be able to differentiate bits of data we think we'll be interested in.

Leo as it exists now does have an ability to recognize certain special kinds of strings: a CTRL-click on one of them performs useful actions:

- on a URL, opens that address in the system browser;

- on a UNL, navigates to that location in a Leo outline;

- coming soon, on a gnx navigates to that gnx;

- on a method invocation, navigates to the node where it is defined;

- also coming soon, on a Python filename, navigates to the @file or @clean node for that file.

These abilities are fantastically useful, depending on what kind of work you do with Leo.

I have devised a simple way to provide Leo nodes with what are essentially key-value pairs, which are the foundation of most data stores.  It requires no programming - though programming can help you find them and move around the data bits.  It is fairly readable and easy to type.  This idea come from two projects I've been working on (on and off) for the last year or so.

1. Each key-value pair take a single line (which could be wordwrapped on the display);

2. Keys are marked using the "Field List" syntax from ReStructured text.  For example (all on one line):

:ref: https://nyuscholars.nyu.edu/en/publications/atreegrep-approximate-searching-in-unordered-trees-2

Here the key is "ref".  

Using this syntax has some advantages:  it's readable, intuitively clear, easy to type, and easy for code to parses out from the body text.  In addition, it renders very nicely in RsT.  I've attached an image of this "card" and its rendered version as seen in a Freewin detached window.

Another card image I've attached is from a family history data set.  It has more keys:

:id: tom.20210606085913.1
:created: 2021-06-06 08:59
:type: person
:born: ~1571
:died: 1635

:marriage: tom.20210522091147.1  Collins-Unknown [1]
:parents: tom.20210606131436.1 `Weeks-Poole`_

You can see that you can make up any key you like.  You can also see how this card could be translated to a relational database.  However, relational databases are not well suited to capturing hierarchical information (though it can be done).  The "marriage" and "parents" lines include the gnx of their respective nodes.  They can be navigated to by pressing CTRL-F9 (an added script) or soon by CTRL-clicking.  So it is very easy to move around the data

In this case, the "id" of the card is its gnx, which is a permanent part of Leo's node structure.  It's appears on the card as a convenience for the viewer.  There is a script that makes it easy to insert a backlink in the target node. 

When the keys are chosen with some consistency, scripts can be written to help with searching and navigation.

With this kind of structure, a Leo outline becomes a flexible data store,  easy to extend, to adapt to changing needs, and to reorganize.

[Christophe Vermeulen]

I like the idea, and I was probably even to some extent thinking of that sort of usage when considering "leo as a PIM". But you see, this approach is likely asking for trouble in many ways. Just a few examples :

- aren't you reinventing the wheel to a large extent ? Using your :key: syntax is Yet Another Way To Code Tags, but is not so different of a json approach or XML, vCard, ... The buzzwords and bad words here are "standard" or "registration", "support", "maintenance", etc.

- leo's way of using XML cannot benefit of this if I understand correctly, as it won't code your keys as XML tags <parents id="tom.xxx."> etc. but <v> and plaintext.

- the creation date is redondant with the ID. That's a feature of leo ! You could have :modified: though and :lastread:, and others

- the last part of your example is supposed to contain notes. Why wouldn't place of birth be a tagged content.

- when this becomes a large address book, managing your scripts to search in it will likely become a nightmare

Don't get me wrong, there is definitely something here. At the same time, I keep getting the feeling that I've seem that already so many times, when I looked at Anki cards, and the incredible discussion of formats for instant messaging back in ... 1998. Ward Cunningham invented the Wiki starting from Hypercard, so why not start from leo to invent ... the next big thing ?

I agree that a SQLite format for leo internally might be a good idea. But apparently Edward won't, and he probably has good reasons. Maybe for leo 8 ?

[tbp1...@gmail.com]

Thanks for the comments.  I don't disagree with most of them, but of course, "reasons".  I had some user requirements in mind that these choices meet well. The beauty of this point of view - "an outline can be your database" is that there can be many ways of getting there.  These examples are only one way, but they are turning out to be very handy.

As for creation date being encoded in the gnx on one of the examples, perfectly true but not very human readable.  For that example, I thought it would be important for the user to have the least cognitive load possible, and the least chance of making a mistake when interpreting the time.  The id itself is included so that the user can copy it and paste it into another node for navigation and linking purposes (can also be done with a script, which I have bound to CTRL-F7, but having the gnx made manifest may be useful).  Also, with the gnx included in the body, an export to text format (see below) will preserve the gnxes and therefore the links.  Keeping them implicit or keeping data in, say, UAs (Unknown or User Attributes) would not export to text easily or automatically.  On last year's discussions of "zettelkasten" systems in Leo, it became clear (at least to some of us) that a capability for export to plain text would be an important consideration.

Some of my user requirements:

1. Simplicity - as much like typing simple notes as possible.  Simple hot-keyed scripts can be allowed.

2. Easily readable.

3. Very extensible.

4. Minimal syntax.

5. Easy to export to other formats, but especially to a text-only format, in case one wants or needs to leave the Leo environment.

6. Maximally simple navigation to other target nodes.

7. Make good use of Leo's tree-rearrangement capabilities without needing to change any of the "cards" or navigation links.

8. Preferably have a practical way to get a easy-to-read rendered view of a card or subtree.

9. Works with clones.

For requirement 5, if you take the top node of your tree of data and make it an @clean node with the only line @others, then presto! your whole tree will be saved in text form.  It would be easy enough to write a program to take the gnx values and recreate all the links, then transform to some other format (json, sql load file, whatever).  And that file would be human-readable, searchable (in a text editor) and immediately usable.

You can't have a simpler export than that!  Well, you do lose the organizer nodes and indentation levels.  There's an easy solution to that.

And if you want, say, an export to some XML format - and I'm an old XML/XSLT hand, I've used them for some pretty cool things - it would be fairly easy to write an export script to XML.  In fact, if you wanted to use XQuery as a search language, you might want to export to XML anyway, if only in-memory.

IOW, I am trying for user simplicity, readability, making good use of Leo's features, maximal ability to save your data out to text if needed so your (potentially years of) work can be saved and ported to another system (no lockin to Leo), and simple  navigation possibilities with little if any programming.

So, to pick up on some of your mentions,  or on Lotus Notes (which you didn't mention), they were handicapped by not having Leo (!), and also they weren't trying to meet my set of user requirements.  I didn't mention any builder requirements, but basically they revolve around the potential for writing Leo scripts and commands to improve searching and visualization.  Those kinds of scripts are still in early stages, and could probably evolve forever.

[rengel]

What if there were such a tool that fullfills all of your requirements?

Have a look at https://tiddlywiki.com/

[tbp1...@gmail.com]

I have used t-w several times over the years.  I admire its accomplishments.  Somehow I have always ended up moving on.  It may be that I found it hard to grasp an overall structure.  It would also be a bear to write one's own software for it, and I'm not sure how your data could be saved out to generic text and still retain the internal links, if that is important.  But yes, if I didn't have Leo I might turn again to t-w.

[rengel]

On Friday, April 1, 2022 at 1:28:15 PM UTC+2 tbp1 wrote:

I have used t-w several times over the years.  I admire its accomplishments.  Somehow I have always ended up moving on.  It may be that I found it hard to grasp an overall structure.  It would also be a bear to write one's own software for it, and I'm not sure how your data could be saved out to generic text and still retain the internal links, if that is important.  But yes, if I didn't have Leo I might turn again to t-w.

That's easy. Here is a compelte exported recod (tiddler) as a pure text file. Firts, fields/attributes as key-value pairs, then a blank line, then any contents of the text field. Internal links in double brackest [[..]]:

""

author: [[Dave Thomas]]
class: Book
created: 20220110171706686
creator: rengel
modified: 20220401151854239
modifier: rengel
publisher: [[The Pragmatic Programmers]]
subtitle: Functional |> Concurrent |> Pragmatic |> Fun
tags: Book
title: Programming Elixir >= 1.6
year: 2018

A book by [[Dave Thomas]] introducing the programming language [[Elixir]].

""

 

[Christophe Vermeulen]

Maybe it doesn't do what I want (I still am quite puzzled with leo, and don't understand most of it, just discovered i t recently and played a bit messing things up), but in the new 6.6, the "Save As" choices seems different : instead of zipped or not, now I see ... JSON/XML/SQLite.. It looks to me a huge change, not exactly advertised as would be expected IMHO.