The Zettelkasten System Revisited After A year

[tbp1...@gmail.com]

Over a year ago we had a long thread on the "Zettelkasten" system:

https://groups.google.com/g/leo-editor/c/TqiNdBfnEig

To briefly reprise, a zettelkasten is a system for capturing and cross-linking of knowledge.  The term can be translated as "card case", and the original - or at least the one that became somewhat well known - comprises a set of index cards, a group of file drawers to store them, and a way of numbering the cards so that they cross-refer to other cards.  Each card (or "zettel") is expected to carry a short, narrowly focused, thought or essay-like paragraph about some topic.

The system was written about by Niklas Luhmann, who apparently was wildly prolific as a writer and known for his deep knowledge of a number of fields of study.  He credited his zettelkasen with much of his success, referring to it as a second brain or a very smart companion.

See, e.g., "Communicating with Slip Boxes by Niklas Luhmann":

http://luhmann.surge.sh/communicating-with-slip-boxes

Luhmann had developed a way of numbering and cross-linking his cards that seems to have worked extraordinarily well for him.

The thread on Google Groups was mostly concerned with whether Leo could be used for such a system, and how that would work.  There are several software systems that claim to be designed for this purpose, but I didn't find any of them satisfactory, and it would generally have been hard to impossible to get your data back out in a usable form if you wanted to switch to another way of working.

Eventually I distilled the points that had been discussed into a set of user requirements, and I realized that with a few simple scripts and a standard way of working, a zettelkasten system in Leo would be extremely easy and effective.

One of the key points is to have a way to extract your data easily, which means working with a text format or something that can be converted to a text format easily. As you will see, that is quite simple.

After all this, I got diverted and didn't do much with it until recently, a year and more later.  I wanted to capture some genealogy/family history data and organize it somehow.  So I have been working with the card-case system, and I've learned some practical things.  This has led me to slightly change the scripts, and to settle on a way to format and use the cards.  I think it's working really well, and I'd like to pass this experience along.

I have attached the requirements I came up with along with some links to more material about this kind of system. In my next post I will talk about the card format and the scripts that make it all work.

[tbp1...@gmail.com]

The card-case system needs each card to have a unique and permanent id number.  This id is used for finding cross-referenced cards.  Leo creates a unique id for each node, so I decided to use this identifier, known as a *gnx*.  An example gnx is *tom.20210521095407.1*.  This id can be extracted by a bit of code in a script.

Make in last year's thread, I had suggested that our "zettel" cards could be marked up with a bit of Restructured Text.  Identifiers and key words, like the id, could be written in RsT's "Field List" format.  This would make them easy to parse, and they would display very readably if rendered as RsT.

This format sets off the key word between colons, like this:

:id: tom.20210521095407.1

:seealso: tom.20210321135407.1

:eventdate: 2021-05-21

It also turned out to be very helpful to add the name of target cards to the field list lines, and the name of the node as a header for the card.

With these additions, the body of a card looks like this:

William Collins

----------------

:id: tom.20210520181722.1

:created: 2021-05-20 18:17

:type: person

:tag: Collins Immigration

:marriage: tom.20210605195824.1 `Collins-King`_

:born: ~1615

:died: 1687

Born in Maidstone, Kent County, England.  Died at Isle of Wight County, VA.

Immigration Into America

^^^^^^^^^^^^^^^^^^^^^^^^^

William Collins emigrated from Kent County, England to King and Queen County, VA in 1635 aboard the ship "Plain Jane".

The two underlines tell RsT to render the previous line as different title levels.  They are optional but look good in a RsT rendered view of a larger part of a tree of cards.

The backticks followed by an underscore are an optional markup that lets an RsT processor turn them into clickable links to the target.  They do not add much visual clutter, and allow for navigation in a rendered browser view.

A rendered view of a card or a subtree can be had using the Viewrendered3 plugin.  This plugin can also export the rendered HTML to your browser.

The attached Leo outline contains the little scripts I use, together with the key bindings I use for them.  You would put them into the @settings tree of your card-case outline, or into MyLeoSettings.leo if you want them available in all outlines.

The two most used commands are zettel-convert-card (F8) and zettel-go-to-target (F9).  When you select a node and press F8, the current node will be converted into a zettel card.  This means that the node headline is inserted with its underline as the title, and the id and date fields are also inserted.

When you select or put the cursor on a field list line that contains a gnx as the second field, the selection jumps to the target card. This makes navigation extremely fast and easy.

Almost as important is zettel-goto-with-backlink (F6).  This jumps to the target node like F9 but also inserts a backlink to the sending card if the target does not already have one.  If the target has a backlink but the backlink line does not contain a label, the script will insert the sending card's title as the label.

Since backlinks are a key feature of any zettelkasten system, this script makes generating them accurate and nearly painless.

The new backlink will be inserted as the first line of the target node, with the generic name :link:.  You probably will want to move the line somewhere else, and to replace the string `link` with something more appropriate.  But even if you don't, navigation will still function normally.

The command zettel-copy-gnx (F7) lets you easily copy a card's gnx to the clipboard so you can paste it into another card as a target.  Then F6 will take you to that target and insert a backlink.  Another F6 will add the target's label to the link line of the sending card.  Now both cards have been labeled and equipped with mutual backlinks.

Pleas note that my original F8 script from last year had a bug that sometimes inserted an incorrect gnx.  Also, it created entire new car, but I find my new way of converting an existing node to be easier and quicker.

To summarize this navigation system, the field list lines like

:link: gnx label

let you navigate to any crosslinked card within a leo outline, and the optional RsT label/backtick/underscore markup lets you navigate in a rendered view displayed in a web browser.

You can also use any of Leo's search capabilities to search through the cards.  I find the Nav pane to be especially good for this.

In my next post, I will cover ways to get your data out of Leo if you need to.  It can be very simple.

[tbp1...@gmail.com]

If you should ever want to export your data from Leo, Leo can do that for you.  Leo's ordinary file saving methods will save your entire tree of cards.  If you follow the format suggested by my previous post, the resulting file will be easy to parse if you want to adapt it to some other system.  For example, a simple AWK script could split the file into individual cards, and other scripts could turn the gnx ids into navigation targets

The two easiest ways to do this are with @clean or @rst files.  For an @clean file, just add the directive @others to the bottom of the body of the first node (the "@clean" one).  Your tree could look like this:

@clean ~/data/zettel_output.txt

    your tree here...

When you save your outline, the @clean tree will be created and saved too.

If you opt for using an @rst tree, you can just select a node in the tree and invoke the minibuffer command rst3.  The @others directive is not necessary.  Your tree's top node could look like this:

@clean ~/data/zettel_rst_output.rst

After saving this way, all your data will be contained in a largish text file.  You can process it with an RsT processor, just scroll through it by hand and edit anything you want, write a script to extract all the individual cards, convert it to XML or JSON and use XML or JSON tools on it, index it with a whole-text indexing system, or do any number of other things.

My next post will cover rendering cards or the tree of cards with Viewrendered3, and working with a set of individual cards using the new Freewin plugin.

[tbp1...@gmail.com]

Oops!  I wrote:

On Thursday, June 17, 2021 at 11:25:04 PM UTC-4 tbp1...@gmail.com wrote:

If you opt for using an @rst tree, you can just select a node in the tree and invoke the minibuffer command rst3.  The @others directive is not necessary.  Your tree's top node could look like this:

@clean ~/data/zettel_rst_output.rst

I meant to write 

@rst ~/data/zettel_rst_output.rst

[tbp1...@gmail.com]

If you were working with a physical zettelkasten system with physical index cards, you would pull some number of related cards and spread them out on a working surface.  You might start a new card, devise an id number for it, and start to write its content.  As you worked, you might find you want to change some of the existing cards in the light of what you have learned or thought about since you wrote them.

The new Freewin plugin was developed to emulate this process.  Each Freewin window shows a single node, the one that was selected when the window was opened.  This window contains a basic text editor.  When you type into it, the body of the underlying host node picks up the same changes.  When you type into the host node, the Freewin window picks up those changes.

You can open and spread out as many of these Freewin windows as you have room for, just as you can spread out a number of index cards on your desk.  You might want to make the main Leo window smaller so the cards will be clear of the new windows.  This is an ideal situation for a second monitor.  You could move all the Freewin views to the second monitor, and work with Leo in the main monitor.

In addition, a Freewin window can render its host node as ReStructured text.  This can be easier to read, in case you don't need to make any editing changes.  The rendered view can't be edited but it is still live and will show any changes made to its underlying host node.

You can't follow any inter-card navigation links in a Freewin view, because it is locked to its one node.  However, embedded URL links will work although this is unlikely to be useful.

The attached image file freewin_cards_1.png illustrates the use of Freewin windows this way.  The card in the upper right is for a man called William Collins.  Its editing view is shown. The card shows that he was in a marriage titled "Collins-King".  I opened a window for that card, which is placed below the first one.

The marriage card shows a son "John Collins".  There are other children shown too, but we don't have cards for them - we can see this because they don't have gnx backlinks.  A Freewin window for this John Collins has been opened, and switched to the RsT rendered view. We see that he was in a marriage of his own, but we don't have a window open for that.

The RsT card format I recommend for these "zettel" cards was in part developed with viewing in the Viewrendered3 plugin in mind.  A VR3 view cannot be edited, but unlike Freewin windows, VR3 can be set to show a part - or all, if you export to a browser - of a tree of cards.  The RsT navigation links will work if their target is included in the part of the tree that you rendered.

An example is shown in the attached file vr3_view_with_links.png. VR3 is shown rendering the entire collection of family history cards.  Of course the entire tree cannot be shown in one static screen shot, but the view is scrollable.  The main thing to notice is the RsT navigation links - the labels in blue, like "Collins-Weeks".  RsT and your browser know nothing about the gnx links, but the Restructured Text processor knows how to make regular HTML links out of those labels.  As long as the target is included in the rendered tree, a link can be clicked to navigate to that target.

I'm going to wrap up this series here.  I apologize for its length, but I wanted to introduce this system.  It has turned out that Leo is almost ideally suited for hosting a modern, convenient Zettelkasten system.  If you compare its capabilities against the user requirements I put together last year, we have covered all of them pretty well.

[Edward K. Ream]

On Thu, Jun 17, 2021 at 10:10 PM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

> ...I realized that with a few simple scripts and a standard way of working, a zettelkasten system in Leo would be extremely easy and effective.

> One of the key points is to have a way to extract your data easily, which means working with a text format or something that can be converted to a text format easily. As you will see, that is quite simple.

Thanks for this clear summary.

Edward

[Edward K. Ream]

On Thu, Jun 17, 2021 at 10:18 PM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

...the body of a card looks like this:

William Collins

----------------

:id: tom.20210520181722.1

:created: 2021-05-20 18:17

:type: person

:tag: Collins Immigration

:marriage: tom.20210605195824.1 `Collins-King`_

:born: ~1615

:died: 1687

Born in Maidstone, Kent County, England.  Died at Isle of Wight County, VA.

Immigration Into America

^^^^^^^^^^^^^^^^^^^^^^^^^

William Collins emigrated from Kent County, England to King and Queen County, VA in 1635 aboard the ship "Plain Jane".

Thanks for this example.

The attached Leo outline contains the little scripts I use, together with the key bindings I use for them.  You would put them into the @settings tree of your card-case outline, or into MyLeoSettings.leo if you want them available in all outlines.

The two most used commands are zettel-convert-card (F8) and zettel-go-to-target (F9)...

Almost as important is zettel-goto-with-backlink (F6).

...

...backlinks are a key feature of any zettelkasten system [This] script makes generating them accurate and nearly painless.

To summarize this navigation system, the field list lines like

:link: gnx label

let you navigate to any crosslinked card within a leo outline, and the optional RsT label/backtick/underscore markup lets you navigate in a rendered view displayed in a web browser.

You can also use any of Leo's search capabilities to search through the cards.  I find the Nav pane to be especially good for this.

Imo, these posts, lightly edited, would make an excellent addition to Leo's Advanced Topics Chapter.

I too like to use posts here as prewriting for documentation.

Edward

[Edward K. Ream]

On Thu, Jun 17, 2021 at 11:37 PM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

If you were working with a physical zettelkasten system with physical index cards, you would pull some number of related cards and spread them out on a working surface.  You might start a new card, devise an id number for it, and start to write its content.  As you worked, you might find you want to change some of the existing cards in the light of what you have learned or thought about since you wrote them.

The new Freewin plugin was developed to emulate this process. 

A great summary.

Edward

[Edward K. Ream]

On Thursday, June 17, 2021 at 11:37:03 PM UTC-5 tbp1...@gmail.com wrote:

If you were working with a physical zettelkasten system with physical index cards, you would pull some number of related cards and spread them out on a working surface.  You might start a new card, devise an id number for it, and start to write its content.  As you worked, you might find you want to change some of the existing cards in the light of what you have learned or thought about since you wrote them.

The new Freewin plugin was developed to emulate this process. 

The screen shots are excellent.  Many thanks for the entire thread.

If you like, you could send me an outline containing your finished documentation. I'll then fold it into LeoDocs.leo.  This will almost certainly be simpler than trying to merge two versions of LeoDocs.leo with git.

Edward

[tbp1...@gmail.com]

I will do that.  I developed these posts in a Leo outline, so everything is there except the attachments, which I could zip together with the outline.  Or maybe I can import them into an outline.

[tbp1...@gmail.com]

I need to add an important caution in case you may want to copy a card-case node or tree to some other location, such as a different outline.  *MAKE SURE* that you paste them *AS A CLONE*.  When you use an ordinary paste, the new nodes are given new gnx values - those crucial node ids.  Since the navigation system depends on the gnxs being correct, an ordinary paste will ruin the navigation and backlinks.

It would not be hard to write a script to correct the ids, but then if you moved the subtree back into the original, the links to the original would never work.

You might want to duplicate a tree when sending it to someone else, possibly as an example.

In Leo, it is usually suggested that you never clone nodes into other outlines.  This is a rare case where it is *ESSENTIAL* to do so.

[tbp1...@gmail.com]

For use in the documentation, I think I should remove the first-person references and some of the history - @Edward, would you agree?  I could also apply formatting and image insertion using RsT - I assume the docs will be produced with Sphinx, is that right?

On Friday, June 18, 2021 at 9:48:08 AM UTC-4 tbp1...@gmail.com wrote:

[Edward K. Ream]

On Sat, Jun 19, 2021 at 11:40 AM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

For use in the documentation, I think I should remove the first-person references and some of the history - @Edward, would you agree?

Sure. It's easy to do the pre-writing in a more informal style, and it's easy to revise once one has a chatty starting point.

I could also apply formatting and image insertion using RsT - I assume the docs will be produced with Sphinx, is that right?

Correct.

Edward