Idea - "SQL Leo"

[Dave]

Hi folks,

I've thought on and off about the merits of a Leo implementation that
stores its nodes and outlines, not in .leo files, but in an SQL
database (MySQL, PostgreSQL, SQLite etc).

The idea is to allow for nodes to be cloned across different outlines,
not just cloned at different points in the one outline.

This would also allow for multi-user access, that is, if there is a
protocol for "locking" and/or "sharing" nodes.

Another option, which doesn't require database implementation, would
be to allow for a node type which allows nodes in one .leo file to ref
nodes in other .leo files. For example, file1.leo has a node /foo,
with a child node /foo/bar which actually references a node /baz/bar
in file file2.leo

The basic idea is to allow one outline to reference nodes in another
outline (or reference the entire outline).

Thoughts?

Cheers
David

[HansBKK]

On Friday, December 16, 2011 6:49:11 AM UTC+7, Dave wrote:

Thoughts?

I think it would have the same sort of dangerous complexity as the concurrent development of full human cloning and mass time-travel tourism.

But I suppose both are inevitable anyway. . .

8-)

[Terry Brown]

On Thu, 15 Dec 2011 15:49:11 -0800 (PST)
Dave <da...@conscious.co.nz> wrote:

> I've thought on and off about the merits of a Leo implementation that
> stores its nodes and outlines, not in .leo files, but in an SQL
> database (MySQL, PostgreSQL, SQLite etc).

There was some work done to get Leo to use ZopeDB as a backend, some
time back. I think it was working at the proof on concept level.

It's an interesting line of thought.

Cheers -Terry

[mdb]

I vote for allowing the saving and reading of nodes to a database
table, but not attempting to replace simple flat file storage.

Instead, a node attribute could be set to define the secondary storage
and/or source. When saving a leo file you should have the option to
break the outline nodes from the database (i.e., you only used the
database to read-in nodes) or change the stored database node if you
made changes, or you could decide to create a new version. And if the
attribute fields are set up well you can have the option to reconnect
or compare to a stored node.

SQLite would probably be the simplest implementation on the user
side. A full featured version would allow for any database system
using python db interfaces, and include 'cloud' computing.

With little doubt, a full-fledged database supported system that uses
the system's locking and sharing tools would be safer and richer than
allowing nodes in one .leo file to reference nodes in other .leo
files.

[Kent Tenney]

There's also been a suggestion of putting nodes in Fossil, which seems
to offer the benefits of a db engine, plus versioning: all in one file.

http://www.fossil-scm.org/index.html/doc/trunk/www/index.wiki

> --
> You received this message because you are subscribed to the Google Groups "leo-editor" group.
> To post to this group, send email to leo-e...@googlegroups.com.
> To unsubscribe from this group, send email to leo-editor+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en.
>

[viva...@gmail.com]

There is also working sqlite representation for leo tree, google for leoqviewer.

I have failed to find time to publish this so far, but it's fully working (if you have n9 ;).

[Edward K. Ream]

On Thu, Dec 15, 2011 at 6:49 PM, Dave <da...@conscious.co.nz> wrote:

> The idea is to allow for nodes to be cloned across different outlines,
> not just cloned at different points in the one outline.

In the past I have been cool to the idea, if not dismissive. But it
does seem that basing the idea on a DB promises the solution to the
real (or imagined) problems.

In particular, gnx's are supposed are unique, so references are well
defined. Presumably the DB would be the platform that solves the
multiple-update problem.

Edward

[Edward K. Ream]

On Sat, Dec 17, 2011 at 8:44 AM, <viva...@gmail.com> wrote:
> There is also working sqlite representation for leo tree, google for
> leoqviewer.
>
> I have failed to find time to publish this so far, but it's fully working
> (if you have n9 ;).

It would be great if you could push what you have, working or not.

Edward

[Ville M. Vainio]

Googling for leoqviewer gets you to github repo. I may push this to contrib branch if needed.

[Ville M. Vainio]

Db also allows you to load partial data, and load more if needed. This allows multi gigabyte outlines with limited ram (like phones)

[Ville M. Vainio]

Also,

see Projects\leoq in Leo "contrib" branch. It has ctrl+b script
(create_leoq) that you can use to dump current outline to an sqlite
database, to be used by my "leoqviewer" N9 application for
browsing/editing the content.

[Ville M. Vainio]

... and, the sql for the database schema I'm using is here, for review:

https://gist.github.com/1493084

I have a separate table for BLOBS (basically body text content, or
some binary data like picture attachments I'm planning to expand this
format with), NODES, and EDGES.

[Edward K. Ream]

On Sat, Dec 17, 2011 at 2:11 PM, Ville M. Vainio <viva...@gmail.com> wrote:
> Googling for leoqviewer gets you to github repo. I may push this to contrib
> branch if needed.

Thanks for this. Rebecca is a great listener. I was chatting away
this morning at breakfast, saying the following:

1. The only way to scale up Leo so that it could, for example, handle
data sets like the human genome project would be to store Leo's nodes
in a DB.

In such a situation, Leo files become "windows" or "views" on the DB.
Rebecca correctly points out that an individual Leo file can contain
multiple views of data, and this quality does not change if the nodes
are stored in the DB.

2. Basing nodes on a DB has the potential to allow cross-file clones:
the DB (potentially) can solve all the worrisome problems with
multiple access.

This has the potential to be a game changer as far as the *idea* of
clones, nodes and views is concerned. I don't know how this will work
out, but just about every Leonista, including me, has wanted more
capable (cross-file) clones.

3. The interaction between bzr and a DB is intriguing. Perhaps we
will indeed use fossil, which I am just reminded is based on SQlite.

In short, I believe that this year will see a grand re-visioning of
what Leo is and can be. This new vision has arisen more from the
usual suspects than from me--a development that pleases me greatly.

Ville, your code is an important step forward. I'll commit it in some
form soon, and begin serious study of it as a way of priming the
subconscious pump. Great things will come of this, I am sure.

Edward

[HansBKK]

I agree that this vision is a inspiring!

Probably obvious to y'all, but I would also suggest building a plugin architecture for data storage first, so it would be easier to switch from SQLite to Oracle or whatever down the road.

Such a model should also allow Leo to treat a given outline/clone organizational model - a given "view" - as a unit, so you could load/unload different views as needed, simplifying navigation and helping to keep intact the overall "shape" offered by that view, without the current requirement of having multiple trees for different purposes being visible at the same time.

And store the views themselves in the database so they're easily shared - definitely would need version-control if these weren't just read-only for the other users. . .

However (back to basics), **please** don't add "database programmer/administrator" to the requirements of using Leo for basic functionality - keep storage of plain text in filesystems as the base/core data model, and the database layer as extended, optional flexibility.

This process could also I imagine enable more flexibility in thinking even about the plain text in filesystems storage model - from a recent conversation with Terry:

> > Fundamentally Leo stores data in XML.  It's always going to be possible to get data back out of a .leo file without Leo, although it might require tools / skills somewhat different from plain text.  But storing a tree with clones in plain text would make the plain text so sentinel heavy it might as well be XML...

> Yes, but the beauty of the @shadow structure is, "who cares"? The only people that are going to look at the sentinels are programmers who care about Leo. Therefore sentinels in @shadow files can carry all the important data right out in the filesystem.

------------
Sidenote as a cautionary example of project evolution, skip if you're busy:

One of my favorite platforms for ad-hoc text-information management, and a brilliant tool for presenting structured views complex data is TiddlyWiki, whose basic value proposition (IMO) is allowing the functionality of a full interactive website in a single HTML+javascript file. A few years ago most of the more tech-savvy members of that community, including the core developer, started focussing a lot of time and attention on various different "external data access" implementations for all the same reasons as this discussion, and IMO this has led to the core code stagnating, to the point where one must use old versions of Firefox or IE to use the tool. BTW apparently this will change as the project's founder is developing a new version, apparently starting from scratch (uh oh?). All this is IMHO of course, and I still just love TiddlyWiki. . .

[Matt Wilkie]

> However (back to basics), **please** don't add "database
> programmer/administrator" to the requirements of using Leo for basic
> functionality - keep storage of plain text in filesystems as the base/core
> data model, and the database layer as extended, optional flexibility.

I'm excited by the idea of cross-file clones and the prospect of
handling multi-gigabyte data files. I'm cautious about the prospect
of moving key intelligence into an opaque storage format. Yes we have
number of prominent unlikely-to-ever-die open source DB platforms,
however we still need special tools -- and skills -- to inspect their
contents. Where is the Vim and Emacs of the DB?

The humble text editor is the only tool I use today that I also used
when I first started to learn computers. Correspondingly, text files
are the only data which could be ported and used across all those many
machines in the intervening twenty years, essentially unaltered. And
I'm just a young fart. This modest quality of text is easy to
overlook, part of the ground underfoot, but also of great
significance, as is the ground.

I'm pretty sure everyone here is cognizant of the inherent value of
plain text, probably deeper than myself, but I figured I'd say this
piece anyway so it's expressly part of the discussion.

(And finally, my apologies to Leo for putting it in the same class as Edlin ;-)

-matt

[HansBKK]

On Monday, December 19, 2011 3:29:39 PM UTC+7, Matt Wilkie wrote:

(And finally, my apologies to Leo for putting it in the same class as Edlin ;-)

Edlin, pshaw - you young whippersnappers don't know how lucky you had it, when I was young we had to make do with nothing but copy con - and then there was the time my F6 key broke 8-)

[Edward K. Ream]

On Dec 17, 2:11 pm, "Ville M. Vainio" <vivai...@gmail.com> wrote:
> Googling for leoqviewer gets you to github repo. I may push this to contrib
> branch if needed.

I just pushed it to contrib.

Edward