Sqlite, leo's future file format?
Ville M. Vainio
Think about it.
Sqlite databases are self-contained in one file, and it seems also the
author suggests sqlite as a possible application file format.
It would also seem to fit leo like a glove. It would get leo much
closer to the goal of lazy loading of nodes, which would radically
speed up startup time and lower the memory footprint.
It would also allow concurrent editing/reading of the same document,
provided that the code does not rely too much on cached data. For
example, you could have external process populate the database with
the results of scanning @shadow nodes while you are already browsing
the outline.
Also, you could have arbitrarily large BLOBs (e.g. attachments,
images...) in the database without them slowing down the read process,
and without any encoding/decoding overhead. Basically, you could also
store the shadow files directly in the database.
Sqlite database would also yield better extensibility (than what is
already provided as uA's) to leo's file format. Plugins could add
their own tables to the db schema without interfering with core leo
(and again, no matter how much crap plugins add, the data would not be
"read" or decoded when loading up the leo file).
Essentially, a leo document is already a database, so this would just
make it more exlipicit ;-)
The current leo document is quite easy to represent, of course - just
have table for tnodes, vnodes and uAs.
--
Ville M. Vainio - vivainio.googlepages.com
blog=360.yahoo.com/villevainio - g[mail | talk]='vivainio'
Edward K. Ream
> Sqlite databases are self-contained in one file, and it seems also the
> author suggests sqlite as a possible application file format.
Are you aware of Leo's support for zodb data stores? Afaik, nobody
has done anything at all with it. Perhaps there is a big untapped
potential in this area.
> It would also seem to fit leo like a glove. It would get leo much
> closer to the goal of lazy loading of nodes, which would radically
> speed up startup time and lower the memory footprint.
This may be a laudable goal. I'm not sure. One problem with lazy
loads is that it may not be possible to know whether a node is a clone
or not.
Also, it's not clear to me whether representing .leo files as a db
will really help. In particular, most of the time loading leoPy.leo
is spent in the code that reads thin derived files: in contrast,
loading the .leo file itself happens almost instantaneously.
> It would also allow concurrent editing/reading of the same document,
> provided that the code does not rely too much on cached data. For
> example, you could have external process populate the database with
> the results of scanning @shadow nodes while you are already browsing
> the outline.
An interesting idea. I don't know how it plays with derived files, though.
> Also, you could have arbitrarily large BLOBs (e.g. attachments,
> images...) in the database without them slowing down the read process,
> and without any encoding/decoding overhead. Basically, you could also
> store the shadow files directly in the database.
Right. This should be possible with zodb.
> Sqlite database would also yield better extensibility (than what is
> already provided as uA's) to leo's file format. Plugins could add
> their own tables to the db schema without interfering with core leo
> (and again, no matter how much crap plugins add, the data would not be
> "read" or decoded when loading up the leo file).
>
> Essentially, a leo document is already a database, so this would just
> make it more exlipicit ;-)
>
> The current leo document is quite easy to represent, of course - just
> have table for tnodes, vnodes and uAs.
In that case, feel free to experiment with a sqlite plugin :-) Or
simply prototype the plugin with @button's.
Edward
Ville M. Vainio
> Are you aware of Leo's support for zodb data stores? Afaik, nobody
> has done anything at all with it. Perhaps there is a big untapped
> potential in this area.
Zodb is a big dependency, while sqlite is bundled with python (like tkinter).
> Also, it's not clear to me whether representing .leo files as a db
> will really help. In particular, most of the time loading leoPy.leo
> is spent in the code that reads thin derived files: in contrast,
> loading the .leo file itself happens almost instantaneously.
Perhaps sqlite would be an ideal "node cache" for thin derived nodes.
It would contain the actual data, and the time stamp of the file it
was derived from. A background process could then be launched to
update the sqlite database with the "real" data, while the user is
already browsing the outline! Most users would probably trade the
startup time to ~ double the disk space use quite willingly.
> In that case, feel free to experiment with a sqlite plugin :-) Or
> simply prototype the plugin with @button's.
This feels like too much like "core stuff" to be implemented as a
plugin. Notably, it would need to hook into the system where leo reads
files and opens nodes (i.e. it's way over my head).
--
Ville M. Vainio
http://tinyurl.com/vainio
Terry Brown
a node hierarchy and stores the nodes as pickled objects in a sqlite3
db. Some nodes have 5 MB .mp3 files as attributes, but this doesn't
seem to be a problem. So I'm impressed with the performance of sqlite,
but I don't think that's enough experience to inform how it would work
for Leo.
I'm thinking of calling my system Dynamic Object Processing
Environment, or Dope for short ;-) The sad truth is that I should have
used Zope, but found it easier to start from scratch using CherryPy /
Genshi / sqlite (and pickle and the rest of python).
http://www.dulcimersinduluth.com/ is made with the system, although the
active site is actually a static mirror of the running 'Dope' site.
It was coded using Leo, of course. It's actually an 1164 line .py file
which wouldn't be manageable if not for Leo - not that such a large
file is great style, but it's very convenient for deployment.
Cheers -Terry
Edward K. Ream
>
> I've just written a through-the-web web authoring system which uses
> a node hierarchy and stores the nodes as pickled objects in a sqlite3
> db. Some nodes have 5 MB .mp3 files as attributes, but this doesn't
> seem to be a problem. So I'm impressed with the performance of sqlite,
> but I don't think that's enough experience to inform how it would work
> for Leo.
>
> I'm thinking of calling my system Dynamic Object Processing
> Environment, or Dope for short ;-)
Great name!
> The sad truth is that I should have
> used Zope, but found it easier to start from scratch using CherryPy /
> Genshi / sqlite (and pickle and the rest of python).
>
> http://www.dulcimersinduluth.com/ is made with the system, although the
> active site is actually a static mirror of the running 'Dope' site.
>
> It was coded using Leo, of course. It's actually an 1164 line .py file
> which wouldn't be manageable if not for Leo - not that such a large
> file is great style, but it's very convenient for deployment.
Cool. I'm becoming more interested in a sqlite back end for Leo. I
would like to see a "dopey" plugin for Leo. I probably won't get
around to it for awhile, but it would be great if somebody else could
do it sooner.
Edward