A new file format for Leo documents (sqlite3 database file)

[vitalije]

In the other thread I have announced my experiment but it seems that I haven't explain it very well. Let me try another way.

The trouble I bumped into few days ago with changing shortcut settings was just a primer that set me in motion. My initial experiment with collecting all settings in sqlite database and speed test were intended just to make an argument. Although it started with the settings it was not only about settings. It was about new file format for Leo.  I knew from earlier discussion in this forum that Edward is/was not willing to approve such ideas. This time however I decided to give it a try just for myself and it happened that this time Edward actually encouraged me to go on with the experiment. But I suspect that my idea is still not very well explained/understood. 

There used to be somewhere on this forum some links to the talk that Dr.Richard Hipp, the author of SQLite, gave explaining what he wish that user of SQLite library have in mind when using it. It is not meant to compete with multi-user, server based database systems. It is meant to replace a humble open (file ) function.

He gave some examples of missed opportunities  in some large software projects like OpenOffice that use xml as file format for their documents. If they used SQLite instead they would be able to open document almost instantaneously, implementing undo/redo that can survive restart of application would be a trivial task, and lot of other advantages he has shown.  There are also several talks about the advantages of SQLite's source code control tool *fossil* over git. Not only that history (undo/redo) of one file is possible to be kept inside document file, but also the history of all project files, with all history details like check-in comments, wiki pages, branches, diffs,... all that stuff can be kept inside one file. User can copy that file on memory stick, send it via e-mail or google drive, it can be displayed online with just one executable file and it can be used for self hosting project site on the web. Something like github site packed in one executable file and one repository file. It handles user accounts, permissions, registrations, bug reports, to-dos,... almost everything that is needed for software-project hosting site. 

Inspired by those talks I have made a script that puts entire Leo tree in fossil repository and the other script that can recreate Leo tree from fossil repository. I remember it was very fast operation even if it used files on disk for the transfer to fossil. About as fast as is checking and parsing external files in Leo. Another script used to give a timeline list of all versions of Leo tree that are recorded in the repository. That list was printed in Log pane and it was possible for user to choose one of listed versions and Leo would recreate that particular version of tree.  

At that point I have abandoned the experiment because I haven't any clue how to create a GUI for all those features. It wasn't so useful with just a bunch of scripts and Log pane as the only GUI elements. I have also realized that there is a problem with the gnx-s. Two different versions of Leo tree had to be shown in different commanders because all changed nodes can not have the same gnx and different bodies or headlines. I still have a strong feeling that this idea of keeping the whole history of the tree in one file has some potential, but it needs suitable GUI to be useful at all.  

As I had no idea how suitable GUI could look like, nor how to solve the issue with conflicts between two different historical versions of the same node, I start to think about using sqlite database as file format for Leo documents. Knowing it would be hard to sell the idea, I let the time to pass and I did nothing about it. However that idea has never left my mind since. Recent events have lead me to final decision to do something about it. 

I have dived once again in Leo start-up code and this time I had more success. I have finally made the first version of my prototype. With my changes Leo can accept as input argument database file. Now I can open any ordinary xml Leo document, click a button and database version of Leo document is generated in the same folder where Leo file is located and with the extension '.db' instead of '.leo'.  Later Leo can open that db file and have the same tree it would have if it parsed xml file. 

leoSettings and myLeoSettings as well as inside document settings are respected with the current implementation. 

The only thing currently missing is restoring the position and dimension of the application window on screen. 

It will be trivial to implement though.

I haven't run tests because I am not sure how to run them. Edward, Terry please give me the proper sequence of steps how to run all tests and check if I broke something or not by my changes to the code. If everything goes well I would publish git branch with my code so that you can review it. I suppose my code style is not in accordance with the standard that is used in Leo, so I expect that I will have to prettify my code but I didn't bother to do it yet.  

I have mention on the other thread that I gathered some web browser client scripting skills (also fair amount of the server stuff skills). If Leo document were sqlite database I would be able to try something with the old idea of showing whole history of Leo tree but this time in web browser. I imagine that perhaps in web browser it would be easier to overcome problems with showing more historical versions of the same node. 

Also someone on this forum announced development of browser viewer of Leo documents. I expect that it could benefit also from sqlite file format and maybe some server stuff. 

I am sorry for very long post and for all mistakes I have made writing it. Hopefully, this time I have better explained the reasons and ideas that I had in mind. If I left something unclear please help me by asking questions and I would try to answer as concise as possible. 

Vitalije

[Offray Vladimir Luna Cárdenas]

Congrats Vitalije!

Sometimes prototyping is the best way to argument in favor of an idea and is nice to see your prototyping going well.

In my case, Grafoscopio [1] is a prototype of the ideas I would like to see in Leo: liveness & interactivity, a simpler storage format (XML is too cumbersome) and fossil integration. Grafoscopio documents are stored in STON [2] (a Smalltalk extension of JSON) and versioned in a fossil repository[3], so an outline includes all its history and the ones of their related artefacts (data, images, other outlines) in a single self-contained, self hosted fossil file. Browsing that history, merging and diffing becomes trivial. Using Pharo have let me to explore this ideas with agility and in a mostly solo endeavor as coder (but with the important help of several communities).

[1] http://mutabit.com/grafoscopio/index.en.html
[2] https://github.com/svenvc/ston/blob/master/ston-paper.md
[3] http://mutabit.com/repos.fossil/grafoscopio/

I'm more interested in literate computing, interactive documentation and reproducible research that in programming/scripting, so, while there is certain overlap between Leo and Grafoscopio, they're looking for different things, but is really nice to see common potent ideas being developed, as the ones that is Vijalije sharing with us now.

Keep the good work and let us know how it goes,

Offray

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To post to this group, send email to leo-e...@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

[Terry Brown]

On Thu, 29 Jun 2017 13:25:54 -0700 (PDT)
vitalije <vita...@gmail.com> wrote:

> In the other thread I have announced my experiment but it seems that
> I haven't explain it very well. Let me try another way.

Actually what you describe below is sort of what I thought you'd ended
up doing. Sounds very interesting.

To run tests I

python launchLeo.py leo/test/unitTest.leo

then press `Alt-4` (not Alt-F4). That file seems to be saved with the
right node selected.

As mentioned in another thread there are some tests that may fail on
system's other than Edward's.

Your file format supports p.v.u too?

Cheers -Terry

[vitalije]

setting leoID from os.getenv('USER'): 'vitalije'

reading settings in /home/vitalije/programi/leo/trunk-git/leo/config/leoSettings.leo

reading settings in /home/vitalije/programi/leo/trunk-git/leo/config/myLeoSettings.leo

reading settings in /home/vitalije/programi/leo/trunk-git/leo/core/LeoPyRef.db

** isPython3: False

Leo 5.5, build 20170624181042, Sat Jun 24 18:10:42 CDT 2017

Git repo info: branch = sqlite-format, commit = 1e6da9e97999

Python 2.7.6, PyQt version 4.8.6

linux2

wrote recent file: /home/vitalije/.leo/.leoRecentFiles.txt

vitalije@phenom:~/programi/leo/trunk-git$ python launchLeo.py leo/test/unitTest.leo 

setting leoID from os.getenv('USER'): 'vitalije'

reading settings in /home/vitalije/programi/leo/trunk-git/leo/config/leoSettings.leo

reading settings in /home/vitalije/programi/leo/trunk-git/leo/config/myLeoSettings.leo

reading settings in /home/vitalije/programi/leo/trunk-git/leo/test/unitTest.leo

** isPython3: False

Leo 5.5, build 20170624181042, Sat Jun 24 18:10:42 CDT 2017

Git repo info: branch = sqlite-format, commit = 1e6da9e97999

Python 2.7.6, PyQt version 4.8.6

linux2

.saved: save-new-test.py

....setting leoID from os.getenv('USER'): 'vitalije'

....................test of at.printError: La Peña

..............s...........................

End of leoAtFile tests

..............sssss........................................

End of leoColor tests

.............................

End of leoCommands tests

........

End of leoConfig tests

......................................s..

.....ss.....................................s..s.................................................................................................sss....sss...

End of typing tests

.

End of leoEditCommands tests.

............

End of leoFileCommands tests.

.....

End of leoFind tests.

.............

End of leoFrame tests.

........................................................................

End of leoGlobals tests.

.sss

End of leoGui tests.

...............................................................................................................................

End of leoImport tests.

.......

End of leoKeys tests.

........................................

End of leoNodes tests.

....................ss..s..ss.....EEEEEE...........................................................

End of leoUndo tests.

.............................................................................................................................................................

End of plugins unit tests

.@test print redraw count: 654

..all unit tests done

..

======================================================================

ERROR: runTest (leo.core.leoTest.GeneralTestCase)

@test rst3Test @no-head

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/vitalije/programi/leo/trunk-git/leo/core/leoTest.py", line 211, in runTest

    builtins.execfile(scriptFile, d)

  File "/home/vitalije/.leo/scriptFile.py", line 6, in <module>

    rst3Test(c,p)

  File "<string>", line 14, in __init__

  File "<string>", line 54, in run

  File "<string>", line 43, in clean

AttributeError: 'NoneType' object has no attribute 'replace'

======================================================================

ERROR: runTest (leo.core.leoTest.GeneralTestCase)

@test rst3Test default

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/vitalije/programi/leo/trunk-git/leo/core/leoTest.py", line 211, in runTest

    builtins.execfile(scriptFile, d)

  File "/home/vitalije/.leo/scriptFile.py", line 6, in <module>

    rst3Test(c,p)

  File "<string>", line 14, in __init__

  File "<string>", line 54, in run

  File "<string>", line 43, in clean

AttributeError: 'NoneType' object has no attribute 'replace'

======================================================================

ERROR: runTest (leo.core.leoTest.GeneralTestCase)

@test rst3Test doc_only_mode (set in headline)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/vitalije/programi/leo/trunk-git/leo/core/leoTest.py", line 211, in runTest

    builtins.execfile(scriptFile, d)

  File "/home/vitalije/.leo/scriptFile.py", line 7, in <module>

    rst3Test(c,p)

  File "<string>", line 14, in __init__

  File "<string>", line 54, in run

  File "<string>", line 43, in clean

AttributeError: 'NoneType' object has no attribute 'replace'

======================================================================

ERROR: runTest (leo.core.leoTest.GeneralTestCase)

@test rst3Test doc_only_mode (set in options doc part)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/vitalije/programi/leo/trunk-git/leo/core/leoTest.py", line 211, in runTest

    builtins.execfile(scriptFile, d)

  File "/home/vitalije/.leo/scriptFile.py", line 7, in <module>

    rst3Test(c,p)

  File "<string>", line 14, in __init__

  File "<string>", line 54, in run

  File "<string>", line 43, in clean

AttributeError: 'NoneType' object has no attribute 'replace'

======================================================================

ERROR: runTest (leo.core.leoTest.GeneralTestCase)

@test rst3Test show_leo_directives=False

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/vitalije/programi/leo/trunk-git/leo/core/leoTest.py", line 211, in runTest

    builtins.execfile(scriptFile, d)

  File "/home/vitalije/.leo/scriptFile.py", line 7, in <module>

    rst3Test(c,p)

  File "<string>", line 14, in __init__

  File "<string>", line 54, in run

  File "<string>", line 43, in clean

AttributeError: 'NoneType' object has no attribute 'replace'

======================================================================

ERROR: runTest (leo.core.leoTest.GeneralTestCase)

@test rst3Test unicode characters

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/vitalije/programi/leo/trunk-git/leo/core/leoTest.py", line 211, in runTest

    builtins.execfile(scriptFile, d)

  File "/home/vitalije/.leo/scriptFile.py", line 10, in <module>

    rst3Test(c,p)

  File "<string>", line 14, in __init__

  File "<string>", line 54, in run

  File "<string>", line 43, in clean

AttributeError: 'NoneType' object has no attribute 'replace'

----------------------------------------------------------------------

Ran 904 tests in 94.022s

FAILED (errors=6, skipped=25)

These are the results of running unit tests. I don't know if those six failing tests are supposed to fail on my machine or they signalize that I broke something. @Edward please look at the results.

Your file format supports p.v.u too?

Cheers -Terry

Yes it does. I actually stored in database all ivars of VNode class defined above the line that says that following attributes will never be written in file. 

v.gnx, v.h, v.b, v.children, v.parents, v.iconVal, v.statusBits and v.u

are all stored in table vnodes. 

Children and parents are represented as a space joined list of gnx-s of children/parent nodes.

v.u is stored as pickle.dumps(v.u). I am not sure if it is enough compatible with the present scheme of storing v.u. I saw that there is some kind of filtering applied in normal Leo write and read code. I don't recall that I used ua recently and I don't posses any test Leo documents to check if it behaves the same way the old code does. 

But I suppose that Leo tree contains the same v.u dictionary it used to have at the moment of storing in database, providing that pickle.dumps(v.u) succeed. 

@Offray Vladimir Luna Cárdenas  thank you for support. I remember that you and I have shared very similar ideas in the past on this forum regarding use of sqlite and fossil in combination with Leo.

Vitalije

[Terry Brown]

On Thu, 29 Jun 2017 14:49:43 -0700 (PDT)
vitalije <vita...@gmail.com> wrote:

> Ran 904 tests in 94.022s
>
> FAILED (errors=6, skipped=25)

I had six errors testing master without modifications a couple of days
ago, so sounds like everything passed. Well done.

Cheers -Terry

[Edward K. Ream]

On Thu, Jun 29, 2017 at 3:25 PM, vitalije <vita...@gmail.com> wrote:

In the other thread I have announced my experiment but it seems that I haven't explain it very well. Let me try another way.

​One of my "hidden" purposes in leading the Leo project is to encourage people to push back when I misunderstand or appear to reject a proposal.  I do this by changing my mind a lot :-)​ ​

 

​[my experiment]​

​ ​

was about new file format for Leo.  I knew from earlier discussion in this forum that Edward is/was not willing to approve such ideas.

​True, I am hesitant to approve any sweeping new idea, but I am not particularly fond of Leo's xml format.  Furthermore, Leo already supports .leo.zip formats, and there have been proposals (not rejected) for using json or yaml for Leo's outlines.
​

 

​[sqlite]​

 is meant to replace a humble open (file) function.

​Well, this certainly ​changes my point of view.

If

​[large software projects] used SQLite instead they would be able to open document almost instantaneously, implementing undo/redo that can survive restart of application would be a trivial task, and lot of other advantages he has shown.

​This gets my attention.
​

 

There are also several talks about the advantages of SQLite's source code control tool *fossil* over git. Not only that history (undo/redo) of one file is possible to be kept inside document file, but also the history of all project files, with all history details like check-in comments, wiki pages, branches, diffs,... all that stuff can be kept inside one file.

​And just when I was getting comfortable with git branches.​

​..​

Inspired by those talks I have made a script that puts entire Leo tree in fossil repository and the other script that can recreate Leo tree from fossil repository. I remember it was very fast operation even if it used files on disk for the transfer to fossil. About as fast as is checking and parsing external files in Leo. Another script used to give a timeline list of all versions of Leo tree that are recorded in the repository. That list was printed in Log pane and it was possible for user to choose one of listed versions and Leo would recreate that particular version of tree.  

At that point I have abandoned the experiment because I haven't any clue how to create a GUI for all those features. It wasn't so useful with just a bunch of scripts and Log pane as the only GUI elements. I have also realized that there is a problem with the gnx-s. Two different versions of Leo tree had to be shown in different commanders because all changed nodes can not have the same gnx and different bodies or headlines. I still have a strong feeling that this idea of keeping the whole history of the tree in one file has some potential, but it needs suitable GUI to be useful at all.

​Yes.  It has to be a powerful as gitk.​

 

As I had no idea how suitable GUI could look like, nor how to solve the issue with conflicts between two different historical versions of the same node, I start to think about using sqlite database as file format for Leo documents. Knowing it would be hard to sell the idea, I let the time to pass and I did nothing about it. However that idea has never left my mind since. Recent events have lead me to final decision to do something about it. 

I have dived once again in Leo start-up code and this time I had more success. I have finally made the first version of my prototype. With my changes Leo can accept as input argument database file. Now I can open any ordinary xml Leo document, click a button and database version of Leo document is generated in the same folder where Leo file is located and with the extension '.db' instead of '.leo'.  Later Leo can open that db file and have the same tree it would have if it parsed xml file. 

leoSettings and myLeoSettings as well as inside document settings are respected with the current implementation.

​Excellent. I was pretty sure that settings were not the real issue.​

 

The only thing currently missing is restoring the position and dimension of the application window on screen. 

It will be trivial to implement though.

​Heh. Last night I realized that that was something that would have had to be handled as a special by the reload-leo command.​

 

I haven't run tests because I am not sure how to run them. Edward, Terry please give me the proper sequence of steps how to run all tests and check if I broke something or not by my changes to the code.

​It's easy.  Load unitTest.leo, select the "Active Unit Tests" top-level node, then do Alt-4 (​run-selected-unit-tests-locally)

Run the tests before making your changes.  Per #512: unit tests are failing for Terry some tests may fail on your machine.  Make a note of those failures.  You can safely ignore those failures after making your changes.

If everything goes well I would publish git branch with my code so that you can review it.

​Excellent. I will be very interested in the what you have done.
​

 

I suppose my code style is not in accordance with the standard that is used in Leo, so I expect that I will have to prettify my code but I didn't bother to do it yet.

​You can experiment with the beautify commands.  The @nobeautify directive will preserve formatting in special cases, such as code containing tables.

But formatting is not that important. pep 8 is a guide, not a straitjacket.​

 

I have mention on the other thread that I gathered some web browser client scripting skills (also fair amount of the server stuff skills). If Leo document were sqlite database I would be able to try something with the old idea of showing whole history of Leo tree but this time in web browser. I imagine that perhaps in web browser it would be easier to overcome problems with showing more historical versions of the same node. 

​.leo.db files would coexist alongside .leo​

 

​and .leo.zip and possibly .leo.json files.
​

Also someone on this forum announced development of browser viewer of Leo documents. I expect that it could benefit also from sqlite file format and maybe some server stuff. 

​That was Joe Orr.​

 

Hopefully, this time I have better explained the reasons and ideas that I had in mind.

 

​Indeed you have.

If I left something unclear please help me by asking questions and I would try to answer as concise as possible. 

​Your intentions seem quite clear.  Furthermore, replacing git is a completely separate matter from replacing .leo files, so we can don't have to change our git work flows just yet ;-)

Great work. What I like most about our (all the Leo devs) collaboration​

 

​is that after everyone has pushed back against initial ideas we end up with something better than any one person would have created on their own.

Edward

[Edward K. Ream]

On Thu, Jun 29, 2017 at 4:49 PM, vitalije <vita...@gmail.com> wrote:

​>

​

FAILED (errors=6, skipped=25)

These are the results of running unit tests. I don't know if those six failing tests are supposed to fail on my machine or they signalize that I broke something. @Edward please look at the results.

​

Those failures can be ignored.  Today I'll attempt to discover why they only work for me. Clearly, it would be best if all tests pass for everyone when run in master.

Edward

[vitalije]

I don't know how to publish my branch. When I try to publish it in leo-editor/leo-editor.git I get:

remote: Permission to leo-editor/leo-editor.git denied to vitalije

Several months ago I have forked leo-editor and made a pull request from there. When I look at that forked repository github reports that it is 8 commits ahead and 735 behind master. I have no idea how to synchronize them. And I don't know what would be the effect of those 8 commits on creating pull request.

I tried to fork it again, but github gives me the same fork I have already made. When I tried to delete that fork, I was seriously warned so I didn't dare to delete it.

What should I do? My local branch is updated to the latest commit of leo-editor/leo-editor.git.

Vitalije

[vitalije]

Ok. I found my way around. I have pulled from my fork into working folder, merged it and then pushed branch to my fork of leo-editor. After that my sqlite-format branch was in sync with the master and I was able to create a pull request that consists only of my recent work. I had to manually copy one file that contained some changes that I have made several months ago and were out of sync with the current master.

I forgot to push script that exports Leo tree in database. Without it new code can't be tested. That reminded me that I haven't done anything about saving to db files.

I am going to do it ASAP and then to push it again.

Vitalije

[Edward K. Ream]

On Fri, Jun 30, 2017 at 9:27 AM, vitalije <vita...@gmail.com> wrote:

I don't know how to publish my branch. When I try to publish it in leo-editor/leo-editor.git I get:

remote: Permission to leo-editor/leo-editor.git denied to vitalije

​I'll attempt to give you permission immediately.  Pushing a branch is preferable to creating a pull request.

Having said that, your PR seems reasonable, but it will be more convenient for everyone to be able to pull down your branch.​

Several months ago I have forked leo-editor and made a pull request from there. When I look at that forked repository github reports that it is 8 commits ahead and 735 behind master. I have no idea how to synchronize them.

​It's easy.  From your branch, do git merge master.

Edward

[Edward K. Ream]

On Fri, Jun 30, 2017 at 10:14 AM, Edward K. Ream <edre...@gmail.com> wrote:

​> ​

I'll attempt to give you permission immediately.

​Just invited you to join leo-editor members, as part of the core team.  That should do the trick.

Edward