Leo and fossil merged with Rust

[vitalije]

It has been a very long time since I've got this idea of combining Leo with fossil. For all these years I felt that there was a great potential in this mixture, but I haven't got the time to do anything about it until recently. 

Fossil uses an extremely good algorithm to calculate the difference between two texts. And the deltas produced by this algorithm are very compact and nice to work with. If the input texts are texts, than the delta is also a plain text. I have ported this algorithm from C in which fossil is originally written, to Rust programming language. The ported code is published on crates.io as fossil-delta. Using this library, I wrote a small web server which accepts snapshots sent from Leo periodically at idle time and calculates the delta between the previous one and the current one, and stores those deltas in the sqlite database. The server also serves small web application that allows user to browse history of any recorded Leo outline. Using two scale widgets user can choose any recorded version of the outline shape, and history of the selected node.

Sending snapshots from Leo is done by the plugin which keeps track of the time passed since the last Leo command has been executed. When 5s pass since the last executed command, Leo calculates the snapshot and if it is different than the previous one it sends it in separate thread to the server. The whole process is almost unnoticeable by the user. The server stores the deltas in the database file (one database per Leo outline) which is located in the same folder as the Leo document, and has the '.history' appended to its name. For example: outline test.leo will have it's history stored in the database test.leo.history in the same folder.

Below is the link to the video demonstration (if you have read everything above you can skip to the 3:00). There are some issues with the sound which I couldn't fix, but there are captions in English.

The demo

Vitalije

PS: the outline is about 210Kb, and database with about 625 recorded versions is about 450Kb. Those 625 versions were recorded during the 30 hours time span.

[Terry Brown]

That is very cool.  I hope Kent sees it, he's talked about such things more than once.

Cheers -Terry

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/897a3b7a-554b-4db6-8501-ebafc1746214%40googlegroups.com.

[vitalije]

On Monday, July 29, 2019 at 9:31:49 PM UTC+2, Terry Brown wrote:

That is very cool.  I hope Kent sees it, he's talked about such things more than once.

 

Yes I just tried to find the exact date when this idea appeared on this list. I haven't found my message that I am rather sure I had written about some experimental code - a Leo script to disassemble Leo outline in small files (one per node) + outline as a separate file and execute fossil command in a separate process to commit those files. And the other script used to execute fossil checkout and then reassemble parts to Leo outline. Most probably it was in 2012 or before.

 

Vitalije

[Edward K. Ream]

On Mon, Jul 29, 2019 at 2:11 PM vitalije <vita...@gmail.com> wrote:

It has been a very long time since I've got this idea of combining Leo with fossil. For all these years I felt that there was a great potential in this mixture, but I haven't got the time to do anything about it until recently. 

Many thanks for this excellent work. Clearly, you've demonstrated that potential.

 

Using two scale widgets user can choose any recorded version of the outline shape, and history of the selected node.

You have convinced me that outline and node diffs could be useful.

Sending snapshots from Leo is done by the plugin which keeps track of the time passed since the last Leo command has been executed. When 5s pass since the last executed command, Leo calculates the snapshot and if it is different than the previous one it sends it in separate thread to the server. The whole process is almost unnoticeable by the user. The server stores the deltas in the database file (one database per Leo outline) which is located in the same folder as the Leo document, and has the '.history' appended to its name. For example: outline test.leo will have it's history stored in the database test.leo.history in the same folder.

Looks good to me.

Below is the link to the video demonstration (if you have read everything above you can skip to the 3:00). There are some issues with the sound which I couldn't fix, but there are captions in English.

Excellent. This kind of demo, with minimal/no captions seems to be popular.  For example, see the emacs demos here.  They let the code do the explaining :-)  I am thinking that I should replace of of Leo's ponderous videos with this kind of thing.

Edward

[Kent Tenney]

Indeed.

I haven't used Leo in a while, been playing with docker containers

which configure and serve Jupyter lab on non-gui machines.

This time machine kind of capability is the kind of thing to lure

me back to Leo.

Terry, I remember you mentioning that you're using Docker, do

you have Leo related Dockerfile or docker-compose.yml ?

Thanks,

Kent

To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/CAFPg46QO%2B_nGcyPs-OV1V8aUbr1JZAooF0CT98NK63XhGTUuCQ%40mail.gmail.com.

[Matt Wilkie]

It has been a very long time since I've got this idea of combining Leo with fossil. For all these years I felt that there was a great potential in this mixture, but I haven't got the time to do anything about it until recently.

I've long been enamored with Fossil, though I've never used it in a real way. I just love it's self contained and portable nature. If python is batteries included, then Fossil is power plant included. ;-)

Below is the link to the video demonstration (if you have read everything above you can skip to the 3:00). There are some issues with the sound which I couldn't fix, but there are captions in English.

 

The demo

This is so cool. Being able to flip through all the versions of a node so quickly, to see it grow before my eyes. Wow, that is exciting.

-matt

[Terry Brown]

Hah - convergent evolution, I've been looking at Jupyter lab. in that kind of context recently too.

The rest of this is off-topic, but I encourage people to play with the Jupyter *lab* (not notebook) UI for Leo ideas.

It's basically multiple notebooks in multiple kernels plus consoles and regular text file editing, file browsing, etc.

Here's an amusing trick to get Google to give you a free computer (for 12 hours, but then another one, and so on).

Sign up for

https://colab.research.google.com/ 

now you have a Jupyter *notebook* on a machine somewhere which can mount your Google drive.  But no one who's used Jupyter lab. wants a Jupyter notebook.  No problem, this notebook:

https://medium.com/@swaroopkml96/jupyterlab-and-google-drive-integration-with-google-colab-42a8d64a9b63

lets you run Jupyter lab. in the notebook, then you can access the lab. UI instead.

I guess the machines are for learning machine learning, but there seems to be no restriction on what you do with them, they're Ubuntu 18.04.  I think they only last for 12 hours but if you mount your google drive storage there is permanent.  I also noticed going from free 15 Gb Google drive storage to 100 Gb is quite affordable, $16/*year* maybe?

I think something was broken in the above recipe for me, so I'll post the code blocks I'm using below.

Leo docker wise not really, although I think I recently tested this Leo as a web app. hack and found it still worked:

https://github.com/tbnorth/leo-docker  so that is Leo running in a headless Docker container available via the web.

Cheers -Terry

Block 1

!pip install jupyterlab

Block 2

from google.colab import drive
drive.mount('/content/gdrive', force_remount=True)

Block 3

jupyter_running = !jupyter notebook list | grep 8888
if not jupyter_running:
  !mkdir -p /root/.jupyter
  !echo '{ "NotebookApp": { "password": "sha1:6f..MAKE YOUR OWN..9c" } }' > /root/.jupyter/jupyter_notebook_config.json
  print(open("/root/.jupyter/jupyter_notebook_config.json").read())
  get_ipython().system_raw('jupyter lab &')

!ssh -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -R YOURNAMEHERE:80:localhost:8888 serveo.net 1>/dev/null 

 

(the MAKE YOUR OWN etc. parts are explained in the linked article)

To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/CAAa_k2dUMHu_X%3DwUSM9F0buxBKAgtAsnwc29gKWbZ%3DtZYdj6fw%40mail.gmail.com.

[Chris George]

I tried to open the Leo file and it crashed the tree widget. Clicking on the tree kills Leo.:

bad .leo file: fossil-delta-ref.leo
g.toUnicode: unexpected argument: ParseError('not well-formed (invalid token): line 44, column 86')


Traceback (most recent call last):

  File "/home/chris/leo-editor/leo/core/leoCommands.py", line 2078, in executeAnyCommand
    return command(event)

  File "/home/chris/leo-editor/leo/core/leoGlobals.py", line 245, in commander_command_wrapper
    method(event=event)

  File "/home/chris/leo-editor/leo/commands/commanderFileCommands.py", line 255, in open_outline
    open_completer(c, closeFlag, fileName)

  File "/home/chris/leo-editor/leo/commands/commanderFileCommands.py", line 200, in open_completer
    c2 = g.openWithFileName(fileName, old_c=c)

  File "/home/chris/leo-editor/leo/core/leoGlobals.py", line 3875, in openWithFileName
    return g.app.loadManager.loadLocalFile(fileName, gui, old_c)

  File "/home/chris/leo-editor/leo/core/leoApp.py", line 3158, in loadLocalFile
    previousSettings = lm.getPreviousSettings(fn)

  File "/home/chris/leo-editor/leo/core/leoApp.py", line 2145, in getPreviousSettings
    c = lm.openSettingsFile(fn)

  File "/home/chris/leo-editor/leo/core/leoApp.py", line 2308, in openSettingsFile
    readAtFileNodesFlag=False, silent=True)

  File "/home/chris/leo-editor/leo/core/leoFileCommands.py", line 688, in openLeoFile
    silent=silent,

  File "/home/chris/leo-editor/leo/core/leoFileCommands.py", line 551, in getLeoFile
    v = FastRead(c, self.gnxDict).readFile(fileName)

  File "/home/chris/leo-editor/leo/core/leoFileCommands.py", line 61, in readFile
    v, g_element = self.readWithElementTree(path, s)

TypeError: cannot unpack non-iterable NoneType object

Traceback (most recent call last):
  File "/home/chris/leo-editor/leo/plugins/qt_tree.py", line 813, in onItemClicked
    mods = g.app.gui.qtApp.keyboardModifiers()
AttributeError: 'NullGui' object has no attribute 'qtApp'

Leo 6.0-b2-devel, devel branch, build 2397655ff2
2019-07-26 10:01:19 -0500
Python 3.7.3, PyQt version 5.9.6

linux

A plugin would be great.

Chris

[vitalije]

On Tuesday, July 30, 2019 at 10:24:32 PM UTC+2, Chris George wrote:

I tried to open the Leo file and it crashed the tree widget. Clicking on the tree kills Leo.:

Can you be more specific about what steps did you take to get this error. I can open fossil-delta-ref.leo  and click in the outline. Everything is in order.

From the traceback you have posted I see that NullGui is in effect and that your version of fossil-delta-ref.leo is not well formed xml document. Try checking it out again.

Editing ref outline is not recommended. Instead make your own copy and save it as fossil-delta.leo.

Vitalije

[Chris George]

Much better. Thanks.

A plugin would still be good. :-)

Chris

--

You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/72aed4d8-69bc-454a-8e30-89af8dc60d9f%40googlegroups.com.

[john lunzer]

I think many of us have had a similar vision for development and development environments. I have long dreamed of this kind of omnipresent "history" integrated into every aspect of computing, but it never seemed to show up. Now you've demonstrated the power it holds.

I envisioned this and in addition "saving" as simply tagging one of these revisions. 

Your demo is missing a powerful demonstration of this feature, the "over the shoulder" demonstration. What is required is a "play" button. If you slide the outline slider back in time and then provide a play button which moves the slider forward after a time increment as well as visually jumps to the largest changes you can "look over the shoulder" of a developer as they develop.

Even without a play button, a check "jump" box which if enabled would jump to the largest change when sliding the outline slider would be nice. 

Great work, there are countless possibilities here. 

[vitalije]

I envisioned this and in addition "saving" as simply tagging one of these revisions. 

I am thinking about having Leo send some tag along with the snapshot. For example somewhere in Leo UI or in one special node user is asked to write short description of what task is he/she doing now. Snapshots are sent along with this description and later user can search for the outline as it was when some particular task was worked on.

Your demo is missing a powerful demonstration of this feature, the "over the shoulder" demonstration. What is required is a "play" button. If you slide the outline slider back in time and then provide a play button which moves the slider forward after a time increment as well as visually jumps to the largest changes you can "look over the shoulder" of a developer as they develop.

Currently there is no single timeline of history that could be played forward or backward. Each node has it's own timeline, and outline itself has timeline too. Those timelines can be synchronized by timestamps, but for real looking over developer's shoulder feel, there has to be saved some additional data like: which node was selected, where was the cursor in the body, vertical scrollbar positions, ... It would be straightforward to add those information though.

My original intention was just to provide unlimited, persistent undo/redo history. But more and more ideas come every day. Adding tag support would allow developer to see time spent on some tasks. Analyzing those timings developer can improve future predictions of how long will it take to do this or that. 

 

Currently I am thinking about incorporating my other Rust project (the one for reading xml Leo outlines and external files), in order to allow Leo not only to send snapshots but also to get snapshots of fully opened and loaded outlines along with belonging external files. That would reduce load time considerably. The largest external file in the Leo core is leo/test/activeUnitTests.txt, and on my machine it is loaded in about 6.4ms. This task is suitable for utilizing several CPU cores where Rust shines. I expect that loading an outline with 100 average external files can be done in less than 300ms.

I have a lot of ideas, but also running out of time. :-(

Vitalije

[Israel Hands]

Hi Vialije,

Genius!

IH

[Offray Vladimir Luna Cárdenas]

Hi,

On 30/07/19 10:26 a. m., Matt Wilkie wrote:

It has been a very long time since I've got this idea of combining Leo with fossil. For all these years I felt that there was a great potential in this mixture, but I haven't got the time to do anything about it until recently.

I've long been enamored with Fossil, though I've never used it in a real way. I just love it's self contained and portable nature. If python is batteries included, then Fossil is power plant included. ;-)

Fossil is really cool. Unfortunately Git has kind of monopolized developers imagination and work flow, which makes even cooler to see this demo of Fossil and Leo working together.

We have been using Fossil for all documentation related projects in Grafoscopio, except for source code itself which is managed in SmalltalkHub. Hopefully at some point we will merge code and doc repos to use Fossil for both.

Below is the link to the video demonstration (if you have read everything above you can skip to the 3:00). There are some issues with the sound which I couldn't fix, but there are captions in English.

 

The demo

This is so cool. Being able to flip through all the versions of a node so quickly, to see it grow before my eyes. Wow, that is exciting.

So true. I was thinking? Can be this used as a way to sync real time multi-author writing in a outline, ala CoreObject?[1]

[1] http://coreobject.org/

Cheers,

Offray