rust and Leo and other editors

[vitalije]

David Szent-Györgyi wrote in another thread:

When was the last work done on building a file system that exposes as files the data and metadata available via Leo's API? Years ago, I read of the development of FUSE, which allows the development of file system code that runs in user space on Linux and macOS; I see reference to similar toolkits for Windows. 

Leo supports headless operation with a null GUI. Would it be simpler for Emacs and other software to manipulate the Leo file through a Leo File System? 

I find your comments very interesting but I think they deserve to be on the new topic thread. So, I started this one :-).

A while ago I suggested the same idea about integrating Leo's unique features in other editors. Almost every programmer's text editor supports some kind of widget for browsing file system. If Leo creates virtual file system recognizable by the operating system and other running applications then every other editor would gain access to Leo's features for free with no added code.

Right now I am very interested in neovim editor and it's architecture. It seems that it would be possible to use neovim as a front end of Leo file through writing a neovim plugin. Recently I have found glrnvim a combination of open-gl accelerated terminal and neovim editor fused together by very small rust program. It is incredibly fast. It redraws a large terminal in about 1ms.

In the lower right corner it says last redraw took 1137 micro seconds.

Inspired with this editor's speed I had once again tried to write code for reading Leo outlines and its derived files using rust. The result so far is amazing. A python extension written in rust that parses derived files allows reading and parsing of one of the largest derived files in Leo's distribution (leo/test/activeUnitTests.txt) in about 6ms. 

Python 3.7.0 (default, Jun 28 2018, 13:15:42)

[GCC 7.2.0] :: Anaconda, Inc. on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> def f():

...     with open('leo/test/activeUnitTests.txt', 'r') as inp:

...       s = inp.read()

...       return _minileo.outline_from_str(s)

...

>>> import _minileo

>>> import timeit

>>> print(timeit.timeit(f, number=100)/100*1000, 'ms')

6.344196610007202 ms

A variant that reads external file content once and then profile just parsing gives average time about 3.5ms.

>>> A, B = f()

>>> def f5(n):

...     lev = A[n] >> 18

...     i = A[n] & 0x3ffff

...     return ('--' * lev) + B[i].h

... 

>>> def f6():

...    return '\n'.join(f5(x) for x in range(len(A)))

... 

>>>  print(f6()[:300])

--@file activeUnitTests.txt

----@mark-for-unit-tests

------Test headline abc

------newHeadline

------importTests

--------importCWEBFiles

----------dialog

--------importDerivedFile

----------dialog

--------importFlattenedOutline

----------dialog

--------importMOREFiles

----------dialog

--------import

>>> print(timeit.timeit(f6, number=100)/100*1000, 'ms')

2.8784249000000273 ms
>>> 

As you can see generating textual representation of the whole outline takes less than 3ms. Taking all those times in account one can deduce that:

1. reading external file,
2. parsing it,
3. generating textual representation of the outline
4. extracting just one body text

takes about 9ms and re-rendering 1ms in total 10ms.

Theoretically speaking this combination of glrnvim editor and _minileo python extension library would allow user to type 100 keystrokes or perform 100 movements per second and have responsive display at the same time.  And that is the worst case scenario!!! In reality external file doesn't need to be parsed every time.

I see in this great potential. I am not sure when and if I would find the time to work more on this. In the beginning I was struggling with the rust compiler restrictiveness, but after a few days I feel more confident and I make less mistakes. OTOH using rust py3o library for writing python extension went very smooth from the start. I feel like it would be a shame not to use this technologies almost everywhere.

The next thing I plan to do is to try exporting _minileo module in the javascript world both nodejs and webassembly. There are quite a few tutorials out there and I hope to figure out how to do it. There is also a third possibility I am going to try: to turn this code in to a web server which would serve data from Leo outline and its derived files and use that data from javascript in browser.

Lots of possibilities... I hope one of them will be realized.

Vitalije

PS: any help with writing neovim plugin using _minileo is more than welcome. I am a total newbie in neovim world.

[Edward K. Ream]

On Wed, Jul 3, 2019 at 6:07 AM vitalije <vita...@gmail.com> wrote:

> If Leo creates virtual file system recognizable by the operating system

> and other running applications then every other editor would gain access to

> Leo's features for free with no added code.

Great idea.  How could it be done?

> Right now I am very interested in neovim editor and it's architecture.

> ... it would be possible to use neovim as a front end of Leo file through writing a neovim plugin.

Could be very useful.

> Recently I have found glrnvim a combination of open-gl accelerated terminal

> and neovim editor fused together by very small rust program.

It is incredibly fast. It redraws a large terminal in about 1ms.

Very interesting.

> Inspired with this editor's speed I had once again tried to write code for reading Leo outlines

> and its derived files using rust. The result so far is amazing.

> A python extension written in rust that parses derived files allows reading and parsing

> of one of the largest derived files in Leo's distribution (leo/test/activeUnitTests.txt) in about 6ms.

This is a great prototype.  I am thinking that in a year or three there will be solid tools that integrate

python and rust, possibly using webassembly.

> I see in this great potential. I am not sure when and if I would find the time to work more on this. In the beginning I was struggling with the rust compiler restrictiveness, but after a few days I feel more confident and I make less mistakes. OTOH using rust py3o library for writing python extension went very smooth from the start. I feel like it would be a shame not to use this technologies almost everywhere.

I agree.  This is worth a look.

> The next thing I plan to do is to try exporting _minileo module in the javascript world both nodejs and webassembly. There are quite a few tutorials out there and I hope to figure out how to do it. There is also a third possibility I am going to try: to turn this code in to a web server which would serve data from Leo outline and its derived files and use that data from javascript in browser.

Please keep us informed.  Breaking down the barriers between Leo and other programs

would be a huge step forward.

Edward

Lots of possibilities... I hope one of them will be realized.

Vitalije

PS: any help with writing neovim plugin using _minileo is more than welcome. I am a total newbie in neovim world.

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To post to this group, send email to leo-e...@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/685922fa-6894-4c5d-9b47-53146bbca7cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

------------------------------------------------------------------------------------------
Edward K. Ream: edre...@gmail.com Leo: http://leoeditor.com/
------------------------------------------------------------------------------------------

[vitalije]

I am thinking that in a year or three there will be solid tools that integrate python and rust, possibly using webassembly.

Well it seems much sooner. I have just discovered PyOxidizer, a tool to turn any Python application into a single executable on Windows, Linux or Mac. According to the author, he was new to Rust programming language and started this project in December 2018. For about six-seven months he made quite a lot of progress. Personally I haven't tried it yet, but it is on my short-term list. According to the documentation it is possible to pack binary extensions too, as well as porting parts of the application in Rust. The resulting executable starts faster than Python interpreter and all Python code is pre-compiled and executes without copying, therefore imports are much faster compared to ordinary Python interpreter (which needs to search file system for modules and loads files at runtime). 

What surprised me the most is that ATM there are no GUI libraries for Rust that are production ready. I've found several interesting projects but all of them are still alpha/beta versions or even abandoned or deprecated. As if Rust is more suitable for writing back-ends?! I really don't see why. After all Rust is a general programming language.

Rust build tools are excellent. Installing any Rust version is one command line in Linux. Cargo - build tool can download all necessary files, compile them and link in a single output file (executable or shared / DLL) with just one command line. 

 

Vitalije

[Edward K. Ream]

On Fri, Jul 5, 2019 at 12:26 AM vitalije <vita...@gmail.com> wrote:

I am thinking that in a year or three there will be solid tools that integrate python and rust, possibly using webassembly.

Well it seems much sooner. I have just discovered PyOxidizer, a tool to turn any Python application into a single executable on Windows, Linux or Mac.

Thanks for this link.  I expect lots more such experiments in the next one to three years :-)

What surprised me the most is that ATM there are no GUI libraries for Rust that are production ready.

Qt is a massive project, supported by a large company.  I would not relish doing yet another gui, regardless of its speed.

Rust build tools are excellent. Installing any Rust version is one command line in Linux. Cargo - build tool can download all necessary files, compile them and link in a single output file (executable or shared / DLL) with just one command line. 

I think it will be several years (at least) before python and rust can interoperate smoothly.  I expect webassembly (for the desktop) may be involved.  Yes, progress may be faster than I expect, but I am not counting on it.

Having said that, execution speed has always been an interest of mine.  Back around 1980 I rewrote the screen driver for an 80x24 black and white display.   Iirc, it involved disassembling the 8080 code, just for fun.  I was able to increase the speed of the main loop by reducing its size to about 10 instructions.  Then the screen scrolled about 10 times faster.

So I'm all for speed, provided it doesn't have ill effects in terms of ease of programming and distributing Leo.  The next few years should be exciting ;-)

Edward