Loading Leo using rust

[vitalije]

I have finally found some time to continue work on python extension written in Rust and aimed to provide some functions useful for Leo. Today I've completed function `load_leo(fname)` which loads outline from given Leo file, and then loads all external files (only at-file kind for now). Preliminary test gives fantastic results. On my machine, it loads LeoPyRef.leo along with all external files in about 69ms. Traversing the whole outline takes about 6.5ms.

>>> from _minileo import *
>>> def f():
...     tid = load_leo('leo/core/LeoPyRef.leo')
...     return tid
...
>>> import timeit
>>> timeit.timeit(f, number=100)*1000/100
68.48035373001039
>>> def f2():
...     n = 0
...     for lev, v in iternodes(0):
...         n += lev
...     return n
>>> timeit.timeit(f2, number=100)*1000/100
6.528250280007342

Code can be found here.

To build it one needs to install rustup, then using it install nightly version toolchain and finally execute `cargo build --lib --release` in the rust folder.

rustup can be found on https://rustup.rs. Follow the instructions there to install rustup.

> rustup toolchain install nightly
> git clone https://github.com/vitalije/mini_leo.git
> cd rust
> cargo build --lib --release

This command will build in the `target/release/libmini_leo.so` on linux or `libmini_leo.dll` on Windows.

This file should be copied somewhere on PYTHONPATH, and it must be renamed to _minileo.pyd or _minleo.so.

Vitalije

[Edward K. Ream]

On Sun, Dec 29, 2019 at 3:30 PM vitalije <vita...@gmail.com> wrote:

I have finally found some time to continue work on python extension written in Rust and aimed to provide some functions useful for Leo. Today I've completed function `load_leo(fname)` which loads outline from given Leo file, and then loads all external files (only at-file kind for now). Preliminary test gives fantastic results. On my machine, it loads LeoPyRef.leo along with all external files in about 69ms. Traversing the whole outline takes about 6.5ms.

That's sweet. Many thanks for your work. webassembly promises to make rust available easily in the python world. It's exciting.

> Code can be found here.

Thanks.  I'll study it with great interest.  And I'll keep my hands off it ;-)

Edward

[Edward K. Ream]

On Sunday, December 29, 2019 at 2:30:52 PM UTC-6, vitalije wrote:

I have finally found some time to continue work on python extension written in Rust and aimed to provide some functions useful for Leo.

I just now got around to installing and building mini_leo.

...

Code can be found here.

To build it one needs to install rustup, then using it install nightly version toolchain and finally execute `cargo build --lib --release` in the rust folder.

Installation was non-trivial. You must install the "nightly" version of rust!

It's not enough to do `rustup toolchain install nightly`. I had to do a complete reinstall of rust.

After that, `cargo build --lib --release` appears to have worked.

> This command will build in the `target/release/libmini_leo.so` on linux or `libmini_leo.dll` on Windows.

Now I'm stuck. How do I run mini_leo?

In ...\mini_leo\rust\target\release I see these files:

Directory of c:\leo.repo\mini_leo\rust\target\release

06/20/2020  04:33 PM    <DIR>          .
06/20/2020  04:33 PM    <DIR>          ..
06/20/2020  03:38 PM                 0 .cargo-lock
06/20/2020  04:30 PM    <DIR>          .fingerprint
06/20/2020  04:30 PM    <DIR>          build
06/20/2020  04:33 PM    <DIR>          deps
06/20/2020  03:38 PM    <DIR>          examples
06/20/2020  03:38 PM    <DIR>          incremental
06/20/2020  04:33 PM               287 mini_leo.d
06/20/2020  04:33 PM           434,176 mini_leo.dll
06/20/2020  04:33 PM               980 mini_leo.dll.exp
06/20/2020  04:33 PM             1,972 mini_leo.dll.lib
06/20/2020  04:33 PM         1,175,552 mini_leo.pdb
               6 File(s)      1,612,967 bytes
               7 Dir(s)  91,636,563,968 bytes free

What do I do now?

Edward

[vitalije]

Installation was non-trivial. You must install the "nightly" version of rust!

It's not enough to do `rustup toolchain install nightly`. I had to do a complete reinstall of rust.

Actually you didn't need to reinstall rust. The rustup utility is capable of keeping several different tool chains at once. You should have run:

rustup default nightly

This would set default tool chain to nightly version, and following cargo commands would use nightly version of rust.

 

After that, `cargo build --lib --release` appears to have worked.

> This command will build in the `target/release/libmini_leo.so` on linux or `libmini_leo.dll` on Windows.

Now I'm stuck. How do I run mini_leo?

mini_leo doesn't have any GUI yet, so it can't be run on its own. I haven't decided which GUI to use yet.

At its current state mini_leo is just a python extension module. Copy mini_leo.dll somewhere on PYTHONPATH and try importing "_minileo".

import _minileo
t1 = _minileo.load_leo('leo/core/LeoPyRef.leo')
print(f'total number of nodes {_minileo.tree_len(t1)}')
for level, v in _minileo.iternodes(t1):
    print('-'*level, v.h)

t1 in the above example is just an int, a handle to loaded outline. Many _minileo functions expect an outline handle as their first argument.

Function iternodes(outlinehandle) returns a generator which yields tuples (level, vdata). VData instances have h, b, and gnx fields. To actually change tree one must use _minileo functions. Yielded vdata instances are just copies taken from the tree, so changing them won't change the original.

Function outline_from_str(content) parses the content which should be in leo-thin-5 format, and returns a handle to the loaded outline. The outline_from_file is the same just reads content from the given file. The outline_from_leo_str and outline_from_leo_file functions expect xml content and return outline handle.

I don't have much time now to write or to work on Leo. I hope next week I will continue to work on the #1598 and write more about it.

I've attached small python script that demonstrate using mini_leo extension for loading Leo outline into Leo commander c.

HTH Vitalije

[Edward K. Ream]

On Mon, Jun 22, 2020 at 1:40 AM vitalije <vita...@gmail.com> wrote:

Actually you didn't need to reinstall rust. The rustup utility is capable of keeping several different tool chains at once. You should have run:

rustup default nightly

This would set default tool chain to nightly version, and following cargo commands would use nightly version of rust.

Good to know. Thanks.

Now I'm stuck. How do I run mini_leo?

mini_leo doesn't have any GUI yet, so it can't be run on its own. I haven't decided which GUI to use yet.

At its current state mini_leo is just a python extension module. Copy mini_leo.dll somewhere on PYTHONPATH and try importing "_minileo".

I've tried various things and can import neither _minileo nor minileo.

This can wait until you have more time.

Edward

[vitalije]

Have you tried renaming mini_leo.dll to _minileo.dll or _minileo.pyd ?

Vitalije

[vitalije]

I am sure I have tried it on Windows too, but I can't remember if the name of the dll file has some significance in Windows or not. I know that on Ubuntu I had to rename libmini_leo.so to _minileo.so.

Vitalije

[Edward K. Ream]

On Mon, Jun 22, 2020 at 12:21 PM vitalije <vita...@gmail.com> wrote:

Have you tried renaming mini_leo.dll to _minileo.dll or _minileo.pyd ?

Renaming to _minileo.pyd worked.  Thanks!

Edward