ENB: The kernel of leojs: Leo in JS

[Edward K. Ream]

This Engineering Notebook post considers strategies for implementing the crucial parts of Leo in javascript. leojs issue #9 recaps what must be done. The following must be present in leojs 0.1:

- Leo's Position and VNode classes,

- Outline-oriented undo,

- Outline-oriented search.

That's a lot of complex code!

True, leojs is worth any amount of work. However, it would be stupid to dive into coding! This morning I considered some cute possibilities for doing significant parts of the work in python. Then I thought to google: "python to javascript" :-)

One of the first hits was this overview, which leads to:

- JavaScripthon: a Python 3 to ES6 JavaScript translator.

- RapyScript: pre-compiler for JavaScript.

  The syntax is very similar to Python but allows JavaScript as well.

- Transcript: compile python to JS.

- Pscript: the compiler part of flexx.

Summary

This has been a much shorter post than I first imagined :-) One or more of the tools listed above will likely suffice. A little googling pays off!

leoInteg requires Leo's bridge to provide most of Leo's functionality. The communication between Leo's bridge and leoInteg is fraught with difficulties. Transcribing the essentials of Leo's core into JS will eliminate many of those difficulties.

Besides transliteration, other problems will surely arise. Félix and I will deal with those problems as they arise. Time to evaluate transpilers!

Edward

[tbp1...@gmail.com]

I believe that Graal can (or soon will be able to) target both python and javascript.  What I'm not sure about yet is whether you could write in python and output javascript.  It may be that you would have to write in java and then you could output either python or javascript (see, e.g., https://www.graalvm.org/reference-manual/native-image/).  That limitation would be a non-starter for me.

[Edward K. Ream]

On Wed, Dec 16, 2020 at 12:43 PM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

I believe that Graal can (or soon will be able to) target both python and javascript. 

Thanks for this.  My preliminary researches:

- javaScripthon is not ready for prime time. It doesn't support f-strings, for example and its error reporting is poor.

- Transcrypt seems to work better, but we shall see.

Edward

[vitalije]

Here are some ideas which you are free to ignore if you find them uninteresting or useless.

There is also a possibility to have Leo's core implemented in rust (or some other low level language) and exported as both python extension module and node module. I've started to work on this idea, but right now I am too busy to work on that project (mini_leo).

I did offer Felix to use this code (instead of leoBridge) when he started his project, but he wasn't interested.

Exporting rust functions to python and node is easy. There is a possibility to export same functions as a wasm module for use in the browsers too.

I don't fully remember current state of the project but IIRC there are already functions for reading/writing .leo files, derived at-file files, iterating through outlines, basic outline manipulations.

Here is a benchmark test:

Python 3.7.0 (default, Jun 28 2018, 13:15:42)
[GCC 7.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def f():
              with open('leo/test/activeUnitTests.txt', 'r') as inp:
              s = inp.read()
              return _minileo.outline_from_str(s)
>>> import _minileo
>>> import timeit
>>> print(timeit.timeit(f, number=100)/100*1000, 'ms')
6.344196610007202 ms

As you can guess function outline_from_str parses derived at-file content and builds outline. leo/test/activeUnitTests.txt used to be the largest derived file in Leo's code. There is also a function load_leo to load outline from .leo file along with all external (derived) files.

I've also some ideas of using sqlite3 in combination with a few functions written in rust for dealing with the outlines. I had some very promising experiments with this idea. The sqlite3 proves to be fast enough (faster than python) for iterating outlines, handling undo/redo, searching for nodes,...

And exporting this functionality to all scripting languages (python, nodejs, and javascript in browsers) is straightforward.

I don't know when will I have the time to continue work on this project, but the code is there, for what it's worth.

Reimplementing Leo's core classes in Python to be compiled to JavaScript, today, is IMHO a bad idea (YMMV). Python is great for other parts of Leo, but not so great for the core. If the core should be reimplemented at all, I believe it would be complete waste of time unless it is reimplemented in a language that is substantially faster than the current python implementation. Rust seems to be fine choice these days for this task. The library can be cross compiled to work on any machine, including browser and it also allows exporting native rust functions to both python and nodejs. But Rust isn't the only one in town. There are also zig and v languages. Both of them claim that they've fixed almost all problems with C language. Both of them offer good cooperation with python, nodejs and javascript in browser.

I dream of a Leo core module that could be easily used in server code, browser front-end and native gui front-end. With such a module and little bit of scripting it would be trivial to have outlines displayed on the web and editing outlines remotely. Imagine the combination of leo-ver-serv (which keeps detailed history of my Leo outlines taking snapshots every few idle seconds), editing outlines remotely, and displaying outlines in browser. That combination would be better for hosting code than github. Currently presentation of Leo outlines on github is terrible. Imagine that people can see the code organized in Leo outlines, and even to suggest some change by editing nodes directly in browser.

My two cents.

Vitalije

[Offray Vladimir Luna Cárdenas]

On the idea of a Leo core in a machine efficient fast language, I think that Nim is a worth exploration also, as is syntax is Python inspired, compiles to JavaScript, has good metaprogramming facilities. I have found it pretty productive when I need a more low level or multi-paradigm, cross browser approach to my coding needs.

https://nim-lang.org/

Offray

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/c4401dca-c02a-4859-a4fe-e652fc7e6548n%40googlegroups.com.

[Edward K. Ream]

On Wed, Dec 16, 2020 at 1:32 PM Offray Vladimir Luna Cárdenas <off...@riseup.net> wrote:

On the idea of a Leo core in a machine efficient fast language, I think that Nim is a worth exploration also, as is syntax is Python inspired, compiles to JavaScript, has good metaprogramming facilities. I have found it pretty productive when I need a more low level or multi-paradigm, cross browser approach to my coding needs.

Thanks for the link. What I want is something that will compile Leo's existing python code to js/ts.

Edward

[tbp1...@gmail.com]

PyPy has a "preliminary" version of a javascript interpreter for Python code.

https://doc.pypy.org/en/latest/faq.html?highlight=javascript#can-i-use-pypy-s-translation-toolchain-for-other-languages-besides-python

[tbp1...@gmail.com]

I also found dukpy:

dukpy

" DukPy is a simple javascript interpreter for Python built on top of duktape engine without any external dependency".  It can transpile to Typescript.

[Félix]

Hi guys, 

Here are my thought and notes :)

I didn't know about rust when  Vitalije  first suggested his ideas. I was interested in ways to rapidly connect python in vscode to run Leo's features from inside vscode. So indeed i wasn't interested - simply because it was not relevant to my immediate goals at the time (bridging leo's python process into vscode) and my ignorance of the rust language.

Now knowing what it can do, and its power to be turned into 'byte code' and ran natively or in webassembly, it's clear that the ultimate platform to have a 'kernel' or 'core' for any service (well, practically any) is rust.

(thanks to  Vitalije  for pointing my attention to the rust language many months ago)

That being said, connecting leojs UI to a 'leo core' in an javascript environment can be done at first with some javascript code either rewritten or automatically transpiled, and then as a bigger endeavor, re-written in rust and used in the same leojs UI through webassembly  without any modifications. (on the UI part.) 

So doing one does not impede the other, nor make leojs's UI interface for one useless for the other: it would use the same.

I also dream of a day where a Rust Leo Core would come. 

Doing one in js, or automatically converted/compiled into js from python, is also a good learning first step.

So i'll try to write leojs vscode/UI side of things so that no changes will be needed if the core is pure js, ts, transpiled, interpreted, live or whatever, even rust through webassembly. Simply by writing it with a clear and concise exposed API. 

--

You guys effin' rock! :D

Félix

[tbp1...@gmail.com]

If you plan to write a server to be the go-between between Leo and vs, it's the API that counts, not the coding language.  The server language can be whatever works well for you. So Rust, sure, go for it.  Leo/python will be able to talk with your API.  If you want to have an especially well-architected design, separate the server input API from the output API.  Then would you have the possibility of writing backends for other systems besides vs.

[Edward K. Ream]

On Wed, Dec 16, 2020 at 1:17 PM vitalije <vita...@gmail.com> wrote:

Here are some ideas which you are free to ignore if you find them uninteresting or useless.

Thanks for checking in, Vitalije. I'm glad you are well and busy :-) The goals of Leo in rust makes a lot of sense in the long term.

To second Félix's comments to your post, for the leojs project we have no choice but to transpile Leo's core to JS.

Happily, Transcrypt seems just about perfect. It transliterates Leo's python sources to JS without much fuss. The ekr-imports branch contains just a few tweaks:

- Transcrypt doesn't support Python's copy module. I added transcrypt pragmas to disable the python code. Other pragmas could be used to insert corresponding JS code.

- Transcrypt doesn't support Leo's legacy imports. It is straightforward (and a welcome simplification) to change, for example,

import leo.core.leoGlobals as g

import leo.commands.leoEditCommands as leoEditCommands

to:

from leo.core import leoGlobals as g

from leo.commands import leoEditCommands

Summary

Leo in rust is worth exploring. Tools such as vs-code might eventually use rust for added speed. When that happens, it would likely be straightforward to port leoInteg and leojs to rust :-)

Edward

[Edward K. Ream]

On Wed, Dec 16, 2020 at 7:57 PM Félix <felix...@gmail.com> wrote:

Here are my thought and notes :)

Thanks. They all seem reasonable to me.

...connecting leojs UI to a 'leo core' in an javascript environment can be done at first with some javascript code either rewritten or automatically transpiled...

 

As I said in my reply to Vitalije, the Transcrypt transpiler seems just about perfect. See the ekr-imports branch for the tweaks.

Edward

[Edward K. Ream]

On Thu, Dec 17, 2020 at 6:43 AM Edward K. Ream <edre...@gmail.com> wrote:

> ...the Transcrypt transpiler seems just about perfect. See the ekr-imports branch for the tweaks.

PR #1787 contains the diffs. I'll be converting all of Leo's imports to the new style.

Edward

[Offray Vladimir Luna Cárdenas]

Transcrypt seems working pretty good in that regard, according to the new mailing list findings. My suggestion was more related with a far ahead  Leo Core (like the one proposed in Rust) also pointing to Nim as a language closer to Python that shares many Rust goals.

Cheers,

Offray