ENB: Syntax coloring in leoInteg

[Edward K. Ream]

This Engineering Notebook post discusses how to support Leonine syntax coloring in vs-code. This post will focus only on the biggest picture. Details must wait until various experiments are complete.

Present status

At present, leoInteg defines Leo-specific syntax in syntaxes/leoBody.tmLanguage.json. This is a start, but it doesn't color non-Leonine syntax.

What must be done

1. Make Leonine syntax available in every node, regardless of what @language directive is in effect, while retaining the normal syntax coloring that should be done for each language.

2. Handle multiple @language directives in a single body text.

Two strategies

There seem to be two possible ways forward:

Strategy 1: Use vs-code's syntax-coloring services, enhanced with Leonine features.

Strategy 2: Use Leo's existing syntax-coloring code, and communicate the results back to vs-code.

Happily, leoInteg can probably combine the best features of both strategies!

The Language Server Protocol

The Programmatic Language Features page suggests implementing a language server that implements the Language Server Protocol (LSP). The Language Sever Extension Guide page indicates that LSP was designed to handle the resource-intensive nature of the problem. This means that Leo should be able to use the LSP without serious performance issues.

Using an LSP is more complicated than implementing a vscode.language hook, but LSP has important advantages:

1. The Language Server can be written in any language.

2. The Language Server can be reused to provide smart editing [or coloring] features for multiple editors.

Bingo! We can repackage (parts of?) Leo's existing syntax coloring code as a language server!

The details are fuzzy. It might be possible to use "leo" as the language, and have the server handle all aspects of coloring. Or it might be more convenient to have the server modify only Leonine constructs such as Leo directives and section references.

Summary

We want to support multiple @language directives in body panes, as well as all Leonine syntax. Aside from that, we want to default to vs-code's syntax coloring. That is, we want to "interweave" Leonine coloring with vs-code's standard coloring.

We don't want to write a new json colorizer description for each language that vs-code supports. Nor do we want to create a bespoke vscode.language hook for every language that vs-code supports. Instead, leoInteg's colorizing code will likely use an LSP. The LSP can be written in python, which means a repackaging of Leo's existing colorizing code may be possible.

Significant new invention will be required. However, the LSP was designed for just such task. I am fairly confident that leoInteg will be able to do Leonine syntax coloring quickly enough.

Edward

[Edward K. Ream]

On Sun, Jun 28, 2020 at 8:11 AM Edward K. Ream wrote:

>  leoInteg's colorizing code will likely use an LSP. The LSP can be written in python, which means a repackaging of Leo's existing colorizing code may be possible.

Yes, Leo's LSP could be written in python, but that may not be the easiest way. Leo's LSP will likely delegate most coloring to vs-code's default colorizers for the @language in effect. This would likely be easiest to do in ts.

Leo's LSP would (sometimes!) break body text into chunks consisting of a range of lines to be colored by the same language, with only Leonine lines (lines containing Leo directives and section references) being colored using syntaxes/leoBody.tmLanguage.json.

It should be possible to avoid re-calculating chunks on every keystroke. After all, existing vs-code colorizers almost certainly do not re-tokenize the entire file on every keystroke. Figuring how to avoid this recalculation is next on my study list.

Edward

[tfer]

Another possibility on the vscode side would be to create a syntax file that is the union of the leosyntax and the language in effect for the node.

[Edward K. Ream]

On Mon, Jun 29, 2020 at 9:26 PM 'tfer' via leo-editor <leo-e...@googlegroups.com> wrote:

Another possibility on the vscode side would be to create a syntax file that is the union of the leosyntax and the language in effect for the node.

Thanks for this. I had not considered this possibility. How would this work for nodes containing multiple @language directives?

Edward

[Félix]

tfer,

My favored solution is to make leo-body-python leo-body-c syntax files etc... for all languages Leo supports and switch accordingly with textmate rules because that supports switching languages in the same node multiple times. ( Leo has trouble starting a child node in the right language when a parent has multiple @languages directives)

I'm consciously letting Edward explore a bit of the language server stuff for another 72 hours while I finish other features.

My hope is that I misunderstood their impertinence and that Edward finds a way to use that which may have gone over my head.

My forecast is that I will soon come and rain on this parade  ;)

Felix

[Edward K. Ream]

On Tue, Jun 30, 2020 at 1:58 PM Félix <felix...@gmail.com> wrote:

My forecast is that I will soon come and rain on this parade ;)

I have no attachment to the LSP. In fact, I would be relieved if there were some other way.

Edward

[Félix]

 I would be relieved if there were some other way.

I suspected that. I have a basic concept of a way to make it work in mind, which I will layout more precisely here and try out soon. 

But who knows....There may be other imaginative ways to make it work... so i'm very curious to hear anyone's ideas about this

Félix

[Edward K. Ream]

On Tue, Jun 30, 2020 at 9:18 PM Félix <felix...@gmail.com> wrote:

 I would be relieved if there were some other way.

I suspected that. I have a basic concept of a way to make it work in mind, which I will layout more precisely here and try out soon. 

Good. It's fine with me if the first draft supports only one @language directive per node. Support for multiple @language directives in a single node can wait till later.

Edward