Body pane's color syntax highlighting

[Félix]

Hello,

I'm wondering if anyone with knowledge of the Leo code base could give me a simple pointer with relation to Leo's body pane's color syntax highlighting running through leobridge.

Specifically, I'm wondering if there's an easy and straightforward way to get to a body pane's color syntax highlighting (in whatever format that would present itself) that would be applied on the text of a (given) body pane? (while running only Leo without gui, i.e. leobridge)

Thanks for any input on that subject! 

Félix

[Edward K. Ream]

On Sun, May 31, 2020 at 5:23 PM Félix <felix...@gmail.com> wrote:

I'm wondering if anyone with knowledge of the Leo code base could give me a simple pointer with relation to Leo's body pane's color syntax highlighting running through leobridge.

Syntax coloring is never going to be easy. An overview:

- Leo's colorizing code is in leoColorizer.py.

- In addition to language-specific syntax coloring, Leo's syntax coloring supports Leo-specific constructs such as directives and section references.

- Leo's uses jEdit descriptions files in leo/modes as the basis for it's jEdit-based code.

- Leo also supports Pygments colorizing, customized using a modified form of code in the Jupyter project.

Specifically, I'm wondering if there's an easy and straightforward way to get to a body pane's color syntax highlighting (in whatever format that would present itself) that would be applied on the text of a (given) body pane? (while running only Leo without gui, i.e. leobridge)

The leoBridge module typically uses a null gui, iirc, which has a null colorizer, which (by default) won't help you at all.

Starting Leo with --trace=coloring will show you the most important calls to the colorizer. To find the traces, do a cff on 'coloring' (single quotes). You will find, in particular, bjc.setTag and jedit.colorRangeWithTag. Important: Even when @bool use_pygments is True, Leo still calls the two methods listed above.

So, to answer your question, you have several ways forward:

1. Arrange things so that setTag and colorRangeWithTag get called, then modify them to do what you want, presumably using js coloring code.

2. Follow the example of the pygments code, and alter the js coloring code to handle Leo directives and section references.

HTH. Feel free to ask more questions.

Edward

[Félix]

Thanks! 

It's the kind of pointers I was looking for to start tackling the syntax-coloring!

Félix

[Edward K. Ream]

On Thu, Jun 4, 2020 at 4:02 PM Félix <felix...@gmail.com> wrote:

Thanks! 

You're welcome.

Edward

[Félix]

The second option 

>  Follow the example of the pygments code, and alter the js coloring code to handle Leo directives and section references.

Is what I'm going for to get some coloring started in leoInteg. It's something that may get revisited later as the feature grows in maturity... 

I'm going to define the same languages dependant highlighting as leo, along with the syntax highlighting for @xxx directives and <<xxx>>  references by converting to the formats required by vscode :) 

I'll start with the directives and references to get the generic 'Leo' feel of an opened body pane.

thanks again for the tips :) 

[tfer]

As far as getting syntax coloring of body text in vscode, just have leo give you what language it considers the body to be and have your extension change it to that language.  That should be possible as you can click on the "applied" language in thy status bar at the bottom of vscode and change to another language from the list that pops up.

Tom

[Félix]

Thanks that is good input.  I'm considering this. There's also the 'injectable' aspect of grammars in vscode that i'm thinking of using for coloring the directives specific to leo. 

I'm currently learning the 'textmate' grammar definition that's used in vscode. Still not 100% sure of the final approach i'll use but i got some good ideas now. 

gonna continue reading and trying out more stuff with this tomorrow

[tfer]

On Saturday, June 13, 2020 at 4:02:26 AM UTC-4, Félix wrote:

Thanks that is good input.  I'm considering this. There's also the 'injectable' aspect of grammars in vscode that i'm thinking of using for coloring the directives specific to leo. 

That's something I hadn't considered, good point.

 

I'm currently learning the 'textmate' grammar definition that's used in vscode. Still not 100% sure of the final approach i'll use but i got some good ideas now. 

gonna continue reading and trying out more stuff with this tomorrow

Would you explain a bit of the vscode/leo back and forth over the websocket? You implemented a gnx based "filesystem" for body text and a vscode "file browser" to serve as the outline tree, (if I'm understanding things correctly).  Do you use the websocket and leobridge to update the internal state of the associated commander, (e.g. its vnodes), on a transaction by transaction basis?  Does leo handle the weaving/saving?

I ask because I'm thinking of hooking leo to neovim.

Tom

[Edward K. Ream]

On Sat, Jun 13, 2020 at 10:19 AM 'tfer' via leo-editor <leo-e...@googlegroups.com> wrote:

Would you explain a bit of the vscode/leo back and forth over the websocket?

I was about to ask the same question, and for the same reason. See below, including the Post Script.

... 

I ask because I'm thinking of hooking leo to neovim.

Oh joy. This is great news.

I was planning a separate post about adding support for Leo to editors such as emacs and vim, but I might as well comment here and now.

We devs recently "agreed" that adding Leo support for emacs might be a drain on Leo's core devs, but there is more to the discussion. In fact, having Leo plugins for emacs and vim are the two projects I would most like to see. Imo, such plugins should be done by a "native speaker" of vim or emacs. Only a native speaker will know the most natural way of getting the job done.

Just this morning I was trying to recall how Leo in a browser worked. Flexx hides a lot of details. For emacs and vim, using Félix's server (leobridgeserver.py) would be more natural. But we have to know the details of how the server works, what the communication protocols are, etc.

Edward

P.S. Someone commented that the emacs community might not be interested in the Leo way, as contrasted with org mode. I'm sure that would be true of many emacs users, perhaps even most emacs users. But it won't be true of all emacs users if there is an emacs champion willing to follow in Félix's footsteps.

A Leonine alternative to org mode would naturally create both competition and cooperation between the two "camps". Not competition or cooperation, but both simultaneously and ongoingly. Imo, a leo mode would do very well in the competition. If I didn't think so, I wouldn't be interested in the project :-)

P.P.S I have spent several happy hours playing with leointeg. I'll report my experiences in a separate thread.

EKR

[Félix]

Hi tfer :) 

I'm glad you're interested and I'm also happy to explain this project, in fact I'll need to document sooner or later so explaining a bit here is a good start for me to copy/paste it later in some real document in leoInteg's repo.

 

Would you explain a bit of the vscode/leo back and forth over the websocket? You implemented a gnx based "filesystem" for body text and a vscode "file browser" to serve as the outline tree, (if I'm understanding things correctly).  Do you use the websocket and leobridge to update the internal state of the associated commander, (e.g. its vnodes), on a transaction by transaction basis?  Does leo handle the weaving/saving?

Yes, Leo does the file derivation/weaving itself, and file change detection, etc... I merely ask a controller in Leo to do stuff. (as documented in Leo's official leoBridge documentation.) Its a 2 way dance: see /src/leoAsync.ts, vscode's side of things listens for log entries in the 'leo log window' output pane I instanciated and also I react in a 'blocking' way if Leo asks for a confirmation to refresh when file changes are detected.

I did implement a gnx based filesystem for the body panes, but for the outline tree, I implemented a treeview, which has nothing to do with 'file browsing' nor the 'filesystem' used for the body panes. 

Those 2 systems are independent and unrelated. 

/src/leoIntegration.ts orchestrates the method calls between class instances of those 2 controller objects. in short, /src/leoIntegration.ts is the central hub for managing concepts such as: "this was clicked in the tree" and "replace the text in the body pane editor" and other precise actions such as "refresh the tree and make sure the selected node matches the one in Leo" (which is performed when a string of commands have finished resolving - it would be unwise to refresh between rapidly entered commands)

On the Leo side of things, in /leobridgeserver.py,. the script receives a json string, which when converted to a python object, makes for a virtual "bridge" between the two paradigms. 

The crucial line takes the json packet, and knows there's going to be an "action" member string name, and also a 'parameter sub object' ... the crucial line makes use of the "getattr" python method (at line 1082):

answer = getattr(integController, w_param['action'])(w_param['param']) # Crux

So this line calls a method on the integController, by name, contained in 'action' part of w_param. This method called will be called with a set of parameters too: the 'param' part of w_param.

The matching side of what I just explained back into vscode 'leoInteg' side is the function _buildActionParameter in  /src/leoBridge.ts :

/**

* * Build JSON string for action parameter to the leoBridge

* @param p_action Action string to be invoked as command by Leo in the leobridgeserver.py script

* @param p_jsonParam Optional JSON string to be added as a 'param' to the action sent to Leo

*/

private _buildActionParameter(p_action: string, p_jsonParam?: string): string {

return "{\"id\":" + (++this._leoBridgeSerialId) + // no quotes, serial id is a number, pre incremented

", \"action\": \"" + p_action + // action is string so surround with quotes

"\", \"param\":" + p_jsonParam + // param is already json, no need for added quotes

"}";

}

LeoInteg is a year's worth of professional web development across many API, languages and platforms. So I'll continue explaining some more later, feel free to divert the subject to any part in particular. I'm going bed now. g'night chat with y'all some more tomorrow :)

Félix

[Edward K. Ream]

On Sunday, June 14, 2020 at 1:48:33 AM UTC-5, Félix wrote:

I'm glad you're interested and I'm also happy to explain this project, in fact I'll need to document sooner or later so explaining a bit here is a good start for me to copy/paste it later in some real document in leoInteg's repo.

Thanks for this documentation.  leobridgeserver.py is quite a piece of work.

As I understand it, a vim or emacs plugin could use leobridgeserver.py unchanged, provided that the plugin sends the same "messages" as does leoAsync.ts. Is this correct?

Note: leobridgeserver.py contains the async keyword, which requires python 3.7 or above. I typically use python 3.7.6. Leo itself should work with python 3.6.8. Imo, it's fine to require python 3.7+ when using leointeg.

Edward

[Félix]

Thanks for this documentation.  leobridgeserver.py is quite a piece of work.

I'ts been a pleasure to interact with the Leo's api! I wish vscode's api was as documented and understandable as the one for Leo! 

leobridgeserver.py was coded in a leonine way in leoInteg.leo. I thought this was important if I am to get a bit of 'Leo' coders to look into this part of leoInteg in particular. :)

(I personally would be curious to know what you think of the code in the "Outline Edition Commands" node because i'm not 100% sure I made good or wrong/not-undoable operations in there...)

As you correctly stated, leobridgeserver.py (which I often call "the server script")  is self-contained and agnostic towards any client that connects to it. It could indeed be used as-is by any other Leo "integration" projects. 

To dig more into the server/client relation and functionality: /src/leoBridge.ts (which uses /src/leoAsync.ts) is the funnel through which /src/leoIntegration.ts (the central hub of operations) interacts with the server script. See the "Commands for leobridgeserver.py" part in /src/constants.ts for the "actions" that the server supports so far.

I'll add more details as i post in this forum - don't hesitate to ask about parts that interest you! :)

--

Félix

PS: This was meant for this thread, i posted it as a response to the wrong stuff earlier : 

Getting the basic "general" Leo color syntaxing was not that hard once I finally understood the 'textMate' standard for specifying tokenization rules!

live on branch #37 as of now, and soon to be on the dev and master branch in a couple of hours. :) 

on the other hand, doing language specific color syntaxing will take another day at least to finish figuring out the 'injection' rules. (injecting languages specific grammars when encountering @languages directives while keeping leo's own grammar to take precedence)

but I think I'll finish "close, save-as and new" before completing total color-syntaxing for each languages !

[Edward K. Ream]

On Mon, Jun 15, 2020 at 1:09 AM Félix <felix...@gmail.com> wrote:

I'ts been a pleasure to interact with the Leo's api! I wish vscode's api was as documented and understandable as the one for Leo! 

Glad you like it.

leobridgeserver.py was coded in a leonine way in leoInteg.leo. I thought this was important if I am to get a bit of 'Leo' coders to look into this part of leoInteg in particular. :)

(I personally would be curious to know what you think of the code in the "Outline Edition Commands" node because i'm not 100% sure I made good or wrong/not-undoable operations in there...)

The best cool thing is that the js (vs-code) side can invoke any method of the leoBridgeIntegController class by name. This greatly simplifies the protocol.

About names

1. It would be better to capitalize Leo the class name: LeoBridgeIntegController. This helps distinguish classes from instances.

2. In English, "Edition" is typically a noun. It would be clearer to call this node "Outline Editing Commands".

3. There is no such thing as a PNode in Leo. I suggest using, say, markAP instead of markPNode. The AP suffix makes clear that these methods take archived positions as arguments.

About undo

w_p is a position. Leo doesn't support undo on positions. Instead, Leo's commander (c) methods support undo.

So if you want, say, markPNode to support undo, you should (maybe!) use code similar to c_oc.markHeadline. I say "maybe" because I'm not sure what the resulting interactions would be.

Anyway, only commander methods support undo. Imo, each open outline in vs-code should have its own leoBridgeIntegController instance, and that instance should define c to be the commander returned by the call to controller.openLeoFile(path) in the bridge. Your code probably already defines per-commander instances.

It might be possible to use vs-code handle undo in the body pane. But vs-code can't handle undo for outline-related commands.

Nits

1. pyflakes complains that "os" is imported but not used. I always run pyflakes when saving files, so this is a bit annoying for me.

2. There is no need for @others in organizer nodes like "JSON Output Functions", and its following siblings.

3. There is no need for parens in "if(self.commander):". This js-like line appears in several places.

As you correctly stated, leobridgeserver.py (which I often call "the server script")  is self-contained and agnostic towards any client that connects to it. It could indeed be used as-is by any other Leo "integration" projects.

That's great news. There is a lot of heavy lifting in leobridgeserver.py. It will be great to reuse the work.

And not just for the code. leobridgeserver.py is a template (or target) for the editor-side work.

To dig more into the server/client relation and functionality: /src/leoBridge.ts (which uses /src/leoAsync.ts) is the funnel through which /src/leoIntegration.ts (the central hub of operations) interacts with the server script. See the "Commands for leobridgeserver.py" part in /src/constants.ts for the "actions" that the server supports so far.

Thanks for these tips. I'll study every line of code in leoBridge.ts, leoAsync.ts and leobridgeserver.py. And other files too.

 

I'll add more details as i post in this forum - don't hesitate to ask about parts that interest you! :)

Thanks very much. I think I understand the big picture.

Here is my highest-level overview of the comparison between your work and leoflexx.py (Leo in a browser). Please correct any misunderstandings.

1. The first fundamental task is to bridge the gap between the js side and the python side.

flexx handles this with @action and @reaction decorators. In flexx, everything looks like python, but in fact everything on the js side is, perforce, javascript.

In leoInteg, the leobridgeserver.py code (and the code that interacts with it) bridges the js/python divide.

Only text can pass over the divide. vs-code knows nothing about Leo's classes. As a result, your js code must use AP's (archived positions), which are just text (json). AP's and related code are likely the biggest legacy of leoflexx.py.

flexx simulates/allows arbitrary two-way interactions between the js and python sides. Your code gets the same effect with a single server. Requests flow from vs-code to Leo's bridge. Responses to those requests are the only data that need to flow in the other direction. You have shown that these messages suffice.

It's super elegant to use "await" to make all this asynchronous. Thank you python devs!

2. The second fundamental task is to implement Leonine widgets. Flexx makes this easy with pre-built js widgets. I assume you use the vs-code api to create your Leonine widgets in vs-code.

Edward

P.S. I recently "rediscovered" Leo info issue #1015, LeoWapp theory of operation. I do recommend writing an theory issue for leoInteg as you go along.

There is an amusing effect that happens often to me. I consider it to be a positive effect of forgetting details, which happens more and more the older I get :-) As I forget details, the bigger picture comes ever more clearly into focus.

I say this is an "amusing" effect because it's easy to miss the biggest picture while documenting the details. You can see this in #1015. It doesn't explicitly mention the js/python divide!

EKR

[Félix]

Here is my highest-level overview of the comparison between your work and leoflexx.py (Leo in a browser). Please correct any misunderstandings.

Ok :)

1. The first fundamental task is to bridge the gap between the js side and the python side.

Indeed, Understanding that there was such a thing in Leo's source as 'leoBridge', and stumbling upon the ap-to-p and p-to-ap source code, is what made me start this project. along with this post : https://github.com/leo-editor/leo-editor/issues/1025 (I thought the current state of things was a month or two away! haha!). Also btw, it's in  'typescript' akin to coffescript and the like, so not technically 'javascript'.

flexx handles this with @action and @reaction decorators. In flexx, everything looks like python, but in fact everything on the js side is, perforce, javascript.

It is with shame that I will admit to you that I have not looked into flexx enough to understand / acknowledge this. I have barely spent enough time and effort in flexx to get to those methods. 

In leoInteg, the leobridgeserver.py code (and the code that interacts with it) bridges the js/python divide.
Only text can pass over the divide. vs-code knows nothing about Leo's classes. As a result, your js code must use AP's (archived positions), which are just text (json). AP's and related code are likely the biggest legacy of leoflexx.py.

Yes, and what you referred to as 'text' is veeerrry sensible to being JSON, and not actually just 'text'. 

flexx simulates/allows arbitrary two-way interactions between the js and python sides. Your code gets the same effect with a single server. Requests flow from vs-code to Leo's bridge. Responses to those requests are the only data that need to flow in the other direction. You have shown that these messages suffice.

It's not exactly right. That's where the /src/leoAsync.ts class comes into play: Some events may be initiated by Leo itself and so JSON messaging may sprout from the python side without being suspected as usual when just responding to a request initiated from the user's (vscode) side. But otherwise for user interaction its just do action->wait response from python. one by one, in a stack, no random order.

use "await" to make all this asynchronous. Thank you python devs!

I just followed the 'websocktets' documentation for a basic server that waits/responds to simple serial queries.

2. The second fundamental task is to implement Leonine widgets. Flexx makes this easy with pre-built js widgets. I assume you use the vs-code api to create your Leonine widgets in vs-code.

Being an integration into vscode, there is indeed, mostly just a ton of vs-code api shenanigans going on :) have not gone mad when trying to go about this api, yet... :)

Vscode and Leo are the 2 coolest and useful tools in a programmer's arsenal. So I really enjoy discussing them!

Félix

[Edward K. Ream]

On Wed, Jun 17, 2020 at 12:48 AM Félix <felix...@gmail.com> wrote:

>> flexx handles this with @action and @reaction decorators. In flexx, everything looks like python, but in fact everything on the js side is, perforce, javascript.

> I will admit to you that I have not looked into flexx enough to understand / acknowledge this. I have barely spent enough time and effort in flexx to get to those methods.

You found archived positions. That's all that matters.

>> flexx simulates/allows arbitrary two-way interactions between the js and python sides. Your code gets the same effect with a single server. Requests flow from vs-code to Leo's bridge. Responses to those requests are the only data that need to flow in the other direction. You have shown that these messages suffice.

> It's not exactly right. That's where the /src/leoAsync.ts class comes into play: Some events may be initiated by Leo itself and so JSON messaging may sprout from the python side without being suspected as usual when just responding to a request initiated from the user's (vscode) side.

Thanks for this important clarification.

> But otherwise for user interaction it's just do action->wait response from python. one by one, in a stack, no random order.

Does this mean that the interaction is actually synchronous?

>> use "await" to make all this asynchronous. Thank you python devs!

> I just followed the 'websockets' documentation for a basic server that waits/responds to simple serial queries.

Heh. You make it sound so easy. For the last several days I've been working on a python version of your typescript code. This would be helpful "documentation". I would use it as a prototype for a leoPyzoInteg. Imo, embedding Leo in pyzo or vice versa is hopeless. You have shown the correct way forward.

For this newbie, just attempting to follow the websockets documentation can be, um, challenging. I'm busy making all the usual newbie mistakes and googling the resulting error messages. And reading, reading, reading, attempting to separate what I need from what I don't.

The names of methods and various code patterns are starting to become familiar, but no joy yet. I'll continue to study your code to see what you have done. If I get stuck I'll ask you for advice.

>> 2. The second fundamental task is to implement Leonine widgets. Flexx makes this easy with pre-built js widgets. I assume you use the vs-code api to create your Leonine widgets in vs-code.

> Being an integration into vscode, there is indeed, mostly just a ton of vs-code api shenanigans going on :) have not gone mad when trying to go about this api, yet... :)

Hehe. It's never going to be easy, even in pyzo.

> Vscode and Leo are the 2 coolest and useful tools in a programmer's arsenal. So I really enjoy discussing them!

Many thanks for your comments. They will help me create a prototype (of two-way communication) for leoPyzoInteg. They should also help tfer (Tom Fetherston) with his leoVimInteg project. I'm also considering leoEmacsInteg, but that will wait.

Edward

[Félix]

You found archived positions. That's all that matters.

I sure f*cking did... haha!! That's really the one most impactful thing that got the ball rolling! :D

Just a thought I often had:  vscode is now, and going to be for many years, the top editor used by the most people on the planet. It has dethroned sublime-text, which had the crown for many years. (or belt, if you're a wrestling fan) 

And it so happens that sublime is scriptable in python... And so I often raged at myself for not doing a Leo integration when the target IDE would have been scriptable in python! There would have been no complicated websocket bridging and all that ! just simple 'import leo.core.leoBridge as leoBridge' or whatever, and directly call stuff in leoBridge.py... haha! (but i'm oversimplifying obviously!)

I can't really comment on stuff relating to Vim and Emacs because I know next to nothing about those except that they're popular editors with their own 'system' of editing text.

sorry for this rambling I'm going back to work on leoInteg now ! 

btw last night I made ctrl+b on selected text a reality, - did'nt know this handled @others and the like- wow!- and also fixed headline editing with double quotes and special characters. So this makes it so that it is now possible to re-do this video in its entirety. https://www.youtube.com/watch?v=tuM8MvI9g6k Which I will do. : )

Félix

[Edward K. Ream]

On Wed, Jun 17, 2020 at 3:07 PM Félix <felix...@gmail.com> wrote:

Just a thought I often had:  vscode is now, and going to be for many years, the top editor used by the most people on the planet. It has dethroned sublime-text, which had the crown for many years. (or belt, if you're a wrestling fan) 

I knew vscode was popular, but I didn't know it was that popular. See this survey. I'm shocked that so few devs use emacs.

And it so happens that sublime is scriptable in python... And so I often raged at myself for not doing a Leo integration when the target IDE would have been python! There would have been no complicated websocket bridging and all that ! just simple 'import leo.core.leoBridge as leoBridge' or whatever, and directly call stuff in leoBridge.py... haha! (but i'm oversimplifying obviously!)

Oh! I didn't get what you just said until the second time of reading! For pyzo, there is no need to transliterate your bridging code into python. Just use leo.core.leoBridge from pyzo!

I'm stunned that I didn't think of this before!

Or maybe I did think of it before, in a slightly different guise. Anyway, your work shows the scale of the effort required. There are two possibilities:

1. Add a "Leo plugin" to pyzo (ignoring the fact that pyzo doesn't support plugins).

2. Add a "pyzo plugin" to Leo. A feeble plugin,  pyzo_in_leo.py, already exists.

Either way, the amount of work to fully integrate Leo and pyzo would be comparable to your work with leoInteg. It hardly seems worth doing.

I can't really comment on stuff relating to Vim and Emacs because I know next to nothing about those except that they're popular editors with their own 'system' of editing text.

sorry for this rambling I'm going back to work on leoInteg now ! 

This post has been earth shaking for me:

I've spent the last several days delving into the innards of leobridgeserver.py and all the code on the JS side. The idea was to transliterate your typescript code into python. But there is no need to do this! There is no ts/python divide when supporting Leo inside pyzo. Just call the leoBridge!

Furthermore, your work shows the scale of the effort required to do the pyzo_in_leo.py plugin properly. I doubt that this level of effort would ever be worthwhile.

I'm dithering about leoEmacsInteg. True, few devs now use emacs. Otoh, I suspect millions of non-devs use org mode...

btw last night I made ctrl+b on selected text a reality, - did'nt know this handled @others and the like-  and also fixed headline editing with double quotes and special characters. So this makes it so that it is now possible to re-do this video in its entirety. https://www.youtube.com/watch?v=tuM8MvI9g6k Which I will do. : )

Many thanks for your continued dedication to the leoInteg project.

Edward