First draft: documentation for leoservers.py

[Edward K. Ream]

The following is pre-writing for a new Chapter in Leo's documentation.

All comments, suggestions, additions, and corrections are welcome.

-----

leoserver.py is a stand-alone web server that provides access to all of Leo's capabilities using Leo's bridge.  leoserver.py is not part of Leo's core.

leoclient.py is an example client, written in python. Clients may be written in any language. Clients send text requests to the server, in JSON format, and receive JSON responses from the server.

This chapter describes how to start the server and the communication protocol, that is, the format of the JSON requests and responses.

Starting the server

To start the server, execute `python leoserver.py <args>` in a separate process, that is, outside of Leo itself. `leoserver -h` prints "leoserver.py -a <address> -p <port>"

 By default, the server listens on "localhost" and port 32125. You can change these defaults with the --address and --port command-line arguments.

Communication protocol: requests
 
The _do_message method expects that incoming requests are JSON objects that can be converted to a python dict, with the following three keys:

"id": A positive integer.

"action": A string, which is either:
  - The name of public method of this class, prefixed with a '!'.
  - The name of a leo command, without prefix, to be run by _do_leo_command.

"param": A dict to be passed to the called "action" method.

Communication protocol: requests

The _make_response and _make_minimal_response methods return python dicts that are then converted do JSON. All responses contain an 'id' key that matches the corresponding requests.

Archived positions

Leo's Position objects can not be converted directly to JSON. Instead, requests and response contain archived positions, that are strings representing positions. The server's _ap_to_p and _p_to_ap methods convert between real Leo Positions and archived positions.

Technical details

The ws_handler function contains the main server loop. It calls Python's asyncio.get_event_loop() to start listening on the designated socket and port.

ws_handler dispatches incoming requests to the _do_request method, which then dispatches the requests to the appropriate handler, either in the LeoServer class, or (via Leo's bridge) to Leo itself.

 Exception handling

Request handlers raise ServerError exceptions for badly formatted user requests. The server returns an response describing the error.

The server raises InternalServerError for errors within the server itself. The server prints an error message and stops.

Request handlers raise TerminateServer to terminate the server normally.

Compatibility and future directions

The present version of leoserver.py exists to support leoInteg: Leo hosted on vs code.

In future, features (request handlers) can be added to leoserver.py, but features probably will never be removed.

Acknowledgements

Félix Malboeuf wrote the original version of leoserver.py including the first draft of the all-important The ws_handler function. Edward K. Ream helped make the code more pythonic, and provided Leo-specific advice.

Edward

[Edward K. Ream]

On Friday, May 14, 2021 at 7:54:08 AM UTC-5 Edward K. Ream wrote:

The following is pre-writing for a new Chapter in Leo's documentation.

LeoDocs.leo in the ekr-docs branch contains the latest version of the docs. Please respond here with your suggestions.

Edward

[Viktor Ransmayr]

Hello Edward,

Am Fr., 14. Mai 2021 um 14:54 Uhr schrieb Edward K. Ream <edre...@gmail.com>:

The following is pre-writing for a new Chapter in Leo's documentation.

All comments, suggestions, additions, and corrections are welcome.

-----

leoserver.py is a stand-alone web server that provides access to all of Leo's capabilities using Leo's bridge.  leoserver.py is not part of Leo's core.

leoclient.py is an example client, written in python. Clients may be written in any language. Clients send text requests to the server, in JSON format, and receive JSON responses from the server.

Would it make sense to ignore mentioning leoclient.py at this point in time completely - or - at least defer mentioning it to the time, when the CP: request / response model has already been described?

This chapter describes how to start the server and the communication protocol, that is, the format of the JSON requests and responses.

Starting the server

To start the server, execute `python leoserver.py <args>` in a separate process, that is, outside of Leo itself. `leoserver -h` prints "leoserver.py -a <address> -p <port>"

 By default, the server listens on "localhost" and port 32125. You can change these defaults with the --address and --port command-line arguments.

Communication protocol: requests
 
The _do_message method expects that incoming requests are JSON objects that can be converted to a python dict, with the following three keys:

"id": A positive integer.

"action": A string, which is either:
  - The name of public method of this class, prefixed with a '!'.
  - The name of a leo command, without prefix, to be run by _do_leo_command.

"param": A dict to be passed to the called "action" method.

 

Communication protocol: requests

A simple copy & paste error: It should read 'response'.

The _make_response and _make_minimal_response methods return python dicts that are then converted do JSON. All responses contain an 'id' key that matches the corresponding requests.

Archived positions

Leo's Position objects can not be converted directly to JSON. Instead, requests and response contain archived positions, that are strings representing positions. The server's _ap_to_p and _p_to_ap methods convert between real Leo Positions and archived positions.

Technical details

The ws_handler function contains the main server loop. It calls Python's asyncio.get_event_loop() to start listening on the designated socket and port.

ws_handler dispatches incoming requests to the _do_request method, which then dispatches the requests to the appropriate handler, either in the LeoServer class, or (via Leo's bridge) to Leo itself.

 Exception handling

Request handlers raise ServerError exceptions for badly formatted user requests. The server returns an response describing the error.

The server raises InternalServerError for errors within the server itself. The server prints an error message and stops.

Request handlers raise TerminateServer to terminate the server normally.

Compatibility and future directions

The present version of leoserver.py exists to support leoInteg: Leo hosted on vs code.

In future, features (request handlers) can be added to leoserver.py, but features probably will never be removed.

Acknowledgements

Félix Malboeuf wrote the original version of leoserver.py including the first draft of the all-important The ws_handler function. Edward K. Ream helped make the code more pythonic, and provided Leo-specific advice.

With kind regards,

Viktor

[Edward K. Ream]

On Fri, May 14, 2021 at 2:39 PM Viktor Ransmayr <viktor....@gmail.com> wrote:

> Would it make sense to ignore mentioning leoclient.py at this point in time completely - or - at least defer mentioning it to the time, when the CP: request / response model has already been described?

I think it's worth mentioning that leoclient.py exists. It's a useful example of client code.

Edward

[tbp1...@gmail.com]

Looking at this and thinking about it a bit for the first time, I have the reaction that an HTTP server should respond to at least GET and POST requests.  The draft docs say nothing about this.  The server docs should at least explain how to send a request to the server. It should not be left to the sample client.  And if there is to be a web server, it would be useful for it to serve one or a few actual web pages, which could in fact contain the documentation for using it.  Tomcat does this by default, for example.

As an example of what I mean, the draft doc says

"The **_do_message** method expects that incoming requests are JSON object ..."

How is such a message to be sent?  Perhaps a GET?

GET: http://localhost:<port>/do_message?method=**&json={.....}

If this is not correct, explain what is correct, and give some examples.  If it is correct (and I am just guessing here), please give an example.  If the request is supposed to be a POST or PUT, make sure to specify what the message encoding is, as well as the format of the message body. 

Typically requests, especially POST requests, are encoded as MIME type application/x-www-form-urlencoded.  Is this the case for this server?  What is the MIME type of the response?  Is the utf-8 character encoding required?  Will the response always by utf-8 encoded?

And if there is to be a web server, security concerns need to be thought about from an early stage.  For example, can the server be made to leak important information by deliberately sending it a message that will cause an error? Can an attacker get to the file system or the server configuration files?  Can an attacker walk up the server's file tree?  Can the server be configured to respond only to a specific URL?

Is the server supposed to be stateful?  HTTP servers are not really expected to contain state.  They are supposed to return representations of a "resource".  If the resource is the state of a Leo outline, the server needs to query something that can provide that information.  This may be how the system is intended to work, but then the document should say so, and give some detail about how this all works.

[Edward K. Ream]

On Fri, May 14, 2021 at 11:39 PM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

Looking at this and thinking about it a bit for the first time, I have the reaction that an HTTP server should respond to at least GET and POST requests. 

Why do you say that?  leoserver.py does what it is supposed to do.

Edward

-----------------------------------------------------------------

Edward K. Ream: edre...@gmail.com

An old man, crazy about computer programming.

Leo Editor: http://leoeditor.com/

-----------------------------------------------------------------

[Viktor Ransmayr]

tbp1...@gmail.com schrieb am Samstag, 15. Mai 2021 um 06:39:53 UTC+2:

Looking at this and thinking about it a bit for the first time, I have the reaction that an HTTP server should respond to at least GET and POST requests.  The draft docs say nothing about this.  The server docs should at least explain how to send a request to the server. It should not be left to the sample client.  And if there is to be a web server, it would be useful for it to serve one or a few actual web pages, which could in fact contain the documentation for using it.  Tomcat does this by default, for example.

As an example of what I mean, the draft doc says

"The **_do_message** method expects that incoming requests are JSON object ..."

After reading the source a bit, it became clear to me, that it's not intended to offer an HTTP(s) [1] interface, but its providing a WS(S) [2] interface.

If I understand it correctly, this is at a much lower / generic level.

With kind regards,

Viktor

---

[1] https://datatracker.ietf.org/doc/html/rfc7231

[2] https://datatracker.ietf.org/doc/html/rfc6455

[Edward K. Ream]

On Sat, May 15, 2021 at 5:16 AM Viktor Ransmayr <viktor....@gmail.com> wrote:

After reading the source a bit, it became clear to me, that it's not intended to offer an HTTP(s) [1] interface, but its providing a WS(S) [2] interface.

If I understand it correctly, this is at a much lower / generic level.

I'm not sure how to characterize this server, except that it is the simplest web interface.

The ws_handler function uses Python's new (Python 3.4) asyncio module to do all the web-related stuff. Everything else supports Leo-related requests.

Imo, this architecture is perfect for the job. It is due entirely to Félix.

Edward

[tbp1...@gmail.com]

I'm not necessarily saying that it doesn't do what it's supposed to.  I'm saying that the draft doc doesn't say what it is supposed to do, at least not so an inexperienced user can do anything with it.  I'm saying that this draft doc doesn't tell the reader what and how it's doing those things.  OK, it says to run it with parameters.  But it doesn't say how to compose a message nor how to receive one, nor which of those **_do_message things are supported.  The other things I wrote were in the way of suggestions of features or behavior of the server.

Viktor wrote that the server isn't really intended to be an HTTP server at all.  Now that I've skimmed the Web Sockets RFC, I see a little better what's going on with respect to the "web server-ness" of the server. So let's have the doc say that it is a WS server per the RFC, and explain more about the messages so I would be able to actually construct, send, and reply to  one.  

[Edward K. Ream]

On Sat, May 15, 2021 at 8:06 AM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

I'm not necessarily saying that it doesn't do what it's supposed to.  I'm saying that the draft doc doesn't say what it is supposed to do, at least not so an inexperienced user can do anything with it. 

At present (I've just pushed some changes to the ekr-docs branch), the first two sentences of the new chapter are:

QQQ

leoserver.py is a stand-alone web server that provides access to Leo’s capabilities using Leo’s bridge.

leoserver.py exists to support leoInteg: Leo hosted on Visual Studio Code. 

QQQ

leoserver.py is not for inexperienced users. Imo, the above description does describe what the server does, and is supposed to do :-)

I'm saying that this draft doc doesn't tell the reader what and how it's doing those things.  OK, it says to run it with parameters.  But it doesn't say how to compose a message nor how to receive one, nor which of those **_do_message things are supported. 

For that one needs to look at the code. I don't think it's necessary to go into the details. The code should be easy enough to understand, for those who really want to know.

Let's let Félix comment on the docs before continuing this discussion.

Edward

[Alexander]

Because it was documented as not being part of core and I was not familiarized with the code of recent years, I was also initially confused by this file[1], residing in /external/leoserver/ -- an older development, which is a web (http) server.

This leoserver.py is a web socket server[2] (probably should have been explicit in the doc from the start), and is located in /core/ despite not being part of core, and isn't yet part of devel.

New implementations through this server are going to be interesting projects. For now, I suppose the key to understanding the payloads will be leoserver.py, leoclient.py[3], and currently implemented leoInteg[4].

[1]: https://github.com/leo-editor/leo-editor/blob/e46e9315f61a1c378900d3ba7fbadf39c5d9fb5b/leo/external/leoserver/leoserver.py

[2]: https://github.com/leo-editor/leo-editor/blob/c8fb3ce820cdf47b6b9a7e7e8cfb939c445ff22a/leo/core/leoserver.py

[3]: https://github.com/leo-editor/leo-editor/blob/c8fb3ce820cdf47b6b9a7e7e8cfb939c445ff22a/leo/core/leoclient.py

[4]: https://github.com/boltex/leointeg/blob/master/src/leoIntegration.ts

[Edward K. Ream]

On Sat, May 15, 2021 at 10:52 AM Alexander <al...@possente.net> wrote:

Thanks for these comments.

Because it was documented as not being part of core and I was not familiarized with the code of recent years, I was also initially confused by this file[1], residing in /external/leoserver/ -- an older development, which is a web (http) server.

leo/external/leoserver.py no longer exists.

This leoserver.py is a web socket server[2] (probably should have been explicit in the doc from the start), and is located in /core/ despite not being part of core, and isn't yet part of devel.

I've just added the word "socket" to the first sentence of the chapter.

New implementations through this server are going to be interesting projects.

Yes. One could imagine additions to the server that communicate with vim, or other editors or IDE's.

For now, I suppose the key to understanding the payloads will be leoserver.py, leoclient.py[3], and currently implemented leoInteg[4].

Exactly. Except for the main loop (ws_handler), the code is surprisingly easy. Incoming requests get dispatched to straightforward handlers.

Edward

[Viktor Ransmayr]

Hello Edward,

Edward K. Ream schrieb am Samstag, 15. Mai 2021 um 17:51:53 UTC+2:

On Sat, May 15, 2021 at 8:06 AM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

I'm not necessarily saying that it doesn't do what it's supposed to.  I'm saying that the draft doc doesn't say what it is supposed to do, at least not so an inexperienced user can do anything with it. 

At present (I've just pushed some changes to the ekr-docs branch), the first two sentences of the new chapter are:

QQQ

leoserver.py is a stand-alone web **socket** server that provides access to Leo’s capabilities using Leo’s bridge.

leoserver.py exists to support leoInteg: Leo hosted on Visual Studio Code. 

QQQ

leoserver.py is not for inexperienced users. Imo, the above description does describe what the server does, and is supposed to do :-)

This describes the current state of the Leo Server work a lot better than the initial version.

It does make it clear, that its focus at the moment is to support leoInteg - and - is not intended to address any other use cases yet ...

With kind regards,

Viktor

[Félix]

Hello everybody !

I'll chime in I guess ;)

Thank to everyone commenting and giving ideas and suggestions! That's how most of the features were born!

I'll make sure to revise the documentation before committing all the changes I'm making.

I'm still working on the new leoserver.py that Edward started (based on the old/original leobridgeserver.py) but with better global/class/methods hierarchy /encapsulation and architecture. (Thanks to Edward !)

To address Viktor's last point: Both the original, and the new one, are made to be totally agnostic of the client and are not made with leointeg exclusivity in mind at all. Any signs of that not being the case are/were accidental, and are being corrected. (it's true that sometimes the nomenclature of some json object members matched vscode terminology instead of Leo's. For example, in Leo we refer to the current cursor locations as the cursor 'insert' point. not the 'active' point like in vscode. This was accidental and not intended)

So yes, it may have been created at first to support leointeg, but it exists to support any and all other web-socket GUI client that may (or will) exist.  And that's why Edward and I think it should be part of Leo-Editor and not leointeg. 

So I'll keep working in this server cleanup & tune up for a couple more days : As soon as I'm done with this I'll commit it and push it to the leo-editor repo. (I'll also modify the client.py to match the modifications and run the tests that goes with it)

In the meantime, if you're curious, you can checkout the issue130 branch of leointeg and open the leo file to see the last node is a backup of my progress with the leoserver.py's main class, and matching changes to the old leobridgeserver.py . (I would have done this on a 'felix-server' branch of the leo-editor repo but their was a problem with the devel branch a few days ago and i just kept adding to my issue1130 branch of leointeg instead to keep an online backup for safety)

Oh - and i'm totally down to provide a 'http get/post' mode also if people are interested. ...the very first versions of the server didn't use websockets nor http, it was just using command line process pipes in/out. so minimal work should provide other ways of communication if needed. :)

--

Félix

[tbp1...@gmail.com]

I think that a basic HTTP capability could be useful.  For example, the server could report its version, status, documentation files, etc., in a browser-readable way for very little effort.  It could be ordered to shut down outside the normal Leo-based chain of command, which could be useful in event of a Leo crash.

[Félix]

Thanks tbp1, very good suggestion - why didnt I think of that? it can totally listen to both http and websockets simultaneously without problem  (altough maybe on different ports) and provide most commands, if not more as you suggested, through basic http requests.

Will do! 

--

Félix

[Viktor Ransmayr]

Hello Félix,

Félix schrieb am Samstag, 15. Mai 2021 um 21:52:25 UTC+2:

To address Viktor's last point: Both the original, and the new one, are made to be totally agnostic of the client and are not made with leointeg exclusivity in mind at all. Any signs of that not being the case are/were accidental, and are being corrected. (it's true that sometimes the nomenclature of some json object members matched vscode terminology instead of Leo's. For example, in Leo we refer to the current cursor locations as the cursor 'insert' point. not the 'active' point like in vscode. This was accidental and not intended)

So yes, it may have been created at first to support leointeg, but it exists to support any and all other web-socket GUI client that may (or will) exist.  And that's why Edward and I think it should be part of Leo-Editor and not leointeg.

Thanks a lot for this clarification & new information.

With kind regards,

Viktor

[Viktor Ransmayr]

Hello Edward,

Edward K. Ream schrieb am Samstag, 15. Mai 2021 um 18:01:12 UTC+2:

On Sat, May 15, 2021 at 10:52 AM Alexander <al...@possente.net> wrote:

Thanks for these comments.

Because it was documented as not being part of core and I was not familiarized with the code of recent years, I was also initially confused by this file[1], residing in /external/leoserver/ -- an older development, which is a web (http) server.

leo/external/leoserver.py no longer exists.

FYI: It's still available & visible in the 'devel' branch. - See

* https://github.com/leo-editor/leo-editor/tree/devel/leo/external/leoserver

What is the planned way going forward? - Will this one be deprecated - or - will they co-exist?

With kind regards,

Viktor

[Edward K. Ream]

On Mon, May 24, 2021 at 1:16 PM Viktor Ransmayr <viktor....@gmail.com> wrote:

> FYI: [leo/external/leoserver.py] is still available & visible in the 'devel' branch.

Ah. I see. This is an ancient file, unrelated to leo-editor/leoserver.py. It's a totally separate project, and probably of no great importance.

> What is the planned way going forward? - Will this one be deprecated - or - will they co-exist?

I think they might as well coexist, unless that causes confusion.

Edward

[Félix]

I might integrate a small http status/admin page, served on a  different port, like this to the websocket server - a nice little touch! :)

--

Félix