At last I understand sockets and servers

[Edward K. Ream]

The biggest roadblock of my career has disappeared. At last I see that:

   Sockets, not servers, are the basis of IPC (inter-process communication)

   Servers exist because of sockets, not the other way around.

Getting this backward created no end of confusion. Everything seems clearer:

- Connecting two processes means creating a socket between them.

- Python's server (http.server in 3), asyncore and asynchat modules are just wrappers around sockets.

- The request/response and publish/subscribe messaging patterns are two ways of using sockets. 

- A server is required for every socket that uses the request/response pattern.

- curl creates traffic on a socket, which a server then may handle, or not.

History

1. All this became clear sometime between Friday morning and Saturday night, but I do not remember exactly when! It probably happened while talking to my brother Speed Friday afternoon.  I took these notes during that conversation:

- Sockets are about IPC that "just works".  Pipes and shared memory are alternatives on a single machine, but sockets are essential for the internet.

- Sockets are protocol agnostic. Sockets can use any protocol, including custom protocols.

- The internet is a network of socket servers.  If I had to guess, it was this statement that triggered the Ahas. I immediately told Speed that this statement changed my understanding of the internet.

2. Yesterday morning I created my first socket using Python 3, created a small Python server based on this python 2 code, and used curl to test that server. That was the first time I ever used curl.

3.  Yesterday afternoon I talked with Speed again, discussing the Ahas in detail. I am fairly confident that I have the big picture right.

4. We then discussed Leo as a web app. Speed suggested that we avoid doing things twice. Python servers have their limitations, so eventually we will want to use a server like Apache.  That being so, I may as well start using Apache now.  That eliminates duplicating server configuration work.

Summary

Sockets, not servers, are the foundation almost all IPC, including the internet. This page is a clear introduction to sockets.

Focusing on sockets ended massive confusion about clients and servers.

All internet software, including curl, handles traffic on sockets.

Leo as a web app will use a server that responds to events in the browser. The socket protocol will likely be a private protocol rather than http. The server will be something like Apache, not a python server.

All my old fears of internet-related technology are gone. Many thanks to Speed for his patient help. I eagerly look forward to lots of study.

All comments, suggestions for further study, and (especially!) corrections are welcome. 

Edward

[Cormac Williams]

<delurk>

Tap, tap, tap...Hi everyone,

Been  lurking here a while, relatively new Leo user after having bounced off it a few times over the last few years.

I would answer some of your questions with more questions, mainly.

1 : Do you want to hang this server on the internet at some point?

2 : What sort of scale do you want in terms of users?

3 : What sort of scale do you want in terms of Leo outline size?

4 : Do you want to get a database involved rather than serving from an xml file?

5 : I there a multi-user story to be told in terms of using an outline?

6 : How much are you prepared to refactor or change code to support the above visions?

Sockets and protocols are great, except if you want to hang it on the internet, in which case you are far better using tried and tested methodologies and frameworks. Actually, I'd say use tried and tested for even local service. A specialist protocol is only really required if you are doing something radically different from what has come before, which I get the feeling we are not talking about here.

Regards,

Cormac.

</delurk>

[Edward K. Ream]

On Sun, Oct 21, 2018 at 10:37 AM Cormac Williams <cormac....@gmail.com> wrote:

<delurk>

Hi.  Welcome to the discussion.

Sockets and protocols are great, except if you want to hang it on the internet, in which case you are far better using tried and tested methodologies and frameworks.

I didn't mean to imply that sockets are all there are to the internet.  Presumably a mature server like Apache will be required. 

 

Actually, I'd say use tried and tested for even local service. A specialist protocol is only really required if you are doing something radically different from what has come before, which I get the feeling we are not talking about here.

We can start by assuming ajax, using http.

Edward

[Offray Vladimir Luna Cárdenas]

Hi,

It's good to see the advances in the servers direction. The idea of a more decoupled Leo with web interfaces and the idea of "outlining" services to other systems and languages that some have proposed seems close.

I would avoid going into the Apache path and I would try to choose a minimalist pure python web framework (bottle, flask, other used by Jupyter) that can deal with the server/sockets core code functionality in Leo and leave the scaling part to a server neutral back-end (Apache, Cherokee, Lighttpd, Enginx, etc.)

When I was trying to implement ideas from Leo in Pharo/Smalltak I remember focusing on the "outlining experience" first. I wanted to be able to add/remove and move nodes that have a "header" and a "body" inside Pharo. Sometimes, when I think in this idea of "outlining" services using a web interface I think which is the minimal web library that provides me a tree interface, for example Inspire Tree[1], and how I would connect it to the tree representation I already have in Grafoscopio. Because I don't want to have a lot of moving parts in different languages and programming paradigms (something that already happens to Jupyter), I would like to create some socket/service that connects the web interface with the Pharo backend. So all the machinery behind trees is already there and the only "new thing" I'm implementing is a way to decouple the GUI so it can be a web app.

[1] http://www.inspire-tree.com/

Hopefully, something like that can be the path for services/sockets in Leo. I wonder how Leo-Vue can connect to Leo server, for example and what is the demo MVP for a web interface that connect with a Leo backend.

Cheers,

Offray

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To post to this group, send email to leo-e...@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

[Terry Brown]

> <https://www.w3schools.com/xml/ajax_xmlhttprequest_create.asp>, using
> http.

I wonder if Cormac was thinking of frameworks like Django, but perhaps
more appropriate (lighter) for Leo, Flask? Django and Flask have
sub-components that handle translation between URL and Python code, and
I think there're libraries which *just* do that. Flask's extra
features may or may not be useful. Also being aware of WSGI is probably
good, but perhaps still a level lower than you need to be.

Also mod_http.py plugin allows execution of arbitrary Python code in a
running Leo, so there's that to look at too.

I've seen setup using Apache at the very front of a web-app hosting
ecosystem, with nginx (another web server) behind that handling
requests to actual backend services in Python (maybe by WSGI, not
necessarily). At some level on all these systems though Python ends up
handling the request, so for development at least, you usually see
Python serving the web part.

Cheers -Terry

[Cormac Williams]

"Edward K. Ream" <edre...@gmail.com> wrote:

> We can start by assuming ajax
> <https://www.w3schools.com/xml/ajax_xmlhttprequest_create.asp>, using
> http.

Sounds a good way of communicating :-)

Terry wrote:

I wonder if Cormac was thinking of frameworks like Django, but perhaps
more appropriate (lighter) for Leo, Flask? 

Cheers -Terry

I was concentrating less on the actual technology to be used, but rather on the questions that I would ask before you start talking about the technology.

For example, I'm sure there are people out there serving leo files from FTP, cloud disks etc. Whilst technically that is client / server, effectively there is little additional benefit from doing so apart from files available at a distance.

On the other hand, there is the example of a web app like google docs spreadsheet, where multiple people can work on the sheet at the same time, it an be scaled to arbitrary sizes in terms of data or load. Code can be written to run server side as well as client side etc.

This thread was posted at the same time as I was exploring the leo code looking to answer the question :- "What is the smallest amount of changes necessary so as to read and write outlines from an ArangoDB database" :-)

Regards,

Cormac.

[Edward K. Ream]

On Sun, Oct 21, 2018 at 11:28 AM Terry Brown <terry...@gmail.com> wrote:

I wonder if Cormac was thinking of frameworks like Django, but perhaps
more appropriate (lighter) for Leo, Flask? 

All options are on the table, but see below.

Also mod_http.py plugin allows execution of arbitrary Python code in a
running Leo, so there's that to look at too.

Hmm.  Is Leo as a web app is feasible? Leo's scripting features make it extremely dangerous. If run on a server, Leo becomes an unverifiable cgi script. This looks like a gotcha.  Am I missing something?

Edward

[Edward K. Ream]

On Sunday, October 21, 2018 at 10:37:32 AM UTC-5, Cormac Williams wrote:

> I would answer some of your questions with more questions, mainly.

All these questions are reasonable.  Here are some preliminary answers.

> 1 : Do you want to hang this server on the internet at some point?

Not sure what you mean by this.  Servers may spawn a new thread for each incoming thread.  In any case, we can assume that we'll use a fully-scaled server like Apache.

> 2 : What sort of scale do you want in terms of users?

Imo, n == 1 is often going to be the limit, for security reasons.  Leo looks like an unverifiable cgi script to the server, which means one user (or small, trusted group of users) must be fully responsible for the damage Leo could cause.

It might be possible to host a Leo server in a per-user (or per-small group) virtual machine somewhere, but that's it.  I see no way to run a public, unsecured, Leo server.

> 3 : What sort of scale do you want in terms of Leo outline size?

The same scale as at present.  This means that loading a Leo view in a browser won't load the entire xml file, but will only load the parents of presently-selected node.  In other words, Leo will use ajax to display the visible part of the outline.

> 4 : Do you want to get a database involved rather than serving from an xml file?

Good question.

I recently thought about this in "plain" Leo.  There have been repeated requests for "node history".  One could imagine a DB containing old versions.  But I just recently saw that node history would not have commit messages.  Without those messages the "raw stream" of node contents is useless.  If the user wants node history, they should use git and be careful to write meaningful commit messages.  There is no way for Leo to provide those messages.

There is another reason for considering using a DB, namely to support cross-file clones.  The likely fatal problem with this is that the entire DB becomes a kind of cache.  External files must take precedence.  That being so, the DB adds a significant level of complexity, without fundamentally solving the problem.  This isn't likely to happen.

The third reason to use a DB would be to support huge outlines.  There may already be code in Leo to do this.  I don't remember.

> 5 : I there a multi-user story to be told in terms of using an outline?

See the answer to question 2.  The answer is yes, but only if all users bear full responsibility for the machine (virtual or not) on which Leo's server is run.

> 6 : How much are you prepared to refactor or change code to support the above visions?

I am willing to treat the project as essentially a new gui:  leo --gui=browser.

I will reject any suggestion to fork Leo's code base.  That would be the end of Leo, imo.

Edward

[Edward K. Ream]

On Sunday, October 21, 2018 at 11:01:57 AM UTC-5, Offray Vladimir Luna Cárdenas wrote:

It's good to see the advances in the servers direction. The idea of a more decoupled Leo with web interfaces and the idea of "outlining" services to other systems and languages that some have proposed seems close.

Interesting idea.  Providing "outlining services" seems like another project.

I would avoid going into the Apache path and I would try to choose a minimalist pure python web framework (bottle, flask, other used by Jupyter) that can deal with the server/sockets core code functionality in Leo and leave the scaling part to a server neutral back-end (Apache, Cherokee, Lighttpd, Enginx, etc.)

Sounds reasonable.  In another answer in this thread I said we could assume that Leo would use something like Apache (server), but I want to make clear that all options are on the table.

When I was trying to implement ideas from Leo in Pharo/Smalltak I remember focusing on the "outlining experience" first. I wanted to be able to add/remove and move nodes that have a "header" and a "body" inside Pharo. Sometimes, when I think in this idea of "outlining" services using a web interface I think which is the minimal web library that provides me a tree interface, for example Inspire Tree[1], and how I would connect it to the tree representation I already have in Grafoscopio. Because I don't want to have a lot of moving parts in different languages and programming paradigms (something that already happens to Jupyter), I would like to create some socket/service that connects the web interface with the Pharo backend. So all the machinery behind trees is already there and the only "new thing" I'm implementing is a way to decouple the GUI so it can be a web app.

[1] http://www.inspire-tree.com/

Hopefully, something like that can be the path for services/sockets in Leo. I wonder how Leo-Vue can connect to Leo server, for example and what is the demo MVP for a web interface that connect with a Leo backend.

I don't know enough to respond in detail just now.

Edward

[Edward K. Ream]

On Sunday, October 21, 2018 at 9:20:43 PM UTC-5, Edward K. Ream wrote:

On Sun, Oct 21, 2018 at 11:28 AM Terry Brown <terry...@gmail.com> wrote:

I wonder if Cormac was thinking of frameworks like Django, but perhaps
more appropriate (lighter) for Leo, Flask? 

All options are on the table, but see below.

Also mod_http.py plugin allows execution of arbitrary Python code in a
running Leo, so there's that to look at too.

Hmm.  Is Leo as a web app is feasible?

To answer my own question, the answer is yes, as discussed further in answer to in my reply to Cormac.  Imo, we must follow the Jupyter model, in which the server runs on the user's own (possibly virtual?) machine. It's time to study how Jupyter handles security issues.

Edward

[Terry Brown]

On Sun, 21 Oct 2018 21:20:33 -0500

"Edward K. Ream" <edre...@gmail.com> wrote:

> Hmm.  Is Leo as a web app is feasible? Leo's scripting features make
> it extremely dangerous. If run on a server, Leo becomes an
> unverifiable cgi script. This looks like a gotcha.  Am I missing
> something?

Sandboxing / encapsulation with Docker or some similar container system
seems like the easy way. Containers are much lighter than VMs, so you
should be able to run many sessions, depending on hardware etc.

Docker containers start fast enough if you're running something
long-lived like a session on leoBridge.py or something that runs for
the same length of time as a typical Leo session. I *think* they'd be
too slow to exist transiently and respond to single requests, although
maybe hot cache on a suitable machine could run that way. But I'm
guessing you're looking for something more persistent, anyway.

I just tried starting a Docker environment I have based on
conda/miniconda3 which is based on debian, and it starts in under a
second. It has tex-live / numpy / matplotlib (which pulls in a chunk
of Qt), so it's probably similar in size to something that would run
Leo.

Cheers -Terry

[Edward K. Ream]

On Mon, Oct 22, 2018 at 10:29 PM Terry Brown <terry...@gmail.com> wrote:

I just tried starting a Docker environment I have based on
conda/miniconda3 which is based on debian, and it starts in under a
second.  It has tex-live / numpy / matplotlib (which pulls in a chunk
of Qt), so it's probably similar in size to something that would run
Leo.

Thanks for this, Terry.  I haven't forgotten what you said.

Edward