Leo's real weaknesses

[Edward K. Ream]

Recent criticisms of Leo and its architecture have been wildly off the mark.  The most ludicrous was the statement that 'c', 'g' and 'p' should have longer names.

Here, I'd like to discuss Leo's real weaknesses. Some have recently been exposed as I study pyzo's operation and code. I'll also explain how they arose and what might be done about them. With one exception, these have nothing whatever to do with Leo's classes or code.

Collaboration difficulties

Imo, this is Leo's main weakness in the world. It's not all that easy to share .leo files. Diffs don't work well with xml files. Converting from xml to (say) json, won't help.

Jupyter is far superior in this regard. The Jupyter project has had to handle the usual security considerations.  It appears that they have come to a reasonable approach. This is one reason why the "Jupyter in Leo" project seems important to me.

Weak or missing basic editor features

Pyzo shows just how much Leo's syntax coloring, code completion and debugging capabilities could be improved.  I intend to remedy these defects this year, basing my work on pyzo's brilliantly simple code.

Dubious decisions

Most of these happened long ago, for what seemed like good reasons at the time.

1. Using wrapper classes for text editing

The present Qt gui is, iirc, Leo's fourth gui.  Two preceded Leo in python.  Previously, the python version of Leo used the much weaker Tk gui.

The pyzo code shows just how big a price Leo has paid for the wrapper text abstraction layer.  Otoh, having a standard text widget greatly simplifies the curses gui plugin. Still, the text wrappers do complicate matters significantly.

2. Commands are defined in the Commands class

This class really should be called the Controller class, using MVC terminology.  Anyway, defining defining commands in the c class made sense initially.  Commands are clearly not parts of Leo's (data) model, nor are they part of a view.

If I were to do things all over again, I would likely split 'c' into 'c' and 'c.commands' components.  But that's not possible now because it would break existing code.  In my defense, the original (python) design happened long before Python had decorators.

In fact, it's not even possible to split leoCommands.py into separate modules. I've actually tried (twice) and given up quickly.  Mistakes would break Leo and there aren't nearly enough unit tests to catch those mistakes.

3. Underestimating regular expressions

Almost all of the g.skip or g.scan functions could be replaced by regular expressions. That would greatly simplify the code.  I haven't done this because there are bigger fish to fry.

The g.skip or g.scan functions could also be rewritten using regular expressions.  But this must be done safely.  I will reject any pull requests that do not contain strong unit tests that demonstrate complete compatibility with previous versions.

Outlines and clones complicate everything behind the scenes

This is unavoidable.  It's not a mistake and it's not the result of bad design or poor programming.  The flip side is that leo can do things no other IDE can do.

But this is still a weakness of Leo.  Leo's code can be complicated.  For example, supporting an in-Leo debugger requires Leo's goto-line-number code.  No big deal: that code is encapsulated.  But there remain off-by-one errors that have to be fixed.

So that's it.  I've confessed to you all of Leo's real weaknesses, as I see them.  Leo's behind-the-scenes complications do impact day-to-day work. Otoh, this "weakness" makes all of Leo's unique features possible.

But a balanced view would also mention Leo's successes.  The main design success is that it is always possible to revise the internals of any class without major effects outside that class.  For example, I'll be able to revise leoColorizer.py so it uses the brilliant pyzo syntax coloring code without having to worry about global impacts.

Finally, I'd like to say a few words about the supposed "haphazard" nature of software design. Imo, such complaints are akin to complaining that a novelist's characters seem to have lives of their own.  In fact, Leo's classes are driven by the tasks to be done.  New classes arise as new tasks arise. This is a completely natural process.  Leo's design means that new classes do not break old classes.

Edward

[Terry Brown]

1. Using wrapper classes for text editing

The present Qt gui is, iirc, Leo's fourth gui.  Two preceded Leo in python.  Previously, the python version of Leo used the much weaker Tk gui.

The pyzo code shows just how big a price Leo has paid for the wrapper text abstraction layer.  Otoh, having a standard text widget greatly simplifies the curses gui plugin. Still, the text wrappers do complicate matters significantly.

I wouldn't be in a huge rush to eliminate wrappers. I'm not sure if they're at the right level of abstraction, but a good wrapper level could be very valuable in moving towards three functional UIs, Qt, HTML/javascript, and curses.  Maybe there are idioms other than wrappers that would work equally well - more use of the leoServer model perhaps.

Cheers -Terry

[Edward K. Ream]

On Wed, Jan 11, 2017 at 12:31 PM, 'Terry Brown' via leo-editor <leo-e...@googlegroups.com> wrote:

​> I wouldn't be in a huge rush to eliminate wrappers.

No worries. They can't be eliminated now, because doing so could break existing code.

EKR

[Terry Brown]

---

From: Edward K. Ream <edre...@gmail.com>
To: leo-editor <leo-e...@googlegroups.com>
Sent: Wednesday, January 11, 2017 1:33 PM
Subject: Re: Leo's real weaknesses

I'm not sure if that can be a blanket statement.  I guess Leo doesn't have a defined boundary between declared API and internals subject to change - perhaps by default we've assumed the stable API is *everything*, that *no* changes should break existing code. But I'm sure that doesn't reflect reality, every time a helper function is eliminated or something like that there's a chance to break someone's code, somewhere.

To me there's an implicit line between API that must not change (p.insertAfter()) and internals that would be used by so few, and those few generally "developers", that the concern about breaking things is outweighed by the need to be able to refactor the code base as Leo evolves.  Unfortunately it would be a very fuzzy line.

I'm not really sure which side of the line the wrappers fall on.  Things like c.frame.body.wrapper.getInsertPoint are probably used in a lot of scripts, so maybe that should be considered API.

But what about things like the free_layout framework.  I'd like to remove it and replace it with Qt Docking, I've been working on that, not ready to release yet, or even close, but do we really need to worry about scripts that might be broken by such a change?  How many people would have interacted with that code?

Tricky.

Cheers -Terry

-- 

You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To post to this group, send email to leo-e...@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

[Edward K. Ream]

​​

​​On Wed, Jan 11, 2017 at 3:57 PM, 'Terry Brown' via leo-editor <leo-e...@googlegroups.com> wrote:

 

​> ​

I'm not sure if that can be a blanket statement.  I guess Leo doesn't have a defined boundary between declared API and internals subject to change - perhaps by default we've assumed the stable API is *everything*, that *no* changes should break existing code. But I'm sure that doesn't reflect reality, every time a helper function is eliminated or something like that there's a chance to break someone's code, somewhere.

​Yes, there is some wiggle room.  But not for the text wrappers ;-)

Happily, we are not stuck with using text wrappers in all situations, or any other part of the api.

​ ​

We can always invent new code.

For example, I am busy​ gutting Leo's syntax coloring code. The old code will go away when the pyzo switch is true in leoColorizer.py. I don't feel any constraints in doing so, except that c.recolor, c.recolor_now and c.outerUpdate methods should continue to work as before.

​> ​

But what about things like the free_layout framework.  I'd like to remove it and replace it with Qt Docking, I've been working on that, not ready to release yet, or even close, but do we really need to worry about scripts that might be broken by such a change?

​You are quite right.  There must be room to maneuver.

This is a judgement call.

I think we will both agree that we won't want to change methods that define user-visible commands.  Unless we are going to remove those commands ;-)

Iirc, there is no list of official ivars, except in one or two unit tests.  Again, we can agree the following will never be removed:

c, c.frame, c.frame.body, c.frame.body.wrapper and c.frame.body.wrapper.widget.  And a few others.

​> ​

How many people would have interacted with that code?

​That's the important question.  Whenever possible, we will probably want to leave old code in place, or make everything work as if the old code was in place. We want to extend API's, not change them.

​> ​

Tricky.

​Yes.  But there are clear ways forward.  We remain compatibility with common API's when possible.  Most importantly, we communicate the issues and our intentions to all who may be affected, and listen to their concerns.

Edward

[vitalije]

Converting from xml to (say) json, won't help.

IMHO it is not necessarily true. I don't suggest changing Leo document format from xml to json or any other format. But there can be found a scheme of encoding a Leo document in such a way that its changes can be expressed in a human readable diffs. That scheme can be just an additional format. Its purpose is not to replace xml.

First let us analyze what kind of changes can we made by editing any given Leo document.

1. we can edit content of any body, which means a standard plain text changes (adding a line, deleting a line or changing some part of a line). A standard diff tools are good in dealing with those kind of changes.
2. or we can rearrange nodes. A simple edit such as sorting children of one node, can make a very hard to understand diffs.
   

A few years ago I tried to combine Leo with fossil to make it possible to see how any particular Leo document evolved. A fossil is just one executable file which can be easily incorporated with the Leo itself, it doesn't need to be installed or configured to work.

I have made a script that encodes entire Leo tree into bunch of files one per each node named after gnx, and one special file named __LEOTREE__. The __LEOTREE__ file contents was just a list of lines that describe connections between nodes. Each line started with the level of the node it represents followed by its gnx. The node files were encoded with the simple scheme. Headline was a first line, followed by the body content of the node. The script would generate all those files in a temporary folder and then invoke a `fossil commit` there. Fossil would analyze content of the folder and record all changes.

The other script I have made would invoke fossil to print a timeline with all recorded versions, and offer a user to choose one of them. After user makes a choice that particular version would be extracted and content of the any given node that existed in that version would be displayed.

I didn't go any further in developing that idea mostly because I couldn't find a good way to show those choices to the user and to show different versions of a node. That was before a view rendered plugin appeared, (which I still don't use). I guess maybe it could be used to show diffs or different versions of a node.
Vitalije

[Edward K. Ream]

On Fri, Jan 13, 2017 at 4:53 AM, vitalije <vita...@gmail.com> wrote:

Converting from xml to (say) json, won't help.

IMHO it is not necessarily true. I don't suggest changing Leo document format from xml to json or any other format. But there can be found a scheme of encoding a Leo document in such a way that its changes can be expressed in a human readable diffs.

​This is very interesting.  Do you think your ideas/scripts can be adapted to git?

> I didn't go any further in developing that idea mostly because I couldn't find a good way to show those choices to the user and to show different versions of a node.

Yes, this a real problem. I'm not sure even perfect, high-level diffs do much good if the user can't use them easily.

Edward

[vitalije]

I suppose it can be done with the git also. But in that case user would need to have git installed and also git would need to recreate '.git' subfolder inside temporary folder every time. Fossil OTOH, keeps all its data in just one file which is in fact just sqlite3 database file. One can access its content with sqlite3 python module. Fossil itself is just one executable file about 1.5 - 2Mb which can be downloaded by a python script if it is not already available in the system path. Fossil executable contains also a standalone web server and can communicate and synchronize
data with remote repositories through HTTP or SSH.

Scripts do not depend that much on fossil. They just run fossil through subprocess module. All that is needed to replace fossil with git is to change command line arguments. I believe fossil and git have similar commands but they use different names.

If you are interested in those scripts I will try to find (or recreate) and share them.
Vitalije

[Edward K. Ream]

On Fri, Jan 13, 2017 at 2:22 PM, vitalije <vita...@gmail.com> wrote:

Fossil OTOH, keeps all its data in just one file which is in fact just sqlite3 database file.

​Interesting.

​> ​

If you are interested in those scripts I will try to find (or recreate) and share them.

​Of course I'm interested ;-)

EKR

[Offray Vladimir Luna Cárdenas]

Hi,

Some years ago I proposed something for Leo, combining two technologies: fossil and Yaml[1], for the same reasons exposed here: fossil a as easier, self contained technology for working with files and Yaml as a diff friendly format to express graphs (trees with clones). At that time the idea didn't advance, but I explored it in Grafoscopio using STON[2] instead of Yaml. Now I have a diff and human friendly format for Grafoscopio notebooks, that can express graphs. For example, the interactive notebook about Panama Papers at [3] is shown in a fossil repository at [4] and a diff with a previous version is on [5].

[1] http://markmail.org/search/?q=leo-editor%20offray%20yaml%20fossil#query:leo-editor%20offray%20yaml%20fossil+page:1+mid:t7xlg7mt5hqnldkn+state:results

[2] https://github.com/svenvc/ston/blob/master/ston-paper.md
[3] http://mutabit.com/offray/blog/en/entry/panama-papers-1
[4] http://mutabit.com/repos.fossil/panama-papers/artifact/cbfe8929edf64212
[5] http://mutabit.com/repos.fossil/panama-papers/fdiff?sbs=1&v1=255e79b046584a36&v2=cbfe8929edf64212

Some notes about my thesis that are in Leo have conflicts created by xml + dropbox, that made the XML unreadable from Leo again, as you can see in this fossil repo at [6]. I will try to fix the file to make it loadable again in Leo.

[6] http://mutabit.com/repos.fossil/doctorado-offray/dir?ci=ab8052282f0847de&name=Tesis

This shows how XML is really a big weakness for Leo and other programs and that other simpler approaches for a diff/human friendly format and technologies for storing complex documents and enabling collaborative projects are available, as the Grafoscopio prototypes shown, but, unfortunately, XML and git have monopolized the developer's imagination. But maybe now we can rethink some core assumptions about technologies and formats behind Leo.

I would try something like this:

- Storing a simple Leo tree as a Yaml tree, including clones and see how far can we push the format, for example, when interaction with external files is represented in that tree (@file, @clean @auto).
- Storing Leo and these external files in a single .fossil repository and explore if having this technology as a simple backed for storing files and their history makes easier to work with the changes in such external files. At least in my preliminary explorations, having fossil compiled with JSON support[7][8] has enabled to query, from Grafoscopio, the fossil repository with a simple interface (I have not made any POST experiment yet). Anyway, I could imagine a single .fossil file that makes Leo store a tree and related files, so if any of them changes, the changes are compared against the repository to ease collaboration.

[7] http://fossil.wanderinghorse.net/repos/fwiki/index.cgi/wiki/README
[8] https://docs.google.com/document/d/1fXViveNhDbiXgCuE7QDXQOKeFzf2qNUkBEgiUvoqFN4/view

Hope this helps,

Cheers,

Offray

[Offray Vladimir Luna Cárdenas]

Hi,

Nice to see how this discussion is moving forward. 

I share the idea of longer names. These 3 letters don't convey anything to me and I have to go always to the docs to remember what they mean (I think g if for gui, and c for commander and no idea about p, but I'm always unsure). That being said, I don't think is "bad style", is not just "my style" and I prefer longer evocative names, but also I think that Leo, as all artifacts, it's  a product of its context. Live coding and power tab completion features encourage longer names, but also the community and authors practices (as in Pharo/Smalltalk) and sometimes one looks his own code and thinks that the future self would write it, in a complete different fashion. At least that happens to me with Grafoscopio and is just reaching 3 years old, so I imagine how this can be amplified in a 20 years old project like Leo.

What is a big advantage of Leo is what it enables: this community and stuff you cannot do in any other software. This uniqueness is a profound source of inspiration, as I have said before, and some inspiration is found in the stuff that Leo doesn't have (live coding, simpler collaboration and file format or even longer names), which doesn't diminish in any way, Leo's history and community. Thanks for them.

Cheers,

Offray

On 11/01/17 08:02, Edward K. Ream wrote:

Recent criticisms of Leo and its architecture have been wildly off the mark.  The most ludicrous was the statement that 'c', 'g' and 'p' should have longer names.

Here, I'd like to discuss Leo's real weaknesses. Some have recently been exposed as I study pyzo's operation and code. I'll also explain how they arose and what might be done about them. With one exception, these have nothing whatever to do with Leo's classes or code.

Collaboration difficulties

Imo, this is Leo's main weakness in the world. It's not all that easy to share .leo files. Diffs don't work well with xml files. Converting from xml to (say) json, won't help.

Jupyter is far superior in this regard. The Jupyter project has had to handle the usual security considerations.  It appears that they have come to a reasonable approach. This is one reason why the "Jupyter in Leo" project seems important to me.

Weak or missing basic editor features

Pyzo shows just how much Leo's syntax coloring, code completion and debugging capabilities could be improved.  I intend to remedy these defects this year, basing my work on pyzo's brilliantly simple code.

Dubious decisions

Most of these happened long ago, for what seemed like good reasons at the time.

1. Using wrapper classes for text editing

The present Qt gui is, iirc, Leo's fourth gui.  Two preceded Leo in python.  Previously, the python version of Leo used the much weaker Tk gui.

The pyzo code shows just how big a price Leo has paid for the wrapper text abstraction layer.  Otoh, having a standard text widget greatly simplifies the curses gui plugin. Still, the text wrappers do complicate matters significantly.

2. Commands are defined in the Commands class

This class really should be called the Controller class, using MVC terminology.  Anyway, defining defining commands in the c class made sense initially.  Commands are clearly not parts of Leo's (data) model, nor are they part of a view.

If I were to do things all over again, I would likely split 'c' into 'c' and 'c.commands' components.  But that's not possible now because it would break existing code.  In my defense, the original (python) design happened long before Python had decorators.

In fact, it's not even possible to split leoCommands.py into separate modules. I've actually tried (twice) and given up quickly.  Mistakes would break Leo and there aren't nearly enough unit tests to catch those mistakes.

3. Underestimating regular expressions

Almost all of the g.skip or g.scan functions could be replaced by regular expressions. That would greatly simplify the code.  I haven't done this because there are bigger fish to fry.

The g.skip or g.scan functions could also be rewritten using regular expressions.  But this must be done safely.  I will reject any pull requests that do not contain strong unit tests that demonstrate complete compatibility with previous versions.

Outlines and clones complicate everything behind the scenes

This is unavoidable.  It's not a mistake and it's not the result of bad design or poor programming.  The flip side is that leo can do things no other IDE can do.

But this is still a weakness of Leo.  Leo's code can be complicated.  For example, supporting an in-Leo debugger requires Leo's goto-line-number code.  No big deal: that code is encapsulated.  But there remain off-by-one errors that have to be fixed.

So that's it.  I've confessed to you all of Leo's real weaknesses, as I see them.  Leo's behind-the-scenes complications do impact day-to-day work. Otoh, this "weakness" makes all of Leo's unique features possible.

But a balanced view would also mention Leo's successes.  The main design success is that it is always possible to revise the internals of any class without major effects outside those classes.  For example, I'll be able to revise leoColorizer.py so it uses the brilliant pyzo syntax coloring code without having to worry about global impacts.

Finally, I'd like to say a few words about the supposed "haphazard" nature of software design. Imo, such complaints are akin to complaining that a novelist's characters seem to have a life of their own.  In fact, Leo's classes are driven by the tasks to be done.  New classes arise as new task arise. This is a completely natural process.  Leo's design means that new classes do not break old classes.

Edward

[Jacob MacDonald]

As a fan of Fossil, I second the motion to perform the necessary archaeology :-).

--

[vitalije]

I couldn't find my earlier works but I have (re)created some script that can be used for experiments. In the attachment of this message is `fossil-leo.leo` file which contains one @button script.
Before running this script one should provide that fossil executable is somewhere in the path. Fossil can be downloaded from fossil-scm.org

The script performs two different actions depending on the switch in its root node.
The first action is to create a repository (if it doesn't exist), and then to put current content of the Leo outline in the repository. One should perform this action several times, changing the outline. Every time this script is executed new version is added (unless there are no changes).

After you create several versions you can change the switch in the root of script so that script performs another action. The other action is to retrieve content of any given node at the given time. Before performing this action one should adjust the arguments of the function call `fossil_cat_at_time` to reflect his/her own Leo document. Time should be after the first version, and the name of file should be existing gnx that has been already stored using the first action. Content of the given node is presented in the log pane named 'History'.

If you put the gnx which didn't exist at the given time, script would produce an error.

This is just for experimenting. Someone good with the Qt may find a better way to show versions and also to allow user to choose particular version.

function fossil_timeline can be used to retrieve names of the last 20 versions, or  with some tweaking any other list of versions. Every version has canonical name like hexadecimal literal and the time it was committed. This list can be parsed and displayed as a list, combo or scale widget where user can easily choose version. A gnx can be either a gnx of the current position or position that was selected when the widget was constructed.
Together gnx and version name (SHA1 hash) can be used by function `fossil_cat` to retrieve content of the node in specified version.

If this should become a plugin than perhaps it could also check whether there is fossil on system path or not and if it is not installed to retrieve correct version for the platform and install it somewhere in Leo folder and all fossil commands should be adjusted to use the downloaded executable.
HTH Vitalije

[vitalije]

FYI the above script will create a repository in current directory. If you execute the following command:

fossil ui leohistory.fossil

(you may need to provide a full path to leohistory.fossil)
it should open your default browser and show you the home page of the repository. There you can find timeline, diffs in different formats, content of each node, etc.
Vitalije

[Edward K. Ream]

On Fri, Jan 13, 2017 at 4:33 PM, Offray Vladimir Luna Cárdenas <off...@riseup.net> wrote:

Some years ago I proposed something for Leo, combining two technologies: fossil and Yaml[1], for the same reasons exposed here: fossil a as easier, self contained technology for working with files and Yaml as a diff friendly format to express graphs (trees with clones).

​[Big snip]

I'm still catching up on all the great recent ideas.  I'll be studying all these links as time permits.

EKR

[Edward K. Ream]

On Fri, Jan 13, 2017 at 4:59 PM, Offray Vladimir Luna Cárdenas <off...@riseup.net> wrote:

 

sometimes one looks his own code and thinks that the future self would write it, in a complete different fashion.

​That often happens with me.  But there is no way that I would consider renaming c, g and p in my own code. Or any other ubiquitous one-letter name. Imo, longer names are pure cruft.

But this is mostly a preference, and we never argue about preferences.  If you don't like c, g or p, define abbreviations that create alternatives that you like better:

  c;;=commander=c

  g;;=globals_module=g

  p;;=presently_selected_position=c.p

or:

  defs;;=commander,globals,present_pos=c,g,p

Edward

[Edward K. Ream]

On Sat, Jan 14, 2017 at 1:21 PM, vitalije <vita...@gmail.com> wrote:

I couldn't find my earlier works but I have (re)created some script that can be used for experiments. In the attachment of this message is `fossil-leo.leo` file which contains one @button script.

​Thanks for this.  I have put your documentation in the .leo file itself.

EKR

[Offray Vladimir Luna Cárdenas]

Hi,

Yes, the discussion about longer or shorter names is not fruitful. Is better to explore some changes, variations and ideas outside of Leo and see where makes sense bring them back here.

Once live coding become available on Leo, anyone could make something like:

from Leo import c as commander
from Leo import g as globals
from Leo import p as position_selected

or anything that works and suits personal preferences, so such distinctions become irrelevant.

Cheers,

Offray