Non essential features are harmful (or not)

[vitalije]

This can be a long post that may or may not be of interest to readers. Please skip it if you find it uninteresting.

In another thread Edward wrote something that I would like to quote here just as an example of something that I notice throughout the Leo source code. It can be described as a design principle that IMO introduces unnecessary complexity everywhere.

We will also need a paste-as-xml command, so that people can paste the xml-version of an outline into an external editor. 

Personally I don't have anything against the xml format being used in clipboard.  I have made a prototype that uses json, but it's not important at all. I could have made a prototype that uses xml. Something other bothers me in the above sentence. Why is it important to allow user to paste outline in another editor? I have never done this and can't imagine for what purpose any user would want to do that. If someone needs to copy part of the outline and make out of it new Leo document, it would be just as easy to open new Leo file and paste outline there.

To repeat again, this is just an example of something that I notice throughout the Leo code. It would make no harm if it were the only exception, but it is present everywhere.

To make the discussion easier I will name this design principle "flexibility no matter what" (FNMW). 

Do I consider having flexibility to be good or not? I would certainly consider lack of flexibility when you need it, to be a misfortune. It is good to have flexibility when you need it. But what about the case when you have some extra flexibility that you don't use at all. If it doesn't cost you much it would be nice to have it just for the case you might need it one day. But what if it costs you dearly and you don't have a real need for it in the foreseeable future? It would be like you own thousand houses world-wide and you live in just one of them. But you pay taxes and bills for all other houses that you never use or use very rarely. You would in this case do much better for yourself if you throw away all those houses which bring you just costs and no benefits. If you don't make a good use of those houses they become just a burden.

The same logic should be applied to code design. If you need to achieve some flexibility in a certain part of code, then no effort should be spared to implement it. But if there is no evidence that the flexibility would be used, usually it will make code shorter, cleaner, easier to understand and maintain, and often faster to execute if you forget about that flexibility and make smallest viable product. Keep the energy for more important things.

I can point out to many examples of this FNMW design principle throughout the Leo code. Many modules, classes, methods and functions are designed as if they were public libraries used by many users for different use cases. But most of those modules are used only by Leo developers and in a very specific way. Those modules could be made much more readable, smaller, more efficient if they were tailored to a more specific task they need to perform and not as a general purpose tool that can be used for many other tasks. 

Without any intention to diminish any effort that recently went into making very flexible theme handling code, I still wander whether it was good thing to do. Let me explain a bit. I haven't looked at that code yet, and I have only skimmed through the theme related discussion. So, I may be very wrong about this. I just can't imagine that lots of users will see or use this flexibility. I guess that only one or two qt stylesheet experts will ever make small set of Leo themes that most of the other users will use just as they are without any customization. Basically qt stylesheets are just plain text files and Leo had already large number of tools for writing text files. Was it really the case that those stylesheets couldn't have been written using only standard Leo features for dealing with external files? If the theme customization in Leo were made to be extremely user-friendly comparing to other editors, I would consider all those efforts worthy. But let's be honest, customization of Leo appearance is not strongest Leo feature. There are tons of user complains about it. I can agree that Leo has a flexible system for writing style sheets, so it may be the tool of choice for some stylesheet designers out there. But I am not sure that even they needed all this flexibility. It seems to me very likely that the qt stylesheet designers would make those stylesheets just as easy using standard Leo features for external files. In the end there is a slight possibility that a few designers will use all the flexibility of Leo's theme related code, to produce ten different stylesheet files. But the rest of users will be afraid to touch any part of the themes and will just choose one of them to use as it is. I know that I am the one of the users afraid to touch the theme content. Last time I tried to customize Leo's appearance it took me several hours. 

Another example: unknownAttributes. In other thread we were seriously discussing possibilities that some user may have some exotic data in uA and what should we do to make this feature most flexible. But where is the evidence that any user would actually use that flexibility? I don't recall that I have ever used uA for any purpose. I can't imagine what would I put in uA and why? As I understand some plugins use this feature. Which plugins? How many of Leo's official plugins do use this feature? What do those plugins put in uA? Do we have ready answers to these questions? Have we ever made a list of this plugins and its uA needs? What if the list had just a few entries and all of them put just strings and numbers in uA? And let me remind you, we had an issue reported when someone used both python 2 and python 3 verisons on the same document that was caused by pickle version protocol. So, some users suffered because we used pickle module for handling uA. It may happen that nobody has ever gained any benefit from such flexibility and yet some of our users had suffered from it. I am not claiming that this is the case. I don't know who uses uA and in what way so I can't be certain about this. But until proven otherwise I doubt this flexibility is required. Given that supporting this feature makes code complex and ugly, I would vote for restricting it. If I am allowed to put a string in uA, then I can encode arbitrary data structure into a string. So I don't think that any user should feel uncomfortable regarding this restriction. Those few who really need some special data to be kept in uA, can encode/decode their own data as they want. But the rest of code should not be further complicated to support some feature that those who need can easy enough achieve even under such restrictions.

I remember the discussion about the book "Made to stick" in this forum several years ago. There is an example described in the book about one project manager who made a wooden block as a model of a handheld PIM device the project was about. Whenever a member of a team had some idea what to add to the device, this manager would take the wooden block in his hand and reply something like: "Great idea, but look there is no space for another button". There is also a rule that most artists honor: "The perfection is achieved not when there is nothing more to be added, but when there is nothing to be removed."

To summarize, I don't think that we should panic. Leo's code has been around for a long time and there is no immediate danger. Leo mostly works as expected. However, if we let FNMW principle to prevail we risk that one day Leo would support so many features that no-one would be able to remember or use just one tenth of them and Leo code complexity would exceed the ability of mightiest human mind to understand. It is already very hard to understand some parts of Leo even with the help of Leo's outlining capabilities.

It may be a good strategy to examine each Leo feature in terms of how often it is used and if the same effect can be achieved through some other features. The questioning should start with those features that introduce most complexity, and give minor value to the Leo. Each feature that complicates much and is used rarely or not at all, should be discarded and code simplified until someone shows up who really needs such feature.

When the code is simple, clean, easy to read and reason about, it is not hard to achieve any missing feature. But when the code is complex it may happen that a feature otherwise achievable, because the complexity of present code base, is labeled as an impossible idea.

Vitalije

[Edward K. Ream]

​​On Fri, Apr 27, 2018 at 7:04 AM, vitalije <vita...@gmail.com> wrote:

Why is it important to allow user to paste outline in another editor?

​Ok, I'll concede that it's not too important.
​

 

To make the discussion easier I will name this

​[dubious] ​

design principle "flexibility no matter what" (FNMW). 

​I don't have this as a conscious design principle.  Leo's commands are modeled after Emacs commands.  The key principle is: ignore the commands that don't interest you.
​

 

The same logic should be applied to code design. If you need to achieve some flexibility in a certain part of code, then no effort should be spared to implement it. But if there is no evidence that the flexibility would be used, usually it will make code shorter, cleaner, easier to understand and maintain, and often faster to execute if you forget about that flexibility and make smallest viable product. Keep the energy for more important things.

​I agree.  But none of Leo's kwargs have ever been added for theoretical flexibility.  They have always been added to satisfy a practical need.  Later, it sometimes happens that the need for a kwarg disappears and the kwarg becomes "vestigial".  Now that we have the clone-find commands, vestigial kwargs can (should!) be removed.

​

I can point out to many examples of this

​​

FNMW design principle throughout the Leo code. Many modules, classes, methods and functions are designed as if they were public libraries used by many users for different use cases. But most of those modules are used only by Leo developers and in a very specific way. Those modules could be made much more readable, smaller, more efficient if they were tailored to a more specific task they need to perform and not as a general purpose tool that can be used for many other tasks. 

​Really?  What modules, classes, methods and functions trouble you?

The actual design principle that I consciously follow all the time is that modules and classes should hide all details about an easily understood idea. In other words, Leo's design is about making modules and classes completely independent from each other.  That's what makes Leo stable.  To the first approximation, nothing else matters, and certainly not details about kwargs.
​

 

Without any intention to diminish any effort that recently went into making very flexible theme handling code, I still wander whether it was good thing to do. Let me explain a bit. I haven't looked at that code yet, and I have only skimmed through the theme related discussion. So, I may be very wrong about this.

​I think Terry and I would agree that you are, indeed, mistaken.  The present system didn't happen as a result of some imagined design principle.  It happened because stylesheets are built into Qt.

Yes, the code is complex, but it's complex for good reasons.  In particular, it must coexist with Leo's configuration code.

I just can't imagine that lots of users will see or use this flexibility.

​The new theming code hides most of this complexity, but we can't just require that everybody throw out their existing myLeoSettings.leo files.  We have to support more complicated ways indefinitely.​

 

Basically qt stylesheets are just plain text files and Leo had already large number of tools for writing text files. Was it really the case that those stylesheets couldn't have been written using only standard Leo features for dealing with external files?

​If we could travel back in time, the answer might be "no".  But now the answer is "yes".
​

 

Another example: unknownAttributes. In other thread we were seriously discussing possibilities that some user may have some exotic data in uA and what should we do to make this feature most flexible. But where is the evidence that any user would actually use that flexibility?

​Leo is a programming environment.  In principle, we don't know how people have already used uA's.​

 No way am I going to change uA's without an exceedingly strong reason for doing so.​

 

I don't recall that I have ever used uA for any purpose.

​It doesn't matter what you or I have done with uA's.  They are a fundamental part of Leo and they aren't going away.
​

 

"The perfection is achieved not when there is nothing more to be added, but when there is nothing to be removed."

​This is a great design principle. I use it all the time.​

 

we risk that one day Leo would support so many features that no-one would be able to remember or use just one tenth of them and Leo code complexity would exceed the ability of mightiest human mind to understand.

​We are already at that point.  However, this didn't happen because of ​

​

FNMW. It happened because Leo creates a new programming and data world, with almost limitless applications.

Nobody, including Leo's core devs, uses all the potential features.  But every single feature, without exception, exists because there was at least one user who made a strong case for it.

It is already very hard to understand some parts of Leo even with the help of Leo's outlining capabilities​.

​We could debate the degree to which this is true, but I don't see any way in general to mitigate this problem.  But there is no reason to despair.

For example, the recent keys-related work has significantly simplified what was (and still is) very complicated​ code.  That complexity is inherent in:

a) how Qt represents incoming key events and
b) how bindings are represented in settings files.

We can't change either a or b.  But that has not prevented me from simplifying the code.

​

It may be a good strategy to examine each Leo feature in terms of how often it is used and if the same effect can be achieved through some other features.

​Imo all of Leo's features deserve to remain.​  Some, like @root, are hidden, but they must remain indefinitely because people still use them.

Summary

My one and only "big" design principle is that modules and classes should hide all details concerning one "topic".  This is Glenford Meyers's great insight. It has worked well for decades.

Leo creates a new programming and data world, with almost limitless applications. That's what drives all of Leo's core code. All of Leo's features are the result of specific user requests.

Changes to Leo's code base can be considered only if they do not greatly impact Leonine code of which we have no knowledge. This includes uA's.

We devs have no choice but to live with the complexity of Leo's code, reducing that complexity incrementally when we can.  We must not sacrifice features merely to make Leo's code base simpler.

Edward

[Edward K. Ream]

On Friday, April 27, 2018 at 9:00:18 AM UTC-5, Edward K. Ream wrote:

> I still wander whether [the new theme code] was good thing to do.

Creating themes and using Qt stylesheets are much more difficult than expected because of serious deficiencies in Qt's stylesheet machinery.  In effect, Qt stylesheets do not reliably cascade as in css.

In part, this is due to Qt5 bugs, but I suspect other aspects are just broken. Alas, the Qt people probably can't change this now, because that would break existing Qt stylesheets.

Edward

[Terry Brown]

On Fri, 27 Apr 2018 09:00:16 -0500
"Edward K. Ream" <edre...@gmail.com> wrote:

> ​​On Fri, Apr 27, 2018 at 7:04 AM, vitalije <vita...@gmail.com>
> wrote:
>
> > Without any intention to diminish any effort that recently went
> > into
> > making very flexible theme handling code, I still wander whether it
> > was good thing to do. Let me explain a bit. I haven't looked at
> > that code yet, and I have only skimmed through the theme related
> > discussion. So, I may be very wrong about this.
>
> ​I think Terry and I would agree that you are, indeed, mistaken. The
> present system didn't happen as a result of some imagined design
> principle. It happened because stylesheets are built into Qt.

I haven't looked at the new code either, so I don't really have an
opinion. My earlier work really only provided @settings substitution
in stylesheets, a feature HTML CSS authors begged for for years, and an
optional tree based approach to writing the stylesheet, which became
obsolete when @data settings became treeish.

My feeling is that layering of code would be good, I've tried to think
about what the layers might be, something like

- node storage and mutation
- outline storage / representation / manipulation
- outline loading / session stuff

gotta go, haven't really finished the thought, but point is XML pasting
and other features useful to some user at some time don't cost too much
if they're at the right level.

Also I know you're (Vitalige) trying to make a general point and not
bog down in specifics, but p.v.u is heavily and broadly used, so the
answers to all the questions you raised need to be assembled to decide
what to do there.

Cheers -Terry

[vitalije]

Really?  What modules, classes, methods and functions trouble you?

The actual design principle that I consciously follow all the time is that modules and classes should hide all details about an easily understood idea. In other words, Leo's design is about making modules and classes completely independent from each other.  That's what makes Leo stable.  To the first approximation, nothing else matters, and certainly not details about kwargs.

It's not about kwargs, but yes they have made me think and try to describe and define what is it that make things worse than they need to be. 

If you really did follow the principle of hiding details and making classes and modules as much independent of each other many classes would look differently. For example one of the most basic element of Leo's data is certainly VNode. And yet one can not instantiate VNode class without acquiring full blown commander, which in VNode is known as context. In the comment of context attribute is explicitly declared that it is called context to indicate its limited usage. However this field is used not only as a holder of hiddenRootNode  but there is a method on VNode that knows about c.frame.body.wrapper and knows that c.frame.body.wrapper has setInsertPoint, setYScrollPosition...

It also knows indirectly about c.fileCommands.gnxDict.

Now, whichever module needs to access VNode class, can't just import leo.core.leoNodes and use it. It needs to acquire commander instance before.

Recently, we discussed the copying and pasting outline. It delegates its task to fileCommands. FileCommands OTOH knows about c.frame.resizePaneToRatio, c.selectPosition, c.frame.initialRatios, c.atFileCommands.readAll, c.frame.body.onBodyChanged...

It seems to me that copying and pasting outlines should belong just to VNode. I would expect that v instance knows best how to encode itself into a string, and to decode itself from string. There is no real need for v instance to know anything about c.fileCommands, c.atFileCommands, c.frame.body.wrapper ... You could say that fileCommands is designed to encode/decode vnodes. But why then it selects position, sets frame ratios ... If we want to extract encode/decode functionality from v in its own module, that's o.k. But extracted module in that case should be even lower level then vnode. It shouldn't know about any other functionality than VNode class. If fileCommands need to provide reading and saving Leo documents, then it would probably use the same low level helper module as Vnodes for encoding/decoding them from/to string. Additionally it may provide some gui related values like aspect ratios and selection. But it should not apply those values directly to c.frame.body.wrapper ... It should return those values to the commander and let the commander to apply those values (if it finds necessary to do so).

I could write many more examples but these few would suffice to make my point clear. 

​Leo is a programming environment.  In principle, we don't know how people have already used uA's.​

 ​

Exactly we don't know but we act as if we are certain that there are users who put in uA's some exotic data. IMO there was no need to introduce such level of flexibility before. Once upon a time there was no support for uA's at all. And then it was created based not on real user needs, but on the FNMW principle as flexible as possible no matter how expensive it may become. 

Do not misunderstand me. I really don't mind using pickle as a format. My point is just that we need to be aware of costs it incurs, and at least to think about whether we have to pay the full price or we can live with the less flexible one. 

Not a single thing that I have complained about is much important taken in isolation. I can live with any one of them. Only when I think about all of them together I see that they are real threat. 

I wrote this mostly because I feel sad about the fact that Leo has few brilliant ideas that every developer, designer, writer world wide would greatly benefit from if they learned about those brilliant ideas. Based on these few ideas Leo could become best and most wanted editor in the world. However, it is not likely to happen ever, because of all other features that we stick to so firmly and which make Leo not so user friendly. I may be wrong but that's how I feel. I hope that one day you may reconsider some of the design decisions.

Vitalije

PS: here are quotes from your last message that repeatedly demonstrate this subconscious principle

 In particular, it must coexist with Leo's configuration code.

 

 We have to support more complicated ways indefinitely.​

 No way am I going to change uA's without an exceedingly strong reason for doing so.​

 

 They are a fundamental part of Leo and they aren't going away.

 

 Some, like @root, are hidden, but they must remain indefinitely because people still use them.

 All of Leo's features are the result of specific user requests. 

 

This last one is debatable. For example the way that Leo handles settings was not what user wanted. In fact on more than one occasion users asked for something simpler and more familiar. The decision  that Leo should handle the settings the way it does can't possibly be motivated by user needs because no other tool handles settings this way. It was motivated only by the (IMO questionable) idea that we can use the same code for reading settings and for reading Leo documents. How many complications this single decision brought is hard to number. It is a complication for the end-users. It also significantly complicates the code. But, we (the users who have learned about at-others and clones), have to live with it. We can't give up on this few brilliant Leo ideas, so we have to live and find our ways to deal with all other obstacles that Leo imposes. 

[Terry Brown]

On Fri, 27 Apr 2018, vitalije wrote:

> as much independent of each other many classes would look differently. For example one
> of the most basic element of Leo's data is certainly VNode. And yet one can not
> instantiate VNode class without acquiring full blown commander, which in VNode is known

I think this is the kind of the thing I was alluding to with the levels
comment.

> I could write many more examples but these few would suffice to make my point clear. 
>
> ​Leo is a programming environment.  In principle, we don't know how people
> have already used uA's.​
>
> Exactly we don't know but we act as if we are certain that there are users who put in
> uA's some exotic data. IMO there was no need to introduce such level of flexibility
> before. Once upon a time there was no support for uA's at all. And then it was created
> based not on real user needs, but on the FNMW principle as flexible as possible no
> matter how expensive it may become. 

I think uA's are a weak example for your point. I think they're more used
and more useful than you perhaps realize. And because of that, the risk
of messing people up with changes is greater than you might think.

So the issue should probably be analyzed more carefully in another thread,
or here: https://github.com/leo-editor/leo-editor/issues/872, but given
their significance, the question "should we change them from Python dicts
to some other more limited mapping type" is not as simple as "json should
be sufficient".

> Do not misunderstand me. I really don't mind using pickle as a format. My point is just
> that we need to be aware of costs it incurs, and at least to think about whether we have
> to pay the full price or we can live with the less flexible one. 

I think we all agree Pickle was an unfortunate choice, and I suspect we
really will be able to switch to JSON, my point for this thread is just
that I don't think uA's are a good example of FNMW. To me they're a core
feature of Leo's data model.

Cheers -Terry

[vitalije]

I think uA's are a weak example for your point.

O.K. as I said before, it is not that one or the other data format bothers me. It is the reasons for making design decisions that I would like to shake a bit. I am afraid that there are lots of cases when the design decisions were made without exploring all possibilities and all consequences. I am not certain at all that uA's format should change. I would like to have this decision properly examined and backed by real data, not hypothetical. But not only this design decision, but many others too. This one just came from recent discussion, not because I think it is the worst problem imaginable. My point was not about one issue that I used as an example, but about the way we make or reject design choices. 

Finally, it seems that you know much more about uA's use cases. Have you ever put something in uA's that isn't json serializable?

Vitalije

[Terry Brown]

On Fri, 27 Apr 2018, vitalije wrote:

> Finally, it seems that you know much more about uA's use cases. Have you ever put
> something in uA's that isn't json serializable?

I want to try and analyze that more, maybe just by searching my drives for
.leo files and passing them through the JSON serialization code in the
leo_cloud plugin.

But the two cases I ran into where datetime.datetime in todo.py,
retrospectively a poor choice, it caused a boatload of trouble with Python
2/3 pickling issues, and the tags plugin wanted to serialize sets. Both
of these cases are fixed now.

I'd be surprised though if I've never used integer keys for example. And
maybe Python JSON allows those, even though they're not valid, I'm not
sure.

Cheers -Terry

[Edward K. Ream]

On Fri, Apr 27, 2018 at 1:53 PM, vitalije <vita...@gmail.com> wrote:

Really?  What modules, classes, methods and functions trouble you?

It's not about kwargs, but yes they have made me think and try to describe and define what is it that make things worse than they need to be. 

​I've responded in full detail in this new thread.

Edward

[Offray Vladimir Luna Cárdenas]

Thanks Vitalije, Edward and Terry for this discussion,

I can go into code details and so, maybe at that level your points are valid. From the user experience, I would say that using trees to setup themes is cumbersome (and preferences and to enable/disable plugins). Maybe there is and important principle there, but exposing Leo potential users to all this complexity (particularly the non-programmers) could result in a lot of incidental complexity. Something with check boxes and scrolling menus and options (like atom and tools alike) could help with settings and plugins use and discovery and may simplify underlying code.

This thread reminds me of the 20K lines of code principle for human understandability of software systems [1] and the Smalltalk culture of fighting actively against incidental complexity (see [3], for example). Of course this is not easy, but any discussion and action in this line should be visible and supported.

[1] https://news.ycombinator.com/item?id=3291085
[2] http://www.vpri.org/pdf/tr2011004_steps11.pdf
[3] http://cuis-smalltalk.org/

Cheers,

Offray

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To post to this group, send email to leo-e...@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

[Edward K. Ream]

On Sat, Apr 28, 2018 at 11:41 PM, Offray Vladimir Luna Cárdenas <off...@riseup.net> wrote:

I would say that using trees to setup themes is cumbersome (and preferences and to enable/disable plugins).

​Right. That's why ​we now have the theme .leo files in leo/themes. So now we have only one required setting: @string theme-name = <name of theme file, w/o .leo>

All the old machinery must remain so that people can won't have to change their myLeoSettings.leo files.

This thread reminds me of

​...​

the Smalltalk culture of fighting actively against incidental complexity (see [3], for example). Of course this is not easy, but any discussion and action in this line should be visible and supported.
[1] https://news.ycombinator.com/item?id=3291085
[2] http://www.vpri.org/pdf/tr2011004_steps11.pdf
[3] http://cuis-smalltalk.org/

​I couldn't agree more. There is a lot more to Leo than there was 25+ years ago, yet Leo's look and feel remain almost completely unchanged.  On the programming side, I continue to discourage any changes to Leo's API.

Edward