I've just killed two issues

[Edward K. Ream]

I've just killed #1298 and #1301. #1298 suggests using a .ini file.  #1301 suggests replacing @string theme-name with an implicit setting in g.app.db.

Both were my ideas.  Both are truly wretched. Each would complicate how Leo handles settings. Each would confuse newbies and seasoned Leonistas alike.

Yes, there are problems with theme files.  Alas, we have to live with those problems. Yes, theme-related settings are inconsistent.  Alas, there are reasons for those inconsistencies.  Even if we could remove some inconsistencies, doing so would imply changing all existing theme files. I'm not going to do this to the authors of Leo's themes.

Yes, it would be better if theme files could load before myLeoSettings.leo.  But there is no good way to do that, and the present way does not, in fact, cause significant problems.  Changing the order in which Leo loads theme files might (will!) cause new headaches for all Leonistas.  It's not going to happen.

Edward

[Matt Wilkie]

It's too bad there's no obvious path forward. Maybe one will present itself eventually as we dwell here longer. Good that increased complication without clear gain has been avoided though!

-matt

[vitalije]

IMHO the real problem is the way Leo handles settings at the moment. I have suggested several times in the past to change this. 

The way I see it is this. Leo handles settings using the very same functionality that is used to handle its primary 

data - outlines. One might think it was a clever idea that saved some coding. But if we honestly and without prejudices, look at the result of this idea, we can see that it caused numerous complications in several other parts of Leo. Initialization code is an excellent example. But there is much more examples. One of the most extreme I can remember is that there is a call to c.endEditing very early before any gui element has any chance of being created. 

By the way this call is spread throughout the code base, contaminating and entangling all Leo's modules. I am not very convinced that it is still necessary because there had been many changes since it was introduced. It would be good to look how to eliminate the need for this call. But this should go to another topic.

Let's continue with the way Leo handles settings. Let's first split this idea in two parts:

1. editing settings as an ordinary outline
2. loading settings from the outline

The two parts are not necessarily bound to one another. They can be changed separately without too much trouble. While I am not very interested in changing the first, the second one IMHO should be changed in order to simplify other parts of Leo, and I am pretty sure the simplifications would be significant.

If we ask users both newbies and seasoned are they satisfied with the way Leo handles settings, I guess we would find many more unsatisfied users than the satisfied ones. I remember loosing many hours on more than one occasion while trying to solve some settings issue. I am afraid of touching any settings because of these bad memories.

Many things are impossible or prohibited because of this. There is no way user can load and unload plugins without restart. Python is dynamic language and there should be no problem to allow this. In almost all other editors plugins are downloaded from the internet, installed and switched on and off with the single click. Leo users can't possibly dream about such a feature mainly because of the way Leo handles its settings.

Switching the theme is impossible without restart.

What can we do about it?

After editing settings in the myLeoSettings.leo or any other settings outline, user should invoke a command to save settings. This command can be executed automatically on each save. This command would read settings from the outline and build settings dict as it is now done during the startup code. The result set of settings can be stored in g.app.db for the next run. During the startup settings can be retrieved from the g.app.db in one line. This would require the least changes in Leo's code. However there might be even a better scheme. Store each setting in the db table with the following columns:

(name, level, filename, type, value) where primary key is (name, level, filename).

Then c.config.get<type> can be changed to execute simple query from the database selecting the value at the highest level for this outline. Leo's defaults defined in leoSettings.leo should have level=0, myLeoSettings.leo - level=1 and all other outlines should have level=2. 

I remember an experiment to test whether accessing settings from db would slow down Leo and result was that sqlite3 database would outperform Leo's present config machinery. 

Now, adjusting  any setting would be as easy as executing a simple SQL command. Calling afterwards c.reloadSettings should apply new setting value immediately without any trouble. 

There will be no problems related to what is read before and what is read after because already compiled set of settings will be present from the early start. And if a user wishes to change some setting it can be done at any moment, and the result of this change will be preserved for the next run too.

Summary

IMHO your idea of having global settings is not bad at all. The only thing I would change is not keeping it in an .ini file, but instead in the sqlite3 database, so we get parsing, querying, persistence for free.

This idea conflicts with the way Leo handles settings now. To resolve the conflict, you should change the way Leo handles settings now and it would simplify many parts of Leo's code base.

Vitalije

PS: I would gladly help with the necessary changes.

[vitalije]

On Saturday, August 24, 2019 at 2:46:02 PM UTC+2, Matt Wilkie wrote:

It's too bad there's no obvious path forward. Maybe one will present itself eventually as we dwell here longer. Good that increased complication without clear gain has been avoided though!

-matt

 

If you ask me, there is an obvious path forward. The question is whether Edward will like it or not :-).

Vitalije.

[Matt Wilkie]

Clearly you've devoted some concentrated thought to the challenge Vitalije. I love the idea of setting levels 0,1,2,... and the setting being able to report on it's level. Is there anything a person of small dev skills like myself can do to help prove/isprove the approach is sound and beneficial?

-matt

[vitalije]

 

Is there anything a person of small dev skills like myself can do to help prove/isprove the approach is sound and beneficial?

-matt

If you have any questions about my idea I would be glad to explain it in more details. That could be a great help to make my idea clear. It is clear in my mind, but I am not sure if I explained it well. Sometimes I feel I should write more detailed explanation. But sometimes too many details can make things less clear. Questions of careful reader are likely to help in achieving an optimal ratio between details and the whole picture.

Vitalije 

[Edward K. Ream]

On Sat, Aug 24, 2019 at 8:59 AM vitalije <vita...@gmail.com> wrote:

> If you ask me, there is an obvious path forward. The question is whether Edward will like it or not :-).

I am working on a long reply to your original proposal.  I'll explain why I think there are problems with the proposal.  I would like to make it easier for you, and all other important devs, to move Leo forward after I am gone.

It may take a day or two more before the long reply is ready.

Edward

[Brian Theado]

Vitalije,

On Sat, Aug 24, 2019 at 9:54 AM vitalije <vita...@gmail.com> wrote:

[...]

After editing settings in the myLeoSettings.leo or any other settings outline, user should invoke a command to save settings. This command can be executed automatically on each save. This command would read settings from the outline and build settings dict as it is now done during the startup code. The result set of settings can be stored in g.app.db for the next run. During the startup settings can be retrieved from the g.app.db in one line. This would require the least changes in Leo's code. However there might be even a better scheme. Store each setting in the db table with the following columns:

(name, level, filename, type, value) where primary key is (name, level, filename).

Then c.config.get<type> can be changed to execute simple query from the database selecting the value at the highest level for this outline. Leo's defaults defined in leoSettings.leo should have level=0, myLeoSettings.leo - level=1 and all other outlines should have level=2. 

I remember an experiment to test whether accessing settings from db would slow down Leo and result was that sqlite3 database would outperform Leo's present config machinery. 

Do you envision distributing the leoSettings.leo derived database with leo itself? Or do you have something else in mind? I was wondering how that first database of settings would be available at the time of leo first use.

Brian

[Matt Wilkie]

After editing settings in the myLeoSettings.leo or any other settings outline, user should invoke a command to save settings. This command can be executed automatically on each save. This command would read settings from the outline and build settings dict as it is now done during the startup code. The result set of settings can be stored in g.app.db for the next run. During the startup settings can be retrieved from the g.app.db in one line.

So, essentially take the live settings after all the startup processes have completed and save that state on disk somewhere? With g.app.db being the "somewhere"?

I guess this is fine if we trust that g.app.db is never changed outside of the "save settings" command. Otherwise, as I'm conceiving of it, this scheme would then also need a mechanism to compare the saved state with the settings .leo files, which would blow any performance gains, and add complications of what to do when sometthing differs.

Hang on, I've just described a cache. I don't think that is what you're pointing towards, because that wouldn't make finding and changing the setting currently of interest any easier.

matt

[Matt Wilkie]

Hang on, I've just described a cache. I don't think that is what you're pointing towards, because that wouldn't make finding and changing the setting currently of interest any easier.

Oh but it could! If each cached setting had a breadcrumb trail back to where it was set.

[vitalije]

Do you envision distributing the leoSettings.leo derived database with leo itself? Or do you have something else in mind? I was wondering how that first database of settings would be available at the time of leo first use.

Leo defaults contained in leoSettings.leo should be distributed in their "pre-compiled" form. For example it could be one python module containing one big tuple of tuples representing the initial content of db. On start if this db doesn't contain table settings, table will be created and populated with the default values from this python module. This action is executed only once and it takes negligible time. For new users the result would be exactly the same as if they installed Leo and run it for the first time before they created myLeoSettings.leo.

For users that have already created myLeoSettings.leo, we can allow some grace time period when Leo starts as usual (reading all settings files), and after start Leo can just store all processed and collected settings in the settings db. On the next run Leo would use db.

Then in the next phase when all users have their settings db created, we can start cleaning the startup code and remove unnecessary code parts. 

That should cover most of the users. Users who install only release editions, will have in the next release both startup routes (one reading leo settings files, and one using settings db). On the first run the old startup route will read and collect all settings and after startup those settings are written to settings db. On every next run when settings db is present, Leo will use the second route. In the next release first route can be removed.

For users who use git version, the situation will be similar only they would sooner get their settings db created.

Almost none of the Leo users would have to do anything special to switch to new settings code. The only case when some action is required from users is if they switch from the current release to the release when startup code has been cleaned and Leo will miss their myLeoSettings.leo. But in this case they would simply solve the issue by opening their myLeoSettings.leo and saving it.

Otherwise, as I'm conceiving of it, this scheme would then also need a mechanism to compare the saved state with the settings .leo files, 

Theoretically speaking yes. But in the reality g.app.db is never changed outside Leo (or at least we can say it is not supposed to be changed outside Leo). If someone changes g.app.db outside Leo, well I don't think that Leo developers should be concerned at all. If someone deletes g.app.db (sometimes it is suggested to do so), user should afterwards open myLeoSettings.leo and save it again, and db will be fully recreated.

The worse possible case that can happen is that on the next run Leo use only default settings ignoring myLeoSettings.leo, and in this case user just need to open myLeoSettings.leo and save it.

Vitalije

[Brian Theado]

Vitalije,

On Mon, Aug 26, 2019 at 3:42 AM vitalije <vita...@gmail.com> wrote:

Do you envision distributing the leoSettings.leo derived database with leo itself? Or do you have something else in mind? I was wondering how that first database of settings would be available at the time of leo first use.

Leo defaults contained in leoSettings.leo should be distributed in their "pre-compiled" form. For example it could be one python module containing one big tuple of tuples representing the initial content of db. On start if this db doesn't contain table settings, table will be created and populated with the default values from this python module. This action is executed only once and it takes negligible time. For new users the result would be exactly the same as if they installed Leo and run it for the first time before they created myLeoSettings.leo.

Sounds reasonable and it would probably also be possible to have a "compile-settings" command line option for leo which could be used in place of the "open myLeoSettings.leo and save it again" you mentioned.

About 10 years ago I worked on some server software we wrote in C. The configuration was controlled via a complicated set of sqlite tables. They were complicated because it was a structure which made it easier for the end-user configuration UI. We decided to have the C code initialization structure be as simple as possible. We kept the complexity of transforming the complicated structure to the simple structure far away from the harder-to-change C code. This decision was very helpful. It resulted in fewer changes to the C startup code and the changes we couldn't avoid were made on simpler code. The simpler structure would have been onerous for the user to construct, but the data transformation took care of hiding that difficulty from the user. Having the transformation separate from the server code, it also made it much easier to troubleshoot.

I see the leo situation as somewhat analogous. The settings-in-outlines is easier for the user to deal with just like the complicated sqlite tables were easier for our configuration users, and the settings db you propose is easier for the code to deal with just as our simplified C structure was for our server software. Data transformations can be useful!

Brian

[Edward K. Ream]

On Sat, Aug 24, 2019 at 8:54 AM vitalije <vita...@gmail.com> wrote:

IMHO the real problem is the way Leo handles settings at the moment. I have suggested several times in the past to change this. 

I do not at all agree with this proposal. First, I'll make specific comments, line by line. Later, I'll discuss what I think Leo's future project leaders would be better off thinking about.

The way I see it is this. Leo handles settings using the very same functionality that is used to handle its primary data - outlines.

Having setting be outlines has many advantages, including the various @button scripts in leoSettings.leo.

 

One might think it was a clever idea that saved some coding.

It's not just a clever idea.  The effect on coding is, from a design point of view, irrelevant.  The question we must ask, first, is whether such settings benefit users.

But if we honestly and without prejudices, look at the result of this idea, we can see that it caused numerous complications in several other parts of Leo. Initialization code is an excellent example.

That may be true, but there is no way I would allow major changes in settings now.  Doing so would unnecessary inconvenience Leo's users.

But there is much more examples. One of the most extreme I can remember is that there is a call to c.endEditing very early before any gui element has any chance of being created. 

Imo, c.endEditing does something reasonable, regardless of the behind-the-scenes complications.

By the way this call is spread throughout the code base, contaminating and entangling all Leo's modules. I am not very convinced that it is still necessary because there had been many changes since it was introduced. It would be good to look how to eliminate the need for this call. But this should go to another topic.

I agree. This belongs a separate discussion. However, as I note below, Leo's devs have much more important things to worry about.

Let's continue with the way Leo handles settings. Let's first split this idea in two parts:

1. editing settings as an ordinary outline
2. loading settings from the outline

The two parts are not necessarily bound to one another. They can be changed separately without too much trouble. While I am not very interested in changing the first, the second one IMHO should be changed in order to simplify other parts of Leo, and I am pretty sure the simplifications would be significant.

I am definitely not going to change the user's view of settings.  We might consider adding to settings outlines, but I have no interest in doing so.

If we ask users both newbies and seasoned are they satisfied with the way Leo handles settings, I guess we would find many more unsatisfied users than the satisfied ones.

Maybe.  I don't know any way to do a valid survey.

I remember loosing many hours on more than one occasion while trying to solve some settings issue. I am afraid of touching any settings because of these bad memories.

This section in Leo's docs says:

QQQ

This search order offers almost too much flexibility. This can be confusing, even for power users. It’s important to choose the “simplest configuration scheme that could possibly work”. Something like:

- Use a single leoSettings.leo file for installation-wide defaults.
- Use a single myLeoSettings.leo files for personal defaults.
- Use local settings sparingly.

QQQ

Imo, this scheme should work for most users.  Leo now has several tools that allow users to see where settings are coming from.

Many things are impossible or prohibited because of this. There is no way user can load and unload plugins without restart. Python is dynamic language and there should be no problem to allow this.

The problems with unloading plugins do not involve settings.  Plugins can modify many aspects of Leo.  Undoing such general actions can be difficult to impossible.

LeoPluginsController.unloadOnePlugin is supposed to unload a plugin.  How well it works is another matter. The issues are similar to Python's imp.reload.

 

In almost all other editors plugins are downloaded from the internet, installed and switched on and off with the single click. Leo users can't possibly dream about such a feature mainly because of the way Leo handles its settings.

Again, I disagree.

 

Switching the theme is impossible without restart.

Maybe, but again Leo's settings really are not the issue.

 

What can we do about it?

After editing settings in the myLeoSettings.leo or any other settings outline, user should invoke a command to save settings. This command can be executed automatically on each save. This command would read settings from the outline and build settings dict as it is now done during the startup code.

Leo pretty much works this way now.

The result set of settings can be stored in g.app.db for the next run. During the startup settings can be retrieved from the g.app.db in one line. This would require the least changes in Leo's code.

This introduces a secondary caching scheme, in addition to Leo's present primary settings code. There is absolutely no way this scheme can simplify Leo's code.

Caching creates the potential for nearly limitless confusion. The real value of your excellent fast @file read code was that it completely eliminated file caching.  Why you think that caching settings would be more benign?

 

However there might be even a better scheme.

I'm not going to comment on the organization of the proposed caching scheme. Caching settings is a truly terrible idea.

Now, adjusting  any setting would be as easy as executing a simple SQL command. Calling afterwards c.reloadSettings should apply new setting value immediately without any trouble.

At present, adjusting any setting is a matter of changing the setting, and then reloading all settings. 

Summary

IMHO your idea of having global settings is not bad at all.

I strenuously disagree.  Leo has no real need for global settings, and it would be foolish to complicate Leo's settings machinery further.

The only thing I would change is not keeping it in an .ini file, but instead in the sqlite3 database, so we get parsing, querying, persistence for free.

This idea conflicts with the way Leo handles settings now. To resolve the conflict, you should change the way Leo handles settings now and it would simplify many parts of Leo's code base.

No way am I going to do this.

That's the end of my specific comments.

The bigger picture

Any future project leader must consider two somewhat contradictory goals.  They would naturally want to add important new capabilities to Leo.  Otoh, they must maintain stability if at all possible.

Principle 1: First, do no harm.

Devs must be aware of the dangers of any significant change to Leo's code base.

Principle 2: Put Leo's users first.

Never needlessly inconvenience Leo's users. Never change any aspect of Leo without the strongest reasons. Respect their investments in settings, scripts, and learning.

Principle 3: Don't bother criticizing Leo's code base.

Leo's code base isn't perfect, but it is not an obstacle to innovation.

By far the safest approach is to minimize changes.  You don't have to debug or document code that hasn't changed. Some recent examples:

- The discovery of how to init pyzo in a sytematic manner.

- This week's discovery of how to add global docks with virtually no changes to Leo's code.

These discoveries where worth almost any amount of work, because they showed how to use existing code bases elegantly.

Corollary:  Study and experiment with existing code a long time before contemplating major changes.

Corollary: Don't waste time criticizing code bases.

The console gui required virtually no changes, either to Leo's code base or to the npyscreen code.  Similarly, thinking about changing pyzo, black or pylint would be a distraction.

Principle 4: Confusion and bugs waste more time than anything else.

When users (or devs) are confused, the effects lasts for minutes, hours, days, months or years.  Coding efficiencies typically save from microseconds to seconds.

Corollary: Never introduce needless caching. 

Summary

My biggest concern is that future devs will have an insufficient sense of danger. 

Leo's future project managers must never inconvenience or confuse existing users.

We haven't finished exploring the Leonine world.  Let's not distract ourselves with small complaints about Leo's code base.

Imo, messing with Leo's settings is a complete waste of time.

Edward

[Matt Wilkie]

Whoa. Edward, the strength of your reaction seems out of proportion.

matt

[vitalije]

Dear Edward,

I feel like you haven't read the whole thread before writing your reply. I have repeatedly written that user will notice nothing and yet your reply is full of warnings and implications as if I were not aware of the danger to cause inconvenience to Leo users.

I can see that your idea of using Leo's reading outline code twice both for reading outlines and for reading settings is too dear to you and you are not going to let it go. As I said before it might seem smart, but it heavily complicates Leo and it does prevent innovation and it does causes frustrating bugs. However I can't prove this claim because of your adamant refusal of my idea.

You have pointed me to the page from the documentation that warns users about possible confusion when setting their preferences. I assure you that on more than one occasion I have spent hours trying to fix problem with my settings that was caused just by updating Leo, and I have always had only one myLeoSettings.leo. The problems were not that I couldn't find which of the settings should I change. Those were real Leo bugs, however it took some time to distinguish between bugs and settings confusions. There are still problems that I just live with. For example if you have defined anywhere any of @font settings for syntax coloring, the relevant part of code will not resize with Ctrl+wheel zooming in and out. 

Right now when I open myLeoSettings.leo this is what happen: 

1. first open new empty Leo, this is how it looks like

2. and after using menu command Settings/myLeoSettings here is how it looks like

notice how small menus and buttons become; almost invisible

3. and when I close myLeoSettings.leo this is how all other tabs look like:

And there is no way I can return to my preference theme. I have to restart Leo. 

No doubt, the problem is in my myLeoSettings.leo. But I haven't created myLeoSettings.leo out of thin air. I used it in accordance to what was proposed as best practice for years. And on one day, after updating Leo, I as a user was required to adopt a new scheme. So I have chosen the BreezeDarkTheme for my preference and now I can't edit myLeoSettings without loosing my preferred look and feel. 

Another example quite recent is introducing new dock layout. As you can see from the previous screenshots my preference is to have outline in the top left and log underneath, and body on the right. Recently when I updated Leo, docks were introduced and I couldn't find the way to layout the docks to look like the screenshot above. So, I had to change all my Leo launchers to add --no-dock argument. 

My point is, you have many times introduced changes that required users to change their settings or to give up updating Leo. You might not be aware of this but it is reality. That makes your argument for refusing my idea even less relevant. If you read my proposition more carefully you might learn that I haven't planned nor suggested to cause any harm nor inconvenience to users, and therefore your only argument against the idea totally misses the point. I see no other arguments against my proposal except that you prefer your idea of using Leo reading code twice.

Oh, yes. You have argued that introducing a cache potentially can cause some confusion. Well I can agree that it is a possibility. But Leo already uses cache. When it needs a value of some setting it doesn't read all those outlines again and doesn't parse them again to retrieve the value, but instead it retrieves the value from the cache that was built during the startup. The difference is when this cache is built. In my idea it is built (and updated too) whenever user changes any setting, and not during the startup. That is why if implemented my idea will help to make things easier for users and allow them to see the effect of their changes immediately and not after the next reload. It would remove a lot of confusions that Leo at the moment imposes on users.

I am not a big fan of this kind of discussion when you just proclaim that something you won't allow and period. I would much prefer if we discuss the idea in terms of what are the pros and cons, and how should we measure and compare the values of each idea. My intention is not to impose my idea at any cost. I think this idea is a good one. It has several pros and all the cons you have pointed, I just don't see relevant. 

• It will not bring any inconvenience to users,
• it will not remove any of Leo features,
• it is not introducing a new caching mechanism because cache is already there. It just changes the time when this cache is built.

The only reason to dismiss it, that I can still see, is that by accepting this idea you'd have to sacrifice one of your own, dear ideas and that is hard. I can understand that, because sometimes it happens to me too. However, when it happens to me I am not very proud of it, and usually when I become aware that I am too attached to some idea, I try to analyze it with the cooler head just by comparing the real pros and real cons and to reduce my own bias.

If you don't want to discuss this idea any further, I won't insist either. Let's move on to something else. But if you ever wish to discuss it again, I'll be glad.

Vitalije

[Brian Theado]

Vitalije,

Could you explain more on how using Leo's reading code for both reading outlines and reading settings is an obstacle to fixing some of your complaints (i.e. switching a theme can't be done without restarting leo)?

Your idea mentions "compiling" the settings to a simpler format at time of save. How does that help compared to "compiling" the settings to a simpler format during leo init? And isn't leo's reading of settings from outlines into some internal data structure also a form of "compiling" the settings to an alternate format? Is it that Leo's internal configuration structure doesn't match what you want or is the the reading from outlines alone which causes your complaint? Or both?

Brian

[Edward K. Ream]

On Tue, Aug 27, 2019 at 2:52 AM vitalije <vita...@gmail.com> wrote:

I can see that your idea of using Leo's reading outline code twice both for reading outlines and for reading settings is too dear to you and you are not going to let it go.

Ah.  I didn't understand that this was the main idea.  It wasn't clear to me from reading your initial post.

You have pointed me to the page from the documentation that warns users about possible confusion when setting their preferences. I assure you that on more than one occasion I have spent hours trying to fix problem with my settings that was caused just by updating Leo, and I have always had only one myLeoSettings.leo.

If, as you assert, the "user will notice nothing", then how can the new behavior possibly solve such problems?

> So I have chosen the BreezeDarkTheme for my preference and now I can't edit myLeoSettings without loosing my preferred look and feel.

I don't have this problem.  My copy of myLeoSettings.leo contains:

   @string theme-name = EKRWindowsDark

Am I correct in assuming your myLeoSettings.leo contains:

    @string theme-name = BreezeDarkTheme?

Another example quite recent is introducing new dock layout. As you can see from the previous screenshots my preference is to have outline in the top left and log underneath, and body on the right. Recently when I updated Leo, docks were introduced and I couldn't find the way to layout the docks to look like the screenshot above. So, I had to change all my Leo launchers to add --no-dock argument. 

You probably have to make the body pane the central widget:

    @string central-dock-widget = body

My point is, you have many times introduced changes that required users to change their settings or to give up updating Leo.

Adding docks to Leo was a major change.  --no-docks makes it possible to use Leo exactly as before.  We spent weeks fixing bugs in the new code. You could have asked for help at any time during this process.

I'm not sure why you have such problems with your settings.  It should be possible to fix them with Leo as it is.

You have argued that introducing a cache potentially can cause some confusion. Well I can agree that it is a possibility. But Leo already uses cache.

Yes, Leo does use a cache.  --trace=cache will show you what it contains.  I've been looking at that a lot recently while I worked on the gui (global docks) branch.  At present, his cache is relatively benign because:

1. The cache contains a small amount of data, data that can be contained nowhere else.  Gui-related data, such as window position and arrangements of docks, arises directly from moving or rearranging docks. This data does not belong in an @settings tree.

2. The cache contains only non-essential data.  Clearing the cache is thus harmless.

3. The cache avoids polluting Leo outlines with non-essential data such marks and expanded "bits".

When it needs a value of some setting it doesn't read all those outlines again and doesn't parse them again to retrieve the value, but instead it retrieves the value from the cache that was built during the startup. The difference is when this cache is built.

I am not convinced that this is an improvement. See below.

In my idea it is built (and updated too) whenever user changes any setting, and not during the startup.

At present, the user "changes" a setting only when saving an outline (including leoSettings.leo and myLeoSettings.leo).  As you imply, this does not immediately update Leo's internal settings data.

That is why if implemented my idea will help to make things easier for users and allow them to see the effect of their changes immediately and not after the next reload. It would remove a lot of confusions that Leo at the moment imposes on users.

The reload-settings and reload-all-settings do (supposedly;-) update all settings data.

If settings were the only thing that mattered, then these commands should (in theory), suffice.  Yes, in practice, there could be bugs.

However, settings are NOT the only things that matter.  Settings, especially theme-related settings, and plugin-related settings, have side effects throughout Leo.  Imo, your proposal has no chance of mitigating these side effects.

My intention is not to impose my idea at any cost. I think this idea is a good one. It has several pros and all the cons you have pointed, I just don't see relevant. 

• It will not bring any inconvenience to users,
• it will not remove any of Leo features,
• it is not introducing a new caching mechanism because cache is already there. It just changes the time when this cache is built.

Imo, putting all of Leo's settings into the cache accomplishes virtually nothing.  It will add code that must be debugged.  Any discrepancy between the cache and settings in the various Leo file will lead to unbounded confusion.

The fundamental problem with this proposal is that no caching scheme can undo side effects arising Leo's myriad settings:

- If reload settings doesn't work, neither will updating cached setting.

- If unloading a plugin doesn't work, neither will updating @enabled-plugins.

The only reason to dismiss it, that I can still see, is that by accepting this idea you'd have to sacrifice one of your own, dear ideas and that is hard.

I have rejected them for the reasons stated.  To summarize:

- Caching settings creates the potential for serious confusion.

- I particularly want to minimize the size of the cache.

- Caching settings can not undo side effects.

  Leo's reload-settings command shows what is, and isn't, possible.

  The only way to be sure of the effects of changing a settings is to reload Leo fully.

If you don't want to discuss this idea any further, I won't insist either. Let's move on to something else. But if you ever wish to discuss it again, I'll be glad.

I think I now understand your original proposal more clearly.  I still think it's a bad idea. If you disagree with anything I have said here I am willing to discuss this topic further.

This discussion does contain several worthwhile sub-topics:

- Difficulties with theme settings.

- Difficulties with dock settings.

- Difficulties with unloading plugins.

- A command to fully reload Leo.

  pyzo has such a command, implemented in a few lines of code!

We can discuss these now, without having to imagine changing Leo's setting machinery in any way.

Edward

[vitalije]

On Tuesday, August 27, 2019 at 1:29:25 PM UTC+2, btheado wrote:

Vitalije,

Could you explain more on how using Leo's reading code for both reading outlines and reading settings is an obstacle to fixing some of your complaints (i.e. switching a theme can't be done without restarting leo)?

Because collecting settings in Leo happens during the startup, it is tied to startup. It can't be executed outside the startup code. That is the main reason why many settings require Leo restart to see the change.

GUI code depends on settings and can't be instantiated without settings but some of the Leo reading code depends on GUI elements. That makes the initialization code very hard to understand and maintain. It is a bit like the question what comes before chicken or egg. 

At the beginning of this thread Edward wrote about his idea of introducing global settings. The need for this was caused because there was an issue in the startup code. It is not clear to me what it was supposed to do: to read theme file before user settings or the other way around. But it is hard problem to solve mainly because startup code is so entangled and complex.

Your idea mentions "compiling" the settings to a simpler format at time of save. How does that help compared to "compiling" the settings to a simpler format during leo init? And isn't leo's reading of settings from outlines into some internal data structure also a form of "compiling" the settings to an alternate format? Is it that Leo's internal configuration structure doesn't match what you want or is the the reading from outlines alone which causes your complaint? Or both?

Main issue is using reading code for collecting settings. Changing this won't immediately solve my troubles with the settings, but it would allow simplifications in the startup code and that could lead to reducing number of bugs in the user preferences related code. Using reading outline code before fully initialized Leo is like creating a bike in two steps: first you attach one wheel to frame and then riding on one wheel to the store to buy the second wheel and other parts necessary to complete the bike. It is possible to do so, Leo has proven this. But it isn't natural.

Vitalije

[Edward K. Ream]

On Tue, Aug 27, 2019 at 7:13 AM vitalije <vita...@gmail.com> wrote:

GUI code depends on settings and can't be instantiated without settings but some of the Leo reading code depends on GUI elements. That makes the initialization code very hard to understand and maintain. It is a bit like the question what comes before chicken or egg. 

That's why Leo uses Apple's finishCreate pattern. It makes the chicken and egg problem explicit.

At the beginning of this thread Edward wrote about his idea of introducing global settings. The need for this was caused because there was an issue in the startup code. It is not clear to me what it was supposed to do: to read theme file before user settings or the other way around. But it is hard problem to solve mainly because startup code is so entangled and complex.

Happily, there is, in fact, no need for global settings.  In some places, such as computing the window's starting position and size, Leo uses the first loaded outline as a proxy.  That's why g.app.db contains entries for specific .leo files.

Edward

[vitalije]

If, as you assert, the "user will notice nothing", then how can the new behavior possibly solve such problems?

I meant to say: users are supposed to notice that Leo works better but they won't have to do anything to have this improvement.

  

> So I have chosen the BreezeDarkTheme for my preference and now I can't edit myLeoSettings without loosing my preferred look and feel.

I don't have this problem.  My copy of myLeoSettings.leo contains:

   @string theme-name = EKRWindowsDark

Am I correct in assuming your myLeoSettings.leo contains:

    @string theme-name = BreezeDarkTheme?

Yes you are correct. But I probably have some remaining settings from the previous ways of configuring the look and feel. I know what I have to do. I have to re-create myLeoSettings from the empty outline copying one by one setting and reloading Leo to see which settings are causing this issue, but as I said I am a bit reluctant to take such an effort because of many painful experiences I had with the changing myLeoSettings. My point was that those additional required steps that I have to do to fix my Leo are introduced just by updating Leo. I guess there are other users like me who just can't find enough will power to enter the settings battle again. But I am afraid that there might be even more users who just gave up Leo after encountering these kind of problems.

 

Another example quite recent is introducing new dock layout. As you can see from the previous screenshots my preference is to have outline in the top left and log underneath, and body on the right. Recently when I updated Leo, docks were introduced and I couldn't find the way to layout the docks to look like the screenshot above. So, I had to change all my Leo launchers to add --no-dock argument. 

You probably have to make the body pane the central widget:

    @string central-dock-widget = body

Probably, but it would again require more of my time to tweak myLeoSettings.leo having to reload it all the time. I might do this, but I prefer to wait until it is stabilized.

My point is, you have many times introduced changes that required users to change their settings or to give up updating Leo.

Adding docks to Leo was a major change.  --no-docks makes it possible to use Leo exactly as before.  We spent weeks fixing bugs in the new code. You could have asked for help at any time during this process.

 

I'm not sure why you have such problems with your settings.  It should be possible to fix them with Leo as it is.

I am not saying it is impossible. It just takes time and effort which I am not so eager to put into the tweaking my settings now. If it was possible for me to change setting central-dock-widget with a menu or even through executing minibuffer command I would have tried this. If it was possible for me to change theme setting in the menu or executing minibuffer command I would have tried this. Maybe it takes just to add two headlines to my settings file, but I am not sure. In my experience it might just turn into an hours long session of fixing some visual issues. That is the reason I rather choose not to follow this path. 

The real issue here is not my configuration problems. I am more concerned with the other users especially new ones. 

You have argued that introducing a cache potentially can cause some confusion. Well I can agree that it is a possibility. But Leo already uses cache.

Yes, Leo does use a cache.  --trace=cache will show you what it contains.  I've been looking at that a lot recently while I worked on the gui (global docks) branch.  At present, his cache is relatively benign because:

I was not talking about that cache. My point here is that c.config and g.app.config are caches in a nutshell. Leo doesn't read and scan outlines each time it needs particular setting value. Instead it retrieves this value from the g.app.config and c.config. So your point that my idea is to introduce cache is not relevant. Leo already uses cache for configuration, although I would say not very efficient one. 

When it needs a value of some setting it doesn't read all those outlines again and doesn't parse them again to retrieve the value, but instead it retrieves the value from the cache that was built during the startup. The difference is when this cache is built.

I am not convinced that this is an improvement. See below.

In my idea it is built (and updated too) whenever user changes any setting, and not during the startup.

At present, the user "changes" a setting only when saving an outline (including leoSettings.leo and myLeoSettings.leo).  As you imply, this does not immediately update Leo's internal settings data.

But it could if the settings are handled differently. 

 

That is why if implemented my idea will help to make things easier for users and allow them to see the effect of their changes immediately and not after the next reload. It would remove a lot of confusions that Leo at the moment imposes on users.

The reload-settings and reload-all-settings do (supposedly;-) update all settings data.

I have just tested that reloading all settings won't change theme. You can say that is how it is supposed to work. But for me this is a bug. If we just put aside question how Leo should handle settings for a moment. Telling user that it is impossible to change theme on the fly because it is technically impossible to achieve this is a bit awkward and false. The sizes, colors, and fonts can be easily changed and immediately visible in PyQt widgets. The fact that we, Leo developers, can't deliver that feature to the Leo users is something we should be ashamed of. The reason why we can't deliver it lays in an inadequate way how Leo handles its own configuration.

 

If settings were the only thing that mattered, then these commands should (in theory), suffice.  Yes, in practice, there could be bugs.

As I said there are bugs.

 

However, settings are NOT the only things that matter.  Settings, especially theme-related settings, and plugin-related settings, have side effects throughout Leo.  Imo, your proposal has no chance of mitigating these side effects.

Maybe not directly, but through simplifying startup code, chances of fixing those problems are increased. 

Imo, putting all of Leo's settings into the cache accomplishes virtually nothing.  It will add code that must be debugged.  Any discrepancy between the cache and settings in the various Leo file will lead to unbounded confusion.

On the contrary in the present scheme there are constant discrepancy between the cached c.config and g.app.config settings and the values user sets. That is why Leo requires restart to pick up the changes.  

 

The fundamental problem with this proposal is that no caching scheme can undo side effects arising Leo's myriad settings:

- If reload settings doesn't work, neither will updating cached setting.

You are right just updating cached settings is not the full solution, but it is necessary precondition to make reload settings mightier. It would require some more refactoring in other parts of Leo to use config cache more efficiently. Anyway reload-settings should change the theme at the minimum.

I have rejected them for the reasons stated.  To summarize:

- Caching settings creates the potential for serious confusion.

You haven't shown any evidence to support this claim. And I say that serious confusion already exists in the way Leo handles settings.  

- I particularly want to minimize the size of the cache.

Interesting. What is the point of this particular wish? Have you encounter any cache-size related issues? Has anybody complained about large cache?

- Caching settings can not undo side effects.

  Leo's reload-settings command shows what is, and isn't, possible.

  The only way to be sure of the effects of changing a settings is to reload Leo fully.

Yes at the moment, but I wouldn't call it a feature, but rather a bug. Side effects should be minimized. But I must say again. There is no valid reason that changing widget sizes, fonts and colors requires restart.

Vitalije

[Edward K. Ream]

On Tue, Aug 27, 2019 at 12:01 AM Matt Wilkie <map...@gmail.com> wrote:

Whoa. Edward, the strength of your reaction seems out of proportion.

Noted.

Edward

[Edward K. Ream]

On Tue, Aug 27, 2019 at 8:25 AM vitalije <vita...@gmail.com> wrote:

> My point was that those additional required steps that I have to do to fix my Leo are introduced just by updating Leo.

I do apologize for the problems you have been having. My intention is to leave existing settings alone, wherever possible.  I do retire newly unused settings from leoSettings.leo.

Behavior such as you describe can legitimately be called a bug.   The general principle is new settings in leoSettings.leo should have default values the preserve Leo's legacy operation.

> You probably have to make the body pane the central widget:

    @string central-dock-widget = body

Probably, but it would again require more of my time to tweak myLeoSettings.leo having to reload it all the time. I might do this, but I prefer to wait until it is stabilized.

It should be stable as of Leo 6.0.  The present work with global docks should not affect you, because it is in the gui branch.  Even when merged into devel, global docks will only happen when --global-docks is in effect, and even then there are, iirc, no new settings.

My point is, you have many times introduced changes that required users to change their settings or to give up updating Leo.

You are the first to complain of such serious settings problems.  As I said in my original reply, the goals of stability and innovation are inherently somewhat at odds.

However, the principle of retaining legacy operation in the default settings in leoSettings.leo should mitigate this tension.

I'm not sure why you have such problems with your settings.  It should be possible to fix them with Leo as it is.

I am not saying it is impossible. It just takes time and effort which I am not so eager to put into the tweaking my settings now.

Fair enough.

If it was possible for me to change setting central-dock-widget with a menu or even through executing minibuffer command I would have tried this. If it was possible for me to change theme setting in the menu or executing minibuffer command I would have tried this.

It's called reload-settings.  I think we both agree that it won't work.

Maybe it takes just to add two headlines to my settings file, but I am not sure. In my experience it might just turn into an hours long session of fixing some visual issues. That is the reason I rather choose not to follow this path. 

The real issue here is not my configuration problems. I am more concerned with the other users especially new ones. 

Let's let newbies speak for themselves.

I have just tested that reloading all settings won't change theme. You can say that is how it is supposed to work. But for me this is a bug.

Fair enough.

If we just put aside question how Leo should handle settings for a moment. Telling user that it is impossible to change theme on the fly because it is technically impossible to achieve this is a bit awkward and false. The sizes, colors, and fonts can be easily changed and immediately visible in PyQt widgets. The fact that we, Leo developers, can't deliver that feature to the Leo users is something we should be ashamed of.

Ahem.  If you think reload settings can update themes, feel free to do so.  It's a reasonable enhancement, that can be done without any major changes to Leo's configuration machinery.  You would most likely want to change (or create) methods called reloadSettings or reload_settings (depending on the class).

The reason why we can't deliver it lays in an inadequate way how Leo handles its own configuration.

No.  It's an indication of the complexity of themes.

I think we are done discussing this topic. 

Edward

[Xavier G. Domingo (xgid)]

Let's let newbies speak for themselves.

I think it's time to speak! 

Regarding the Leo settings "world": for a Leo newbie like me, settings in Leo are a complicated issue for sure. So it's something I try to avoid touching as much as I can... but just because the UX is really poor. Some of the problems I find:

1) It's difficult to know what settings do I have "available" to tweak. 

As an example, I have just opened my myLeoSettings.leo file to take a look (after more than a year of not doing so) and I've found there a node with just this:

 

@bool minibuffer_find_mode = False

 

But the node body is empty, so I don't know what that setting is about at all. So I've tried the "obvious" thing (though probably that's not what we can expect a newbie to do... so I'm probably moving away from the newbie "hat" here...): I've opened the leoSettings.leo file and did a search for "minibuffer_find_mode" to see what's this setting all about. Result: I've found nothing.

Then I've done something a newbie would (more) probably do: a search in leodeditor.com: https://leoeditor.com/search.html?q=minibuffer_find_mode&check_keywords=yes&area=default

Result: no results. 

So I've "guessed" that I probably got that setting from someone else making a comment in this forum. And I've done the search: https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!searchin/leo-editor/minibuffer_find_mode$20%7Csort:date

With success! There are several posts from Edward and others which include the setting, the most relevant one being this one from Edward stating that:

 P.S. I recommend minibuffer-find mode for experts.  Enable with @bool minibuffer_find_mode = True.

 

After a lot mooore digging into the issue, I've finally found within leoSettings.leo a node with header "Find/replace options" and as a child I've found a node with, guess what...:

@bool minibuffer-find-mode = False 

So well: this story had a nice ending... but I would not call this a good UX for newbies (or anyone). And all the above is just for one only setting!

2) It's difficult to know what settings are really "in effect" at any moment.

Yes, there's the "show-settings" command, I know that. But it shows you a long list of settings in a separate (text) pane, which for the example above happens to show that setting with yet another different "name":

[M] @bool   minibufferfindmode = False 

  

So although "in theory" the command should be "all you need to know"... I wouldn't qualify that as a nice UX.

But I'm the same newbie when using VSCode and there I have none of the above problems of settings discoverability and knowing which settings are in effect at any moment. The UX there is really great! Even if I directly edit the settings.json file instead of using it's settings GUI editor. I always know what I'm doing. 

And I can say that I've had the same good UX, or even better, with Pycharm, for example.

I don't pretend that we now radically change the way settings are "entered" in Leo (leoSettings, myLeoSettings and so on), but it's revealing to notice that the "show-settings" command is showing a "flattened" view of the settings! Doesn't that sound as a hint to you?

For me it's a clear indication (despite being a newbie, I'm a programmer) that what Leo uses internally to access those settings is a flattened list of settings and that there's probably a dictionary (in the Python sense) or a map that Leo is using in memory to access the settings. Isn't that the in effect "the cache" that Vitalije is talking about? I think so.

And that takes me to my second contribution today... regarding how Leo loads settings during startup time. 

When we discussed this same issue years ago (I have no time now to search in the forum) Vitalije and Terry did already commented some possible ways for improving it and I myself did also suggested another simple possible path to move forward, which at that time was simply ignored.

My contribution was something in this line of thinking: "You don't need during startup all the Leo machinery just to load the settings files".

And I do have stored in my Leo notes this quote from Terry about the issue:

I think there's great potential for simplifying init. code and settings 

editing by using something simpler than full blown Leo outlines for the 

key/value store needed for settings.  Sqlite is certainly an option - 

if you're just using it for settings, it may not be necessary to 

include those files in VCS. 

So I think that this all shows that there's really room for improvement in how Leo currently handles settings, both from the end-user point of view (UX) and from the "Leo internals" point of view (code).

Edward: what else do you need to see the evidence...? Why are you having a so strong reaction to Vitalije's proposal?

---------------------------------------------

I think we are done discussing this topic.

 

Edward, I'm not a native english speaker, but I have to say that a sentence like the one above gives a very bad impression: it's not encouraging participation from the community, but the opposite.

And here it comes my second suggestion: I agree we probably don't need a Steering Council for Leo, but I think it would really great for our community if we moved our discussions to a real modern forum like Discourse, as Python and many other communities have already done.

This post from the Python forum gives some explanations of the particular benefits/features of Discourse, though it is in response to someone comparing it to a simple mailing list.

But some of the greatest features that they don't mention there which I think would really benefit our community are:

• Easy refactoring of forum threads! Moderators can easily move off-topic posts to a new thread, so the current topic can be kept focused, without losing any contributions. This is for me a feature that sets Discourse apart from any other forum software (and I've looked at many).
• Easy creation of polls!

Though there are probably many more that I even yet don't know.

I stongly feel that we have to move forward as a Team and Discourse could be a game changer in many ways.

Your comments are wellcome!

[Edward K. Ream]

On Wed, Aug 28, 2019 at 2:26 AM Xavier G. Domingo (xgid) <xgdo...@gmail.com> wrote:

Regarding the Leo settings "world": for a Leo newbie like me, settings in Leo are a complicated issue for sure. So it's something I try to avoid touching as much as I can... but just because the UX is really poor. Some of the problems I find:

1) It's difficult to know what settings do I have "available" to tweak. 

As an example, I have just opened my myLeoSettings.leo file to take a look (after more than a year of not doing so) and I've found there a node with just this:

 

@bool minibuffer_find_mode = False

 

But the node body is empty, so I don't know what that setting is about at all. So I've tried the "obvious" thing (though probably that's not what we can expect a newbie to do... so I'm probably moving away from the newbie "hat" here...): I've opened the leoSettings.leo file and did a search for "minibuffer_find_mode" to see what's this setting all about. Result: I've found nothing.

Big sigh.  This problem, as you discovered, is the result of a change awhile back, which uses "-" everywhere instead of "_".  This was intended to make searches easier! In your case, it had the opposite effect.

There is no way to improve Leo's past.  You just have to be aware that the preferred names (in leoSettings.leo) use dashes (minus signs) instead of underscore.

> 2) It's difficult to know what settings are really "in effect" at any moment.

Yes, it's a problem. I think there is a feature that finds settings for you, but I can't find it at present.  Maybe it's a plugin.

> when using VSCode...I have none of the above problems of settings discoverability and knowing which settings are in effect at any moment...I always know what I'm doing.

This is a frequently requested enhancement. I've just reopened #852 for this. The first comment in #852 contains my present thoughts. To summarize...

Leo has a huge number settings. The set of all settings is dynamic. To show all these settings clearly, they must be shown in a Leo outline, organized both by settings source (the local .leo file, myLeoSettings.leo and leoSettings.leo) and by the given organization of nodes within the @settings tree in each settings source.

This active settings outline will be a real Leo outline, shown, say, in a new tab.

This outline must combine both the contents and the organization of the @settings trees of the three settings sources. It's tricky, but possible.

Furthermore, we would like to be able to change settings (that is change the original @settings trees) simply by changing nodes in the active settings outline.  Alas, the active settings outline can't clone nodes from other outlines.  Instead, saving the active settings outline will (somehow!) update all settings sources.

I think this project is worth doing, given the problems with the show-settings command.

Edward

P.S. I do not want to discuss Leo's startup code further.

EKR