[proposal][patch] Want to stop supporting troublesome options 'gdefault' and 'edcompatible'.

[h_east]

Hi Bram and all,

'gdefault' invertes the 'g' flag of `:substitute`.
In addition to 'edcompatible', it also inverts the 'c' flag.

When using `:substitute` with plugin, save and restore of the above options are necessary, it is a little weird.

I propose to stop supporting these options.
Please check the attached patch.

P.S.
I'm sorry if my choice of words was wrong and you made unpleasant.
--
Best regards,
Hirohito Higashi (h_east)

[Christian Brabandt]

Hi h_east!

On Mi, 03 Mai 2017, h_east wrote:

> 'gdefault' invertes the 'g' flag of `:substitute`.
> In addition to 'edcompatible', it also inverts the 'c' flag.
>
> When using `:substitute` with plugin, save and restore of the above options are necessary, it is a little weird.
>
> I propose to stop supporting these options.
> Please check the attached patch.

I made a little github code search for some vim settings. The results
are a bit surprising:

Search Term Results
-------------------------------
set edcompatible 6
set gdefault 8,289
set langnoremap 718
set nolangremap 207
set termguicolors 3,425
set t_Co 98,886
set laststatus 75,336
set nocompatible 94,745
set incsearch 79,623

So gdefault seems to be popular to a certain degree. I wouldn't have
thought that, it is just such an obscure option.

By contrast the usage of edcompatible can be neglected, that option
seems to be in almost no use. Somewhat surprisingly, 'langnoremap' is
still somewhat widespread, also it is replaced by 'langremap' option.

The relative newly introduced option termguicolors already seems to have
widespread (at least it is already used more than the langnoremap
option, which was introduced earlier).

I also included a search for settings I expected to be in wide use, e.g.
'nocompatible', 't_Co', 'laststatus' and 'incsearch' and that seems to
be the case.

Coming back to the topic:
I am all for removing those options, however this is a hard change that
may break scripts and will probably also annoy users (especially of
'gdefault', from which I would expect several bug reports, if that
option will be dropped).

I see two possible solutions:

1) start echoing warning messages, that those options are deprecated and
after a grace period (e.g. the switch to Vim9) remove those options.
2) make a distinction between interactive usage and script usage and for
the script usage provide a clean environment where those options are not
set, so scripts/functions do not need to handle that. That could also be
enhanced later for other settings or even mappings.

2 seems to be the cleaner approach and does not bug the user much, so
that might be preferable. However I don't know how hard to implement
this would be.

But anyhow, either approach is okay for me.

Best,
Christian
--
Der Baum hat Äste, das ist das Beste. Denn wäre er kahl, dann wär's ein Pfahl.

[Justin M. Keyes]

Neovim removed 'edcompatible', we've received zero complaints about
that. 'gdefault' is a misfeature that probably can't be removed,
though I would be in favor of it.

Approach (2) would open a great opportunity for removing a lot of pain
of writing plugins. The fact that it hasn't been done in the last 20
years makes me think there is too much uncertainty about the
side-effects; it's an obvious approach that should have been taken
long ago, but now the technical debt has piled up.

---
Justin M. Keyes

[François Ingelrest]

Hi,

On 3 May 2017 at 11:11, Christian Brabandt <cbl...@256bit.org> wrote:
> So gdefault seems to be popular to a certain degree. I wouldn't have
> thought that, it is just such an obscure option.

I've always used it, it's quite handy to always replace all
occurrences and not just the first one. In fact I can hardly remember
a time where I wanted just the first occurrence in a line to be
replaced, hence the gdefault option.

Not that I wouldn't be able to live without it, but I would certainly
not consider it useless and from your stats it seems I'm not the only
one.

[h_east]

Hi ChrisBra,

2017-5-3(Wed) 18:11:16 UTC+9 Christian Brabandt:

> Hi h_east!
>
> On Mi, 03 Mai 2017, h_east wrote:
>
> > 'gdefault' invertes the 'g' flag of `:substitute`.
> > In addition to 'edcompatible', it also inverts the 'c' flag.
> >
> > When using `:substitute` with plugin, save and restore of the above options are necessary, it is a little weird.
> >
> > I propose to stop supporting these options.
> > Please check the attached patch.
>
> I made a little github code search for some vim settings. The results
> are a bit surprising:
>
> Search Term Results
> -------------------------------
> set edcompatible 6
> set gdefault 8,289
> set langnoremap 718
> set nolangremap 207
> set termguicolors 3,425
> set t_Co 98,886
> set laststatus 75,336
> set nocompatible 94,745
> set incsearch 79,623
>
> So gdefault seems to be popular to a certain degree. I wouldn't have
> thought that, it is just such an obscure option.

Wow! There was a person setting 'gdefault' so much!

>
> By contrast the usage of edcompatible can be neglected, that option
> seems to be in almost no use. Somewhat surprisingly, 'langnoremap' is
> still somewhat widespread, also it is replaced by 'langremap' option.
>
> The relative newly introduced option termguicolors already seems to have
> widespread (at least it is already used more than the langnoremap
> option, which was introduced earlier).
>
> I also included a search for settings I expected to be in wide use, e.g.
> 'nocompatible', 't_Co', 'laststatus' and 'incsearch' and that seems to
> be the case.
>
> Coming back to the topic:
> I am all for removing those options, however this is a hard change that
> may break scripts and will probably also annoy users (especially of
> 'gdefault', from which I would expect several bug reports, if that
> option will be dropped).

Surely. It is better to delete the 'gdefault' carefully.
Probably, I think that there is no big problem even if 'edcompatible' is removed.

>
> I see two possible solutions:
>
> 1) start echoing warning messages, that those options are deprecated and
> after a grace period (e.g. the switch to Vim9) remove those options.
> 2) make a distinction between interactive usage and script usage and for
> the script usage provide a clean environment where those options are not
> set, so scripts/functions do not need to handle that. That could also be
> enhanced later for other settings or even mappings.
>
> 2 seems to be the cleaner approach and does not bug the user much, so
> that might be preferable. However I don't know how hard to implement
> this would be.

I like 1.
2 seems more confused for me.

>
> But anyhow, either approach is okay for me.

Thanks for search, suggest and opinion.

[h_east]

Hi Justin,

2017-5-3(Wed) 18:44:14 UTC+9 Justin M. Keyes:

Yeah, I knew you removed it.
But I didn't know without any complaints. Thank you for telling us.

> 'gdefault' is a misfeature that probably can't be removed,
> though I would be in favor of it.

As you say, it seems that it can't be removed immediately.

>
> Approach (2) would open a great opportunity for removing a lot of pain
> of writing plugins. The fact that it hasn't been done in the last 20
> years makes me think there is too much uncertainty about the
> side-effects; it's an obvious approach that should have been taken
> long ago, but now the technical debt has piled up.

"now the technical debt has piled up"

Good words.
I think so too.
It isn't preferable that the meaning of flag is reversed by the option value.
Or We should adding a flag that does not affect the value of the option.
e.g.: 'i' and 'I' flag that not dependent on option 'ignorecase' and 'smartcase'.

[Bram Moolenaar]

The problem with removing options is that you always hurt some users.
And most probably the ones that just use whatever they have on their
system. Thus complaints might come very late. Neovim is certainly not
a good indication, because these users have made a choice to use a
non-standard Vi/Vim.

Also, when one option only has 0.01% usage, and we remove a dozen of
them, we are already at 0.12%, one in a thousand users. That's likely
more than users of more advanced features.

Besides that, plugin writers also have a problem with very common
options, such as 'wrapscan' and 'ignorecase'. We are nog going to
remove these. Having an easy way to set these to their default, and
restore them later (without side effects), would be very useful.

For flexibility this needs to work recursively. We could do something
like:

let saved_options = options_save()
... do your stuff ...
call options_restore(saved_options)

The options being saved should be small to keep this efficient. We need
to make a list of the ones that are useful, such as 'ignorecase' and
'gdefault'.

--
Kisses may last for as much as, but no more than, five minutes.
[real standing law in Iowa, United States of America]

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Christian Brabandt]

Would those then also be set to a default by options_save? Or does every
plugin write have to set them as well?

Also worth containing: 'magic', 'cpo' and 'cp' options.

Best,
Christian
--
Auf welcher Gesetzestafel steht: Die heiligen Gefühle der Theisten
müssen respektiert werden, die heiligen Gefühle der A-Theisten aber
nicht?
-- Ludwig Marcuse

[Justin M. Keyes]

That's true, but in the case of 'edcompatible' the usage is << 0.01%.
And probably accidental.

In the UB thread[1] it was implied that 99.9% is the target audience:

> The C language was originally made for a wide range of CPU, even weird
> ones. Now that 99.9% of computers are fairly standard it would be a bad
> idea to suffer from supporting those conrder cases.

Not only that, but decisions about Vim's default behavior have
_benefits_ for most users. So it's incomplete to only analyze
potentially "harmed" users, without also balancing against potential
benefit.

There's also no purpose in optimizing for the use-case of users that
literally don't care about any new features in Vim. Yet Vim continues
to orient itself towards this "silent majority" of users that leverage
Vim as a glorified Notepad.

[1] https://groups.google.com/d/msg/vim_dev/_tqf8eQy5eA/jhHqnroaCAAJ

> Besides that, plugin writers also have a problem with very common
> options, such as 'wrapscan' and 'ignorecase'. We are nog going to
> remove these. Having an easy way to set these to their default, and
> restore them later (without side effects), would be very useful.
>
> For flexibility this needs to work recursively. We could do something
> like:
>
> let saved_options = options_save()
> ... do your stuff ...
> call options_restore(saved_options)
>
> The options being saved should be small to keep this efficient. We need
> to make a list of the ones that are useful, such as 'ignorecase' and
> 'gdefault'.

Sure. Meanwhile, `:set all&` still has bugs.

---
Justin M. Keyes

[Bram Moolenaar]

options_save() is not the best name. It should also set those options to
their default value. options_default() perhaps.

> Also worth containing: 'magic', 'cpo' and 'cp' options.

All options changing behavior of regular expressions, at least.

--
Biting someone with your natural teeth is "simple assault," while biting
someone with your false teeth is "aggravated assault."
[real standing law in Louisana, United States of America]

[Bram Moolenaar]

True, some options are only there for Vi compatibility (or Posix
standard conformance). But removing them doesn't really have a benifit
either.

> In the UB thread[1] it was implied that 99.9% is the target audience:

No, that's completely out of context.

> > The C language was originally made for a wide range of CPU, even weird
> > ones. Now that 99.9% of computers are fairly standard it would be a bad
> > idea to suffer from supporting those conrder cases.
>
> Not only that, but decisions about Vim's default behavior have
> _benefits_ for most users. So it's incomplete to only analyze
> potentially "harmed" users, without also balancing against potential
> benefit.
>
> There's also no purpose in optimizing for the use-case of users that
> literally don't care about any new features in Vim. Yet Vim continues
> to orient itself towards this "silent majority" of users that leverage
> Vim as a glorified Notepad.

Please don't insult our users. There are lots of features they haven't
discovered yet, sure, it's really up to the user how much time he wants
to spend finding better ways to do something.

Also, there have been features added that turn out to get very little
use, no matter how enthousiastic people where when it was introduced.

It might be better to spend time on helping users find and use the
existing features, than adding more of then. E.g., I am often suprised
people don't know of persistent undo.

> [1] https://groups.google.com/d/msg/vim_dev/_tqf8eQy5eA/jhHqnroaCAAJ
>
> > Besides that, plugin writers also have a problem with very common
> > options, such as 'wrapscan' and 'ignorecase'. We are nog going to
> > remove these. Having an easy way to set these to their default, and
> > restore them later (without side effects), would be very useful.
> >
> > For flexibility this needs to work recursively. We could do something
> > like:
> >
> > let saved_options = options_save()
> > ... do your stuff ...
> > call options_restore(saved_options)
> >
> > The options being saved should be small to keep this efficient. We need
> > to make a list of the ones that are useful, such as 'ignorecase' and
> > 'gdefault'.
>
> Sure. Meanwhile, `:set all&` still has bugs.

It's in the todo list, but apparently nobody has taken on the job of
working on this. I admit it's going to be a bit of work to refactor the
code to handle the side effect of changing an option (instead of a very
long if-else statement).

Anyway, how does this relate to new features? Or perhaps we should
first fix existing features before adding more? (that might then have
bugs again).

--
Why don't cannibals eat clowns?
Because they taste funny.

[Andy Wokula]

I suppose one reason for the many gdefault hits is that quite a lot of script
writers are aware of the trouble with this option and take care of
saving and restoring it. This is in turn is the case because gdefault is
somewhat popular (though not as popular as the number suggests ...).

--
Andy

[Andy Wokula]

Setting 'cp' is too invasive.
Anyway, most plugins finish early if 'cp' is set.

--
Andy

[Andy Wokula]

Am 08.05.2017 um 12:53 schrieb Bram Moolenaar:
> For flexibility this needs to work recursively. We could do something
> like:
>
> let saved_options = options_save()
> ... do your stuff ...
> call options_restore(saved_options)
>
> The options being saved should be small to keep this efficient. We need
> to make a list of the ones that are useful, such as 'ignorecase' and
> 'gdefault'.

What about a different approach:

Have a mode (for debugging only) in which builtin commands and functions complain when options they use are not at the default value.
This way the script writer learns which options need to be set and restored.

(maybe: When used at toplevel (at the cmdline) there is no warning)

The advantage is that exactly the required options are set and restored.


And I'm not sure if the "last set from" info would work after options_save().

--
Andy

[Gary Johnson]

On 2017-05-08, 'Andy Wokula' via vim_dev wrote:
> Am 08.05.2017 um 12:53 schrieb Bram Moolenaar:
> >For flexibility this needs to work recursively. We could do something
> >like:
> >
> > let saved_options = options_save()
> > ... do your stuff ...
> > call options_restore(saved_options)
> >
> >The options being saved should be small to keep this efficient. We need
> >to make a list of the ones that are useful, such as 'ignorecase' and
> >'gdefault'.
>
> What about a different approach:
>
> Have a mode (for debugging only) in which builtin commands and
> functions complain when options they use are not at the default
> value. This way the script writer learns which options need to be
> set and restored.

Problems usually occur not with the options the script writer has
set to non-default values, but with options the script user has set
to non-default values. This mode is not going to tell the writer
which options users may have set.

The safest approach, and the one most considerate of the user, is
for the plugin to save and restore the current value of any option
it depends on and sets.

Regards,
Gary

[h_east]

Hi Bram and all,

2017-5-8(Mon) 19:53:37 UTC+9 Bram Moolenaar:

Certainly I have proposed to remove the option easily.
That's right, I should suggest a good solution before choosing to remove the option.
You have maintained "Vim specification" for over 20 years.
I don't know if your judgment is always right :-), but I would to respect your opinion strongly.

>
> Besides that, plugin writers also have a problem with very common
> options, such as 'wrapscan' and 'ignorecase'. We are nog going to
> remove these. Having an easy way to set these to their default, and
> restore them later (without side effects), would be very useful.
>
> For flexibility this needs to work recursively. We could do something
> like:
>
> let saved_options = options_save()
> ... do your stuff ...
> call options_restore(saved_options)
>
> The options being saved should be small to keep this efficient. We need
> to make a list of the ones that are useful, such as 'ignorecase' and
> 'gdefault'.

For 'ignorecase' and 'smartcase', We have `\c` or `\C` that can ignore that setting. (`:help /\c`, `:help /\C`)
Also, for 'magic', We have `\m` and `\M`. (`:help /\m`, `:help /\M`)

It may be good if there is a similar mechanism for 'wrapscan', 'gdefault' and 'edcompatible'.

Thanks.

[Tony Mechelynck]

I'd rather remove as little existing stuff as possible, especially
when (for someone who knows how to read the help) it is known to work
as specified. Why not concentrate on the items in todo.txt before we
start waving sabers around into the existing code?

Best regards,
Tony.

[Andy Wokula]

Am 08.05.2017 um 22:28 schrieb Gary Johnson:
> On 2017-05-08, 'Andy Wokula' via vim_dev wrote:
>> Am 08.05.2017 um 12:53 schrieb Bram Moolenaar:
>>> For flexibility this needs to work recursively. We could do something
>>> like:
>>>
>>> let saved_options = options_save()
>>> ... do your stuff ...
>>> call options_restore(saved_options)
>>>
>>> The options being saved should be small to keep this efficient. We need
>>> to make a list of the ones that are useful, such as 'ignorecase' and
>>> 'gdefault'.
>>
>> What about a different approach:
>>
>> Have a mode (for debugging only) in which builtin commands and
>> functions complain when options they use are not at the default
>> value. This way the script writer learns which options need to be
>> set and restored.
>
> Problems usually occur not with the options the script writer has
> set to non-default values, but with options the script user has set
> to non-default values. This mode is not going to tell the writer
> which options users may have set.

I hit the Send button, then it hit me ... right.

> The safest approach, and the one most considerate of the user, is
> for the plugin to save and restore the current value of any option
> it depends on and sets.

But the question remains how to most easily and certainly find out
*which* options to set. It should be more transparent which commands
(and functions) are affected by which options.

When everybody knows that for sure (maybe via dedicated help page?)
:command affected by 'option', ...
...
function() affected by 'option', ...
...
h affected by 'whichwrap', ...

then there could be a helper which traces executed (builtin) commands
and functions along a given way of execution, eg within the scope of a
modifier command, and fills a v:variable ... something like that.
Sounds like a lot of work ... each builtin command and function would
need to support it.
And :normal commands (e.g. `h' affected by 'whichwrap') are probably not
included. Maybe `:normal h' is included, but not `feedkeys("h")' (out of
modifier command scope).

--
Andy

[Nikolay Aleksandrovich Pavlov]

This can be emulated currently, and this is one of the worst approaches.

Problems with emulations which may be worked around:

1. It resets place from which option is considered to be set (i.e.
what `:verbose set option?` reports).
2. Care must be taken about regarding option scope, especially if “…
do your stuff …” has switched buffers and windows. But unlike Python
API Vim will not bark at you if you use `setlocal gdefault` even
though &gdefault is a global-only option, it will silently set global
option instead of what was requested (when creating Python API I
intentionally made `vim.options` work with globals only,
`vim.Buffer.options` work with buffer-local, etc;
`vim.current.buffer.options['gdefault'] = False` is not going to
work).

Problems which cannot, not without breaking semantics at least:

1. If “… do your stuff …” raises exception and `call
options_restore()` was not placed inside `:finally` while stuff placed
in `:try` restoring options will never occur, presenting strange bugs
to users.
2. Programmer may forget to do restoring on his own.

One of the logical options would be introducing context managers,
similar to e.g. Pythonic:

`with … | stuff | endwith` works like putting `stuff` in `try` and
making `finally` block. `…` may be the following:

`with set sanedefaults`: temporary set a small selected set of options
to default values and restore them after that. Suggested options:
nogdefault, magic, noignorecase, wildignore=, fileignorecase&vim,
maybe something else.

`with let &option=expr1 &l:option2=expr1`: temporary set some options
and restore them afterwards. Cares about where to restore them: e.g.
if switched to different window/buffer restores option on the
window/buffer where it was reset, or nowhere at all if those were
wiped out.

Note: I am not fond of the idea of using `execute 'with …'`, so no
`with set x=…`. No *at all*, unless you want to ban unfinished
contexts inside `execute` because without banning them you can never
get a proper parser for VimL. Currently things like `execute 'if 1'`
work and if you allow `with set x=y` while having `with let &x='y'` a
number of developers will for sure ignore `with let` and use
`:execute`.

Note 2: `verbose set option?` will report the SID context manager is
running in for the duration of the context manager, then SID is
restored alongside with the option value for both managers which
save/restore options.

Note 3: `with let` may be extended, but at start just supporting
options is enough. Saving/restoring e.g. global settings of plugins is
far less common because they are far less intrusive.

`with saved input`: alias for using inputsave()/inputrestore().

`with saved view`: alias for winsaveview()/winrestview().

`with context expr1 [as var]`: user context manager, `expr1` must
evaluate to a dictionary, treating this context manager like this:

let _d = eval(expr1)
if type(_d) != type({}) | throw 'Not dict' | endif
let _enter = get(_d, 'enter', 0)
let _exit = get(_d, 'exit', 0)
if _enter isnot 0 && type(_enter) != 2 | throw 'enter not callback' | endif
if _exit isnot 0 && type(_exit) != 2 | throw 'exit not callback' | endif
if _enter is 0 && _exit is 0 | throw 'no callbacks' | endif
if _enter isnot 0
let var = call(_enter, [], _d)
endif
try
stuff
finally
if _exit isnot 0
call call(_exit, [], _d)
endif
endtry

Notes: intentionally setting callback `self` to `_d` regardless of
what may be saved there. Either enter or exit may be omitted, but not
both. If `as var` was not provided return value if `_enter` should be
just thrown away. Not providing any arguments to exit (Python provides
exception details if it occurred) because expecting things like
v:exception to be set. Calling `_enter` outside of `try`.

>
> --
> Kisses may last for as much as, but no more than, five minutes.
> [real standing law in Iowa, United States of America]
>
> /// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
> /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
> \\\ an exciting new programming language -- http://www.Zimbu.org ///
> \\\ help me help AIDS victims -- http://ICCF-Holland.org ///
>

> --
> --
> You received this message from the "vim_dev" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
>
> ---
> You received this message because you are subscribed to the Google Groups "vim_dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Bram Moolenaar]

> >> Besides that, plugin writers also have a problem with very common
> >> options, such as 'wrapscan' and 'ignorecase'. We are nog going to
> >> remove these. Having an easy way to set these to their default, and
> >> restore them later (without side effects), would be very useful.
> >>
> >> For flexibility this needs to work recursively. We could do something
> >> like:
> >>
> >> let saved_options = options_save()
> >> ... do your stuff ...
> >> call options_restore(saved_options)
> >>
> >> The options being saved should be small to keep this efficient. We need
> >> to make a list of the ones that are useful, such as 'ignorecase' and
> >> 'gdefault'.
> >
> > Would those then also be set to a default by options_save? Or does every
> > plugin write have to set them as well?
> >
> > Also worth containing: 'magic', 'cpo' and 'cp' options.
>
> Setting 'cp' is too invasive.
> Anyway, most plugins finish early if 'cp' is set.

Right, plugins usually work better with Vim defaults. Including allowing line
continuation.

--
A special cleaning ordinance bans housewives from hiding dirt and dust under a
rug in a dwelling.
[real standing law in Pennsylvania, United States of America]

[Bram Moolenaar]

Using a "context" / "endcontext" block could be useful to make sure the restore
is always invoked. It would work similar to a try/finally.

Whether there are more things to restore, I'm not sure if a context block is
the best solution for that. If so, there could be a way to add callbacks to
invoke at the end of the block. Go has a mechanism like that (always end of
the function though).

It is illegal to take more than three sips of beer at a time while standing.
[real standing law in Texas, United States of America]

[Bram Moolenaar]

Andy Wokula wrote:

> Am 08.05.2017 um 12:53 schrieb Bram Moolenaar:
> > For flexibility this needs to work recursively. We could do something
> > like:
> >
> > let saved_options = options_save()
> > ... do your stuff ...
> > call options_restore(saved_options)
> >
> > The options being saved should be small to keep this efficient. We need
> > to make a list of the ones that are useful, such as 'ignorecase' and
> > 'gdefault'.
>
> What about a different approach:
>
> Have a mode (for debugging only) in which builtin commands and functions complain when options they use are not at the default value.
> This way the script writer learns which options need to be set and restored.

Suppose we add a new option which, when set, interferes with some plugins.
With the options_save()/options_restore() approuch we can avoid breaking
plugins without making any changes. With your method we would need to wait for
plugin writers to update for the new option, which can take a long time.

> (maybe: When used at toplevel (at the cmdline) there is no warning)
>
> The advantage is that exactly the required options are set and restored.
>
>
> And I'm not sure if the "last set from" info would work after options_save().

It woud be restored as well.

--
No man may purchase alcohol without written consent from his wife.
[real standing law in Pennsylvania, United States of America]