Theme dev (Selenized) help: SpellRare and undercurl not working in Vim, works in NeoVim

[Daniel Gray]

Hi,

I'm currently working on a theme called Selenized https://github.com/jan-warchol/selenized/

We're using https://github.com/lifepillar/vim-colortemplate to template
our theme.

The issue I'm having is https://github.com/jan-warchol/selenized/pull/71#pullrequestreview-375757756

We cannot seem to get SpellRare to work in Vim, yet it works in NeoVim.

The Vim config is extremely simple:

set t_Co=16
set termguicolors
colorscheme selenized
set spelllang=en_au
setlocal spell spelllang=en_au
set spell
set nu
syntax on

My NeoVim config is the same except that it doesn't have set t_Co=16
because that's not supported.

We want the theme when running in Vim to look the same as it does in
NeoVim.

I included a screenshot:

https://user-images.githubusercontent.com/48640805/76826205-c3640200-6813-11ea-859f-e6c9a3c6e0c4.png

as well as the generated theme:

https://github.com/jan-warchol/selenized/pull/71#discussion_r401316369

I'm hoping someone can give us a hand as we really want to support Vim
too, not just NeoVim.

--
Daniel Gray (dng) 0x41911F722B0F9AE3
https://social.privacytools.io/@dngray

[Christian Brabandt]

First of all, I am not really sure what the problem is. I saw a long
discussion, but did not find a clear problem description :/

So what exactly is not working with SpellRare? Can you please provide a
reproducible example with just the required highlighting commands?

Also `:set t_Co=16` and `:set termguicolors` does not make any sense.
Either your terminal does support only 16 colors (in which case it won't
support rgb colors, or it supports rgb colors, so the `:set t_Co=16`
should not be set.

Them you shouldn't have to `:set t_Co=16` and rather rely on Vim for
determining the capabilities of the terminal rather than overriding it.
This means, you have to configure your terminal correctly (e.g. if you
export TERM=xterm, then Vim will think your terminal is only capable of
using 16 colors).

And finally, I haven't actually seen any modern terminal, that does not
support at least 256 colors. So you might want to decide not to support
16 color terminals only.

Best,
Christian

[Tony Mechelynck]

AFAIK, the Linux non-X console ($TERM == 'linux', accessed with
Ctrl-Alt-F1 to Ctrl-Alt-F6, come back with Ctrl-Alt-F7) can only do 8
background and 16 foreground colours; or maybe also 16 background
colours without the ability to blink. Of course it might be regarded
as “non-modern”, but it is still in use. I occasionally use it, even
with Vim.

Best regards,
Tony.

[Christian Brabandt]

I believe even the linux console (since v3.16) can detect and use the
rgb ansi terminal color codes, but rounds them back to 16 color codes.

Best,
Christian
--
DAU: Ich habe ein Makro geschrieben, das testet ob der Benutzer anwesend ist.
EDV: Wie das denn?
DAU: Naja, es fragt eben danach.

[Lifepillar]

On 2020-04-02, Christian Brabandt <cbl...@256bit.org> wrote:
>
> On Mi, 01 Apr 2020, Daniel Gray wrote:
>
>> Hi,
>>
>> I'm currently working on a theme called Selenized https://github.com/jan-warchol/selenized/
>>
>> We're using https://github.com/lifepillar/vim-colortemplate to template
>> our theme.
>>
>> The issue I'm having is https://github.com/jan-warchol/selenized/pull/71#pullrequestreview-375757756
>>
>> We cannot seem to get SpellRare to work in Vim, yet it works in NeoVim.

> First of all, I am not really sure what the problem is.

I think that the OP's issue amounts to `undercurl` not producing an
underline in Vim's terminal. I can reproduce it in my terminals
(macOS's Terminal.app and iTerm2):

hi SpellBad cterm=undercurl
hi SpellBad cterm=underline
hi SpellBad cterm=undercurl

(The highlight group, termguicolors, and `gui` do not really matter).

Is that intended behaviour?

Life.

[Lifepillar]

On 2020-04-03, Lifepillar <lifep...@lifepillar.me> wrote:

> in Vim's terminal

Sorry, I mean in terminal Vim. And btw, I am using Vim 8.2.0350.

Should Vim fallback to `underline` when the terminal does not support
`undercurl`?

Life.

[Dominique Pellé]

In vim :help undercurl, I see:

=== BEGIN QUOTE ===
"undercurl" is a curly underline. When "undercurl" is not possible
then "underline" is used. In general "undercurl" and "strikethrough"
is only available in the GUI.
=== END QUOTE ===

So we should check whether we at least see an underline in
the terminal.

In any case, this ticket needs a better description with
a clear reproducible case to avoid wasting time of developers.

Regards
Dominique

[Daniel Gray]

On 20/04/03 11:57AM, Dominique Pellé wrote:
>In vim :help undercurl, I see:
>
>=== BEGIN QUOTE ===
>"undercurl" is a curly underline. When "undercurl" is not possible
>then "underline" is used. In general "undercurl" and "strikethrough"
>is only available in the GUI.
>=== END QUOTE ===
>
>So we should check whether we at least see an underline in
>the terminal.
>
>In any case, this ticket needs a better description with
>a clear reproducible case to avoid wasting time of developers.
>

The reality seems to be that is not occuring in both Gnome Terminal or
Alacritty.

For example I tested this with:

hi SpellBad ctermfg=NONE ctermbg=NONE cterm=underline
hi SpellCap ctermfg=NONE ctermbg=NONE cterm=underline
hi SpellLocal ctermfg=NONE ctermbg=NONE cterm=underline
hi SpellRare ctermfg=NONE ctermbg=NONE cterm=underline

and:

hi SpellBad ctermfg=NONE ctermbg=NONE cterm=undercurl
hi SpellCap ctermfg=NONE ctermbg=NONE cterm=undercurl
hi SpellLocal ctermfg=NONE ctermbg=NONE cterm=undercurl
hi SpellRare ctermfg=NONE ctermbg=NONE cterm=undercurl

https://github.com/jan-warchol/selenized/pull/71#discussion_r403417407

This was the result (a screenshot).

https://user-images.githubusercontent.com/48640805/78417697-3814aa00-7624-11ea-8b22-6da139c012d7.png

Everything seems to work as intended in NeoVim, but not Vim :(

The undercurl does seem to work in gVim though.

[Dominique Pellé]

If I do that:

$ cat > test.vim <<EOF
set spellfile=/tmp/en.utf8.add spelllang=en_us spell t_Co=256
hi clear

hi SpellBad guisp=red gui=undercurl guifg=NONE guibg=NONE
ctermfg=NONE ctermbg=NONE term=undercurl cterm=undercurl
hi SpellCap guisp=orange gui=undercurl guifg=NONE guibg=NONE
ctermfg=NONE ctermbg=NONE term=undercurl cterm=undercurl
hi SpellRare guisp=blue gui=undercurl guifg=NONE guibg=NONE
ctermfg=NONE ctermbg=NONE term=undercurl cterm=undercurl
hi SpellLocal guisp=green gui=undercurl guifg=NONE guibg=NONE
ctermfg=NONE ctermbg=NONE term=undercurl cterm=undercurl

spellrare emacs
call setline(1, 'is your favourite edditor vim or emacs?')
EOF

$ vim -f -g --clean -S test.vim # Gives the expected undercurl in
gvim-gtk3 (good)
$ vim --clean -S test.vim # No undercurl and no underline in terminal (bug!)

I expected that vim should fallback to underline in the terminal
since the doc says:

=== BEGIN QUOTE :help undercurl ===

"undercurl" is a curly underline. When "undercurl" is not possible
then "underline" is used. In general "undercurl" and "strikethrough"
is only available in the GUI.
=== END QUOTE ===

A workaround is to use term=underline cterm=underline
instead of term=undercurl cterm=undercurl as in this example:

$ cat > test.vim <<EOF
set spellfile=/tmp/en.utf8.add spelllang=en_us spell t_Co=256
hi clear

hi SpellBad guisp=red gui=undercurl guifg=NONE guibg=NONE
ctermfg=NONE ctermbg=NONE term=underline cterm=underline
hi SpellCap guisp=orange gui=undercurl guifg=NONE guibg=NONE
ctermfg=NONE ctermbg=NONE term=underline cterm=underline
hi SpellRare guisp=blue gui=undercurl guifg=NONE guibg=NONE
ctermfg=NONE ctermbg=NONE term=underline cterm=underline
hi SpellLocal guisp=green gui=undercurl guifg=NONE guibg=NONE
ctermfg=NONE ctermbg=NONE term=underline cterm=underline

spellrare emacs
call setline(1, 'is your favourite edditor vim or emacs?')
EOF

Then it works as expected:

$ vim --clean -S test.vim # underline in terminal (good)

But still, I think that term=undercurl cterm=undercurl should
have a fall-back on underline automatically.

Regards
Dominique

[Dominique Pellé]

I've just built neovim and handling of undercurl is indeed nicer
with neovim. I can actually see colored undercurl in the terminal
(not even having to fallback to underline). I never realized
that some terminals support undercurls. Very nice! I'd love
to see this in Vim too. I see undercurls using neovim in at least
those terminals which are Ubuntu-18.04 standard packages:

- xfce4-terminal-0.8.4
- gnome-terminal-3.28.2

But I don't see undercurls with neovim using xterm (XTerm(330)).

And with konsole-17.12.3, neovim misbehave: it shows spurious
characters around words that should have undercurls.

I also noticed that neovim does not have the ":spellrare" command
which was introduced in vim-8.1.1838.

And I noticed yet a bug in vim. Neovim uses a blue undercurl
for both words "vim" and "emacs", whereas vim never show that "vim"
is a rare word. Using :spelldump, I can see indeed that "vim" is marked
as rare word in the en_us dictionary since there is a question mark in its
tag
vim/?25

So why isn't word "vim" highlighted as a rare word with spelllang=en_us
in vim gtk3?

Regards
Dominique

[Dominique Pellé]

I see that this patch was meant to make undercurl work in
terminals that support it:

===
commit 8b9e20afb0da79a21155988a38704b5777cc2efe
Author: Bram Moolenaar <Br...@vim.org>
Date: Tue Nov 28 21:25:21 2017 +0100

patch 8.0.1358: undercurl is not used in the terminal

Problem: Undercurl is not used in the terminal. (Kovid Goyal)
Solution: Only fall back to underline when undercurl highlighting is not
defined. (closes #1306)
===

But somehow it does not work: I see neither undercurl nor underline
when using vim >= 8.0.1358 in various terminals.

There was no tests added to verify 8.0.1358.

If I go back to vim-8.0.1357, then I could see at least underline being
used in the terminal where undercurl is specified.
I tried with xterm (330), xfce4-terminal 0.8.7.4 and gnome-terminal 3.28.2
and all used underlines prior with vim < 8.0.1357 as fallback to undercurl.

So patch 8.0.1358 introduced a regression.

Regards
Dominique