[patch] Problem with syntax highlighting and "skipwhite"

[Christian Brabandt]

Bram,

a popular questions at Stackoverflow covers the topic how to display
whitespace distinct so that it is easily visible where whitespace is.
Since starting with Vim 7.3 Concealing became available, my answer to
that problem was this:

,----
| syn match WhiteSpace / / containedin=ALL conceal cchar=·
| setl conceallevel=2 concealcursor=nv
`----

That should theoretically match everywhere, but I noticed it doesn't. I
have digged into the syntax highlighting rules and apparently, this is
caused by the "skipwhite" argument in the syntax files. The help however
states this:

,----[ :h syn-skipwhite ]-
| When "skipwhite" is present, the white space is only skipped if there
| is no next group that matches the white space.
`----

So I would argue, that a syntax rule, that has the containedin argument,
should also apply at that space. The patch seems relativly simple:

diff --git a/src/syntax.c b/src/syntax.c
--- a/src/syntax.c
+++ b/src/syntax.c
@@ -2275,12 +2275,14 @@ syn_current_attr(syncing, displaying, ca
* If a nextgroup was not found, continue looking for one
if:
* - this is an empty line and the "skipempty" option was
given
* - we are on white space and the "skipwhite" option was
given
+ * (but allow containedin groups to still match that)
*/
if (!found_match)
{
line = syn_getcurline();
if (((current_next_flags & HL_SKIPWHITE)
- && vim_iswhite(line[current_col]))
+ && vim_iswhite(line[current_col])
+ && !syn_block->b_syn_containedin)
|| ((current_next_flags & HL_SKIPEMPTY)
&& *line == NUL))
break;

Attached is also a screenshot, that shows the difference.

Best,
Christian

[Bram Moolenaar]

Hmm, a nextgroup is something else than containedin. I suspect this
might break some syntaxes in obscure ways. I think consistency is more
important here, the syntax highlighting already is too complex.

Why not use a match to show whitespace?

--
Not too long ago, a program was something you watched on TV...

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Christian Brabandt]

Hi Bram!

It has nothing to do with a nextgroup. It depends on the skipwhite
keyword which prevents the containedin item to match at that particular
point.

> I suspect this might break some syntaxes in obscure ways. I think
> consistency is more important here, the syntax highlighting already is
> too complex.

I agree, syntax highlighting is already so complex, that I usually try
to avoid it as much as possible. Unfortunately there is no really a way
around that.

> Why not use a match to show whitespace?

Because a simple highlighting is too invasive (too much whitespace gets
suddenly highlighted). If I could use the match() familiy to conceal
items, that would actually a lot easily, but unfortunately you can only
use syntax highlighting if you want to actually conceal items. I even
made a patch¹ several years ago to allow hiding matches (though it
wasn't possible to set the cchar with that), but you didn't like it.

¹) https://groups.google.com/d/msg/vim_dev/8bKa98GhHdk/VOzIBhd1m8YJ

Best,
Christian

[Christian Brabandt]

Bram,
this problem is not mentioned in the latest todo list. Is that no issue
for your?

Best,
Christian
--
So lange man lieset, besinnt man sich auf all(es), nur nicht auf sich.
-- Jean Paul

[Bram Moolenaar]

> this problem is not mentioned in the latest todo list. Is that no issue
> for your?

As I already mentioned: It might break some things.

The flag does apply to "nextgroup", as documented. That's why it checks
current_next_flags.

Also, the rules become too complex, making a difference for a group that
has a "containedin" argument changes how the white space before it
matches. That's not what someone expects.

--
The early bird gets the worm. If you want something else for
breakfast, get up later.

[Christian Brabandt]

On Mi, 05 Nov 2014, Bram Moolenaar wrote:

> > this problem is not mentioned in the latest todo list. Is that no issue
> > for your?
>
> As I already mentioned: It might break some things.

Currently, this would not affect many syntax files:
#v+
chrisbra@debian ~/code/vim/runtime/syntax - % grep -i 'containedin=ALL\>' *
ada.vim: syntax match adaLineError "\(^.\{79}\)\@<=." contains=ALL containedin=ALL
cobol.vim: syn match cobolBadLine "\%73c.*" containedin=ALL
desktop.vim:syn match dtALocale /\[.\{-}\]\s*=\@=/ containedin=ALL
dnsmasq.vim:syn match DnsmasqTrailSpace "[ \t]\+$" containedin=ALL
rcs.vim:syn match rcsEOFError ".\%$" containedin=ALL
redif.vim:syntax match redifComment /^#.*/ containedin=ALL display
#v-

>
> The flag does apply to "nextgroup", as documented. That's why it checks
> current_next_flags.
>
> Also, the rules become too complex, making a difference for a group that
> has a "containedin" argument changes how the white space before it
> matches. That's not what someone expects.

Well my expectation is differently. If I specify containedin=ALL I
usually want it to match everywhere. Even if some syntax rules specifies
a skipwhite rule.

In any case, it would be nice to have some possibility to match some
tokens and have them concealed. Either through the use of syntax
highlighting or through the matchadd() functions (which do not allow to
conceal characters currently) or by the use of the 'listchars' option,
for which a patch has also been posted
https://groups.google.com/d/msg/vim_dev/dIQHjW1g92s/CtBo6MfNXN4J
(but which only applies to space characters and doesn't allow to conceal
arbitrary characters)


Best,
Christian

[Christian Brabandt]

On Mi, 05 Nov 2014, Bram Moolenaar wrote:

> > this problem is not mentioned in the latest todo list. Is that no issue
> > for your?
>
> As I already mentioned: It might break some things.
>
> The flag does apply to "nextgroup", as documented. That's why it checks
> current_next_flags.
>
> Also, the rules become too complex, making a difference for a group that
> has a "containedin" argument changes how the white space before it
> matches. That's not what someone expects.

How about the current patch. This doesn't touch syntax highlighting.
This is an enhanced version of the old matchadd_conceal patch from
https://groups.google.com/d/msg/vim_dev/8bKa98GhHdk/VOzIBhd1m8YJ
The difference is, by default matchadd('Conceal', '/pattern/') works as
is, but you can add a special dictionary that allows to specify a custom
cchar value for that particular concealed match.

So use matchadd('Conceal', ' ', 10, -1, {'conceal' "\u2d1"})
to have spaces highlighted with 'ˑ'

This has the advantage of being backwards compatible, while still
allowing to fine tune each separate Conceal match and also allows for
future extensions, so one could theoretically add other values to the
dictionary.

If we could agree to merge this version, we also do not need the
space_listchars patch
(https://groups.google.com/d/msg/vim_dev/dIQHjW1g92s/CtBo6MfNXN4J) as it
could be achieved by using matching functions.

The patch includes a new test to make sure matchadd and matchpos do work
as expected, it also tests the getmatches() and setmatches() function as
well as working correctly together with syntax highlightinG. I have
tried to test all different things I could imagine.

Best,
Christian
--
Gern lesen heißt, die einem im Leben zugeteilten Stunden der
Langeweile gegen solche des Entzückens einzutauschen.
-- Charles-Louis de Montesquieu

[Christian Brabandt]

Slightly Updated patch available at
https://github.com/chrisbra/vim-mq-patches/blob/master/match_conceal

Update includes small update to the test (mainly to make it work with
dumb terminals for travis-ci, and add an unused parameter to prevent a
warning for non-concealed builds.)

Best,
Christian
--
Gunst als Symbol der Souveränität, von schwachen Menschen
ausgeübt.
-- Goethe, Maximen und Reflektionen, Nr. 254