syntax match bug: 'keepend' not officially supported, but required in some cases

[Brett Stahlman]

The syntax documentation suggests that the 'keepend' argument applies
only to syntax regions, not syntax matches. But if I highlight a
buffer containing only the following line...

ABCDE

...with the following syntax definitions...

syn match A /A/
syn match Inside /[A-Z]\+/ transparent contained containedin=A
hi A guifg=blue

...the match group "A" extends (apparently incorrectly) from A through
E, though its pattern matches only the "A". Stranger still, one
additional nested "Inside" group is recognized in each successive
letter. I.e., "Inside" is contained within "Inside", in spite of the
`containedin=A'. Thus, running synstack on successive characters, I
see...

-- cursor on A --
A
Inside
-- cursor on B --
A
Inside
Inside
-- cursor on C --
A
Inside
Inside
Inside


I can fix the problem by adding the 'keepend' argument to the
definition for match group "A", but the "keepend" is highlighted as an
error by the vim syntax, and the help on :syn-match gives me no reason
to believe the argument should be supported.

Here are the results of synstack() with the following, "fixed" definition for A:
syn match A /A/ keepend

-- cursor on A --
A
Inside
-- cursor on B --
-- cursor on C --

Is 'keepend' meant to be supported for syn-match? The purpose of
'keepend' is to prevent a nested group from obscuring a match with a
containing group's "end pattern". Although a match group has no
explicit end pattern, the principle of checking for the end of the
match pattern certainly still applies, so I suppose it makes sense
that 'keepend' would be accepted for both syn-match and syn-region. If
that is the intent, then the documentation needs fixing.

As for the infinitely nested "Inside" groups, I can't account for
that, and suspect it's a bug, but perhaps I'm misunderstanding
something about contained and containedin?

Thanks,
Brett Stahlman

[Bram Moolenaar]

Brett Stahlman wrote:

> The syntax documentation suggests that the 'keepend' argument applies
> only to syntax regions, not syntax matches. But if I highlight a
> buffer containing only the following line...
>
> ABCDE
>
> ...with the following syntax definitions...
>
> syn match A /A/
> syn match Inside /[A-Z]\+/ transparent contained containedin=A
> hi A guifg=blue
>
> ...the match group "A" extends (apparently incorrectly) from A through
> E, though its pattern matches only the "A".

This is correct, the inner match continues, causing the outer match not
to end. :syn match does not have "keepend" but that doesn't mean it
always keeps its end.

> Stranger still, one
> additional nested "Inside" group is recognized in each successive
> letter. I.e., "Inside" is contained within "Inside", in spite of the
> `containedin=A'. Thus, running synstack on successive characters, I
> see...
>
> -- cursor on A --
> A
> Inside
> -- cursor on B --
> A
> Inside
> Inside
> -- cursor on C --
> A
> Inside
> Inside
> Inside

That looks like a bug.

> I can fix the problem by adding the 'keepend' argument to the
> definition for match group "A", but the "keepend" is highlighted as an
> error by the vim syntax, and the help on :syn-match gives me no reason
> to believe the argument should be supported.
>
> Here are the results of synstack() with the following, "fixed" definition for A:
> syn match A /A/ keepend
>
> -- cursor on A --
> A
> Inside
> -- cursor on B --
> -- cursor on C --
>
> Is 'keepend' meant to be supported for syn-match? The purpose of
> 'keepend' is to prevent a nested group from obscuring a match with a
> containing group's "end pattern". Although a match group has no
> explicit end pattern, the principle of checking for the end of the
> match pattern certainly still applies, so I suppose it makes sense
> that 'keepend' would be accepted for both syn-match and syn-region. If
> that is the intent, then the documentation needs fixing.
>
> As for the infinitely nested "Inside" groups, I can't account for
> that, and suspect it's a bug, but perhaps I'm misunderstanding
> something about contained and containedin?

Something seems wrong. Also try this:

syn match A /A/
syn match Inside /[A-Z]/ transparent contained containedin=A
hi A guifg=green ctermfg=green

Now every fourth character is highlighted. Hmm.


--
Facepalm statement #8: "Drive faster, the petrol is running out"

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Brett Stahlman]

On Mon, Sep 19, 2016 at 9:05 AM, Bram Moolenaar <Br...@moolenaar.net> wrote:
>
> Brett Stahlman wrote:
>
>> The syntax documentation suggests that the 'keepend' argument applies
>> only to syntax regions, not syntax matches. But if I highlight a
>> buffer containing only the following line...
>>
>> ABCDE
>>
>> ...with the following syntax definitions...
>>
>> syn match A /A/
>> syn match Inside /[A-Z]\+/ transparent contained containedin=A
>> hi A guifg=blue
>>
>> ...the match group "A" extends (apparently incorrectly) from A through
>> E, though its pattern matches only the "A".
>
> This is correct, the inner match continues, causing the outer match not
> to end. :syn match does not have "keepend" but that doesn't mean it
> always keeps its end.

But syn-match *does* have 'keepend' - at least, it seems to be
respecting it (though as I mentioned, neither the docs nor the vim
syntax seem to think it should). Without 'keepend', group "A" extends
until "Inside" ends, at which point, "A" ends too. It's as though the
syntax engine knows A's pattern match has ended, but the presence of a
contained group is keeping the containing group active. I added
'keepend' because I wanted the end of "A" to determine the end of
"Inside", not the other way around, and the 'keepend' argument to
syn-match does indeed give the desired behavior. In fact, not having a
way to specify this behavior would be a real limitation if contained
groups can extend the containing syn-match group. Is :syn match's
current handling of 'keepend' not by design? I can fix the issue I was
having by using 'keepend', but don't want to do so if syn-match is
honoring 'keepend' only by accident...

Thanks,
Brett Stahlman

[Bram Moolenaar]

Yes, the "keepend" actually works, although it's not documented. I'll
add a note in the documentation.

> Here are the results of synstack() with the following, "fixed" definition for A:
> syn match A /A/ keepend
>
> -- cursor on A --
> A
> Inside
> -- cursor on B --
> -- cursor on C --
>
> Is 'keepend' meant to be supported for syn-match? The purpose of
> 'keepend' is to prevent a nested group from obscuring a match with a
> containing group's "end pattern". Although a match group has no
> explicit end pattern, the principle of checking for the end of the
> match pattern certainly still applies, so I suppose it makes sense
> that 'keepend' would be accepted for both syn-match and syn-region. If
> that is the intent, then the documentation needs fixing.
>
> As for the infinitely nested "Inside" groups, I can't account for
> that, and suspect it's a bug, but perhaps I'm misunderstanding
> something about contained and containedin?

This is actually working as intended. When using a transparent item,
checking for contained items ignores that transparent item. So the
"containedin=A" applies every time.

This can be avoided by add "contains=NONE to the Inside item:

syn match Inside /[A-Z]\+/ transparent contained containedin=A contains=NONE

--
A mathematician is a device for turning coffee into theorems.
Paul Erdos
A computer programmer is a device for turning coffee into bugs.
Bram Moolenaar

[Brett Stahlman]

Thanks.

>
>> Here are the results of synstack() with the following, "fixed" definition for A:
>> syn match A /A/ keepend
>>
>> -- cursor on A --
>> A
>> Inside
>> -- cursor on B --
>> -- cursor on C --
>>
>> Is 'keepend' meant to be supported for syn-match? The purpose of
>> 'keepend' is to prevent a nested group from obscuring a match with a
>> containing group's "end pattern". Although a match group has no
>> explicit end pattern, the principle of checking for the end of the
>> match pattern certainly still applies, so I suppose it makes sense
>> that 'keepend' would be accepted for both syn-match and syn-region. If
>> that is the intent, then the documentation needs fixing.
>>
>> As for the infinitely nested "Inside" groups, I can't account for
>> that, and suspect it's a bug, but perhaps I'm misunderstanding
>> something about contained and containedin?
>
> This is actually working as intended. When using a transparent item,
> checking for contained items ignores that transparent item. So the
> "containedin=A" applies every time.
>
> This can be avoided by add "contains=NONE to the Inside item:
>
> syn match Inside /[A-Z]\+/ transparent contained containedin=A contains=NONE

Hmm... I see it now:
"The "contains=" argument is also inherited from the item it is contained in..."

Excluding the transparent item itself from the inherited list might
have produced less surprising behavior, but I guess this is sort of an
uncommon corner case, and I agree that Vim's behavior is consistent
with the documentation.

Thanks for looking into it.
Brett Stahlman

[Charles E Campbell]

Hello:

Since the help will be changed to reflect keepend being allowed at the
end of a match... well, try
http://www.drchip.org/astronaut/vim/index.html#SYNTAX_VIM for an update
to the highlighting thereof.

Regards,
Chip Campbell