[patch] Add a 'modifier' keyword to the :syntax commands, allowing syntax matches to stack.

189 views
Skip to first unread message

So8res

unread,
Dec 2, 2012, 10:30:55 PM12/2/12
to vim...@googlegroups.com
Motivation:

http://stackoverflow.com/questions/13640538/vim-syntax-files-add-to-cterm

Example:

syntax region Italic start='_' end='_' modifier contains=Bold
syntax region Bold start='\*' end='\*' modifier contains=Italic

highlight Italic cterm=italic
highlight Bold cterm=bold

_this is both italic *and bold*_

The code is surprisingly simple, as vim already has a stack of syntax matches which it walks up to determine the attributes. It's trivial to add a flag which merges attributes instead of clobbering them.

I'm not sure how to add tests that test text color and format.

synmod.runtime.diff
synmod.src.diff

So8res

unread,
Dec 2, 2012, 10:47:56 PM12/2/12
to vim...@googlegroups.com
Typo in synmod.runtime.diff. I failed to notice because my dev runtime files aren't in my $VIMFILES.

Fixed below. Sorry for the noise.

synmod.runtime.diff

Bram Moolenaar

unread,
Dec 3, 2012, 2:37:24 PM12/3/12
to So8res, vim...@googlegroups.com
Hmm, I think "combine" would express better what this is doing:
combining the existing attributes with those in the current
syntax/highlight item. One can also combine a fg and bg color, but a
new fg color would overrule an existing fg color (we don't mix colors).

We don't need the patch for += and -= then, right?


--
I AM THANKFUL...
...for the piles of laundry and ironing because it means I
have plenty of clothes to wear.

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

Nate Soares

unread,
Dec 3, 2012, 2:52:01 PM12/3/12
to Bram Moolenaar, vim_dev
Hmm, I think "combine" would express better what this is doing:
combining the existing attributes with those in the current
syntax/highlight item.  One can also combine a fg and bg color, but a
new fg color would overrule an existing fg color (we don't mix colors).

SGTM
 
We don't need the patch for += and -= then, right?

Correct. If you agree, I'd like to hijack that syntax to remove the need for "removeattrs" (see the other patch I posted) and to inherit from other highlight groups, as shown below.

I played with this patch all Sunday, and the two things it left me really wanting to do were:

    highlight inherit BoldComment Comment term-=italic term+=bold

and

    syntax region Unbold start='\~' end='\~' combine
    highlight Unbold cterm-=bold

Highight-inherit should be pretty easy to implement. Allowing -= on syntax-combine groups will be a little trickier, requiring a second set of flags for attributes that have been explicitly turned off.

+= wouldn't be strictly necessary (= could be used in its place w.l.g.) but is nice to provide symmetry for -=. Another possible syntax would  be using '=' and '!', as in

    highlight Unbold cterm!bold

or perhaps

    highlight Unbold cterm~bold

or !=, ~=, or whatever you prefer. I'm indifferent to the syntax.

Regardless, both of those features can come in later patches: the 'combine' functionality is quite useful as-is. I'll update the name to 'combine' when I get home from work tonight.

Nate Soares

unread,
Dec 3, 2012, 10:12:30 PM12/3/12
to Bram Moolenaar, vim_dev
Updated modifier -> combine
combine.runtime.diff
combine.src.diff

Ingo Karkat

unread,
Dec 4, 2012, 7:01:36 AM12/4/12
to vim...@googlegroups.com
First of all, thank you for all your work on this; your initiative sounds like a
great enhancement for syntax plugins!

On 03-Dec-12 20:52:01 +0100, Nate Soares wrote:

> I played with this patch all Sunday, and the two things it left me really
> wanting to do were:
>
> highlight inherit BoldComment Comment term-=italic term+=bold

Nitpick, but I would find the following syntax more intuitive:

highlight BoldComment inherit=Comment term-=italic term+=bold

> and
>
> syntax region Unbold start='\~' end='\~' combine
> highlight Unbold cterm-=bold

Please don't forget that many plugins use matchadd() instead of :syntax to add
highlighting (without the risk of interfering with the syntax definition). I
haven't followed all your proposed changes that closely, but it might make sense
to allow the combination of syntax attributes to matchadd(), too.

-- regards, ingo

Ben Fritz

unread,
Dec 4, 2012, 11:44:45 AM12/4/12
to vim...@googlegroups.com

I'm not clear on how synIDattr() will work with combined syntax items, or synIDtrans() for that matter.

In the example you give, synIDtrans() needs to somehow return an ID for both Bold AND Italic.

Do we need to add a new function or modify synIDattr to get whether a syntax ID stands on its own or needs to be combined with the previous item on the syntax stack? Then scripts can call synstack() and check the combine attribute of each item in the stack.

Or maybe modify synIDtrans() to return a List of items if a combined item is used. Then scripts can just check each item actually used to highlight the text.

I think I like the idea of synIDtrans() returning a list in this case a little better.

I'm bringing this up because I maintain TOhtml, which needs to build CSS information for each highlight attribute. It is easy enough to combine CSS classes to obtain a combined syntax highlight, but the information to know which classes to combine needs to be available.

I do think this sounds like a useful feature.

So8res

unread,
Dec 5, 2012, 1:03:31 PM12/5/12
to vim...@googlegroups.com
I'm a bit stuck on how synID* should work in this context.

The problem is that synID() returns the id of the *highlight group*, not the syntax match. In the following example:

syntax match Foo 'foo' combine
syntax match Bar 'bar'
highlight link Foo Comment
highlight link Bar Comment

if you have the highlight id for 'Comment', I can't reliably tell you whether or not it's going to combine with attributes further up the stack.

The best I can think to do is one of:

1. Add synmatchID() and synmatchIDattr()
2. Add syncombines(line, col) which tells you if you also need to consult the rest of the synstack() for attributes.

Honestly I think that these functions are a bit misnamed and would prefer them to be 'hlID', 'hlIDattr', hlIDtrans', 'synID', 'synIDattr', and 'synIDtrans' (the *trans functions always returning lists), but that's backwards compatibility breaking.

Does anyone have preference where we go? I don't have the experience with synID* to choose the 'right' decision.

Ben Fritz

unread,
Dec 5, 2012, 2:49:53 PM12/5/12
to vim...@googlegroups.com
On Wednesday, December 5, 2012 12:03:31 PM UTC-6, So8res wrote:
> I'm a bit stuck on how synID* should work in this context.
>
> The problem is that synID() returns the id of the *highlight group*, not the syntax match. In the following example:
>
> syntax match Foo 'foo' combine
> syntax match Bar 'bar'
> highlight link Foo Comment
> highlight link Bar Comment
>
> if you have the highlight id for 'Comment', I can't reliably tell you whether or not it's going to combine with attributes further up the stack.
>

I don't understand. synIDattr(synID(...),"name") on foo returns "Foo". synIDattr(synIDtrans(synID(...)),"name") on foo returns "Comment". Can't we just add something like synIDattr(synID(...),"combine") which returns true/false? Determining the highlight would then be a matter of going up the syntax stack until a non-combining item is found, then walking back up the stack getting the attributes for the translated syntax ID of each item.

That said, doing all that in vimscript would probably be quite inefficient. The TOHtml implementation even has a comment on the loop that finds all characters in a particular highlight group, that the loop needs to be kept small. A traversal up and down the syntax stack for every character in the document would be a very bad thing. Doing the same thing for every entire highlighted item found would also probably not be very good. So I think we want a new set of functions to help out.

> The best I can think to do is one of:
>
> 1. Add synmatchID() and synmatchIDattr()

What would these do?

> 2. Add syncombines(line, col) which tells you if you also need to consult the rest of the synstack() for attributes.
>

This could work but would likewise affect the performance of a tight inner loop requiring "if" logic and a vimscript stack traversal on each character.

Syntax highlighting internally uses a unique number for each individually highlighted item. What if we add a function to return this number (synconcealed() already returns it as one item in a list for different reasons), and another function to get a list of syntax IDs which apply to an item's highlighting (in order of application), given this item number?

So8res

unread,
Dec 5, 2012, 11:47:42 PM12/5/12
to vim...@googlegroups.com
Oops, sorry, I was misunderstanding how synID* worked.

I've updated the patch to add the "combine" {what} to synIDattr, which is the bare minimum we need here.

synID should definitely return only one ID in this case, both because we don't want to duplicate synstack and because it's the only way to get interoperability with the extended links patch.

I'm still not sure how best to deal with synIDattr. How would you feel about adding a [, {flatten}] argument to it that flattens out the stack before checking the attr (analagous to [, {trans}] in synID)?

Ben Fritz

unread,
Dec 6, 2012, 9:08:00 AM12/6/12
to vim...@googlegroups.com

I like that idea, but in order for it to work it would need a row+col because the attributes for combined items depend on what they are combined with which depends on where in the document you are.

So I guess I could get the attrs flattened always at a particular row and column, then if the "combine" item shows that at least one item in the flattened view is a combining syntax item, look at the syntax stack for the names used.

The problem I see is in detecting two regions side-by-side with the same top-level (combining) syntax item but different underlying syntax stacks which therefore need different highlighting. I'd like to do this without checking the entire stack for every character in the document which I believe will be much slower than otherwise. I suppose I could keep a list of synIDs which combine and only check the stack for those but I'd still like a faster method.

Charles Campbell

unread,
Dec 6, 2012, 10:23:22 AM12/6/12
to vim...@googlegroups.com
I just am letting you know that I'm interested in this modification, too.

I have two plugins (hilinks.vim, which allows one to see which syntax
and highlighting group(s) are active under the cursor; and synchk.vim,
which is a personal plugin which compares syntax highlighting
correctness (converts syntax highlighting IDs for various test files
into line-by-line hashed codes using
synIDattr(synID(line("."),col("."),1),"name"); I can compare proposed
changes to syntax highlighting plugins with the previous highlighting of
the test files, thereby permitting some automated syntax highlight
checking)). Obviously, if synID* will be returning lists, I may need to
change these plugins to accommodate that.

I see other users of synID* : SrchRplcHiGrp.vim (David Fishburn),
foldutil.vim (Hari Krishna Dara), 2tex.vim (Francois Rigault), in my own
files. I'm certain that there are many more plugins on vim.sf.net that
may be affected.

I think that perhaps [,{flatten}] should be the default, and
[,{expanded}] give out the list.

Regards,
Chip Campbell

Nate Soares

unread,
Dec 6, 2012, 1:33:02 PM12/6/12
to vim_dev
Oh, right, of course we can't have a {flatten} arg in synIDattr without knowing the row/column.

It seems like the right thing to do here is have synID return a list of IDs being used, and have synIDattr take a list of IDs and flatten them when looking up an attribute. This has the nice property that existing code keeps working, and that flattening is the default. You can still pull any id from the list and call synIDattr on it alone if you want the "non-flattened" attribute.

The complexity comes when you try to integrate this behavior with the extended link behavior: what happens when you are using both extended links and combining syntax? My inclination is that synID creates a flat list of *all* highlight groups being used. The user can use the synIDattr(..., "combine") on each element to tell if it comes from an extended link or a syn-combine if they really need to, but in the average case they'll probably just pass the list on to synIDattr.

So8res

unread,
Dec 7, 2012, 1:03:45 AM12/7/12
to vim...@googlegroups.com, na...@so8r.es
synID now returns a list.
synIDattr now takes a list.

They both still handle the single-id case, but the code's a bit messy and the name is stale. I think we should add synActive and synActiveAttr, which take/return lists instead of single numbers, and then deprecate synID*.

combine.runtime.diff
combine.src.diff
Reply all
Reply to author
Forward
0 new messages