[vim/vim] proposal: add prop_add_list() to enable adding multiple text properties per call (#8675)
Pontus Leitzler
Is your feature request about something that is currently impossible or hard to do? Please describe the problem.
As a part of govim, a Go development plugin for vim, I'm adding support for the LSP feature "semantic tokens".
In essence it lets an external process (language server) provide syntax highlighting that is more detailed than the one provided by vim (since it is has more knowledge about the specific language in detail).
I'm using text properties to apply these highlights to a buffer, by defining a couple of different text property types (like comment, namespace or keyword; as defined by the language server protocol).
Since a file often has many different highlights, this yields many calls to prop_add(), even if we only highlight the visible part of a buffer (i.e. lines visible in a window).
Describe the solution you'd like
I'd like to be able to add multiple text properties (of a specific type?) in a single call.
So that I can, instead of calling (real life example only rendering the visible part of a buffer):
prop_add(1,1,{"type":"comment","id":2,"end_lnum":1,"end_col":55,"bufnr":1})
prop_add(2,1,{"type":"comment","id":2,"end_lnum":2,"end_col":54,"bufnr":1})
prop_add(3,1,{"type":"comment","id":2,"end_lnum":3,"end_col":50,"bufnr":1})
prop_add(5,1,{"type":"keyword","id":2,"end_lnum":5,"end_col":8,"bufnr":1})
prop_add(5,9,{"type":"namespace","id":2,"end_lnum":5,"end_col":12,"bufnr":1})
prop_add(7,1,{"type":"keyword","id":2,"end_lnum":7,"end_col":7,"bufnr":1})
prop_add(8,3,{"type":"namespace","id":2,"end_lnum":8,"end_col":8,"bufnr":1})
prop_add(9,3,{"type":"namespace","id":2,"end_lnum":9,"end_col":10,"bufnr":1})
prop_add(10,3,{"type":"namespace","id":2,"end_lnum":10,"end_col":6,"bufnr":1})
prop_add(11,6,{"type":"namespace","id":2,"end_lnum":11,"end_col":9,"bufnr":1})
prop_add(12,6,{"type":"namespace","id":2,"end_lnum":12,"end_col":11,"bufnr":1})
prop_add(13,6,{"type":"namespace","id":2,"end_lnum":13,"end_col":11,"bufnr":1})
prop_add(14,8,{"type":"namespace","id":2,"end_lnum":14,"end_col":16,"bufnr":1})
prop_add(15,3,{"type":"namespace","id":2,"end_lnum":15,"end_col":7,"bufnr":1})
prop_add(16,3,{"type":"namespace","id":2,"end_lnum":16,"end_col":10,"bufnr":1})
prop_add(17,3,{"type":"namespace","id":2,"end_lnum":17,"end_col":7,"bufnr":1})
prop_add(19,31,{"type":"namespace","id":2,"end_lnum":19,"end_col":36,"bufnr":1})
prop_add(20,35,{"type":"namespace","id":2,"end_lnum":20,"end_col":43,"bufnr":1})
prop_add(21,35,{"type":"namespace","id":2,"end_lnum":21,"end_col":41,"bufnr":1})
prop_add(22,35,{"type":"namespace","id":2,"end_lnum":22,"end_col":43,"bufnr":1})
prop_add(23,2,{"type":"namespace","id":2,"end_lnum":23,"end_col":8,"bufnr":1})
prop_add(26,1,{"type":"comment","id":2,"end_lnum":26,"end_col":84,"bufnr":1})
prop_add(27,1,{"type":"comment","id":2,"end_lnum":27,"end_col":84,"bufnr":1})
prop_add(28,1,{"type":"comment","id":2,"end_lnum":28,"end_col":64,"bufnr":1})
prop_add(30,1,{"type":"comment","id":2,"end_lnum":30,"end_col":55,"bufnr":1})
prop_add(31,1,{"type":"keyword","id":2,"end_lnum":31,"end_col":6,"bufnr":1})
prop_add(31,7,{"type":"variable","id":2,"end_lnum":31,"end_col":22,"bufnr":1})
prop_add(31,23,{"type":"type","id":2,"end_lnum":31,"end_col":26,"bufnr":1})
prop_add(31,29,{"type":"number","id":2,"end_lnum":31,"end_col":35,"bufnr":1})
prop_add(33,1,{"type":"keyword","id":2,"end_lnum":33,"end_col":5,"bufnr":1})
prop_add(33,7,{"type":"variable","id":2,"end_lnum":33,"end_col":8,"bufnr":1})
prop_add(33,9,{"type":"operator","id":2,"end_lnum":33,"end_col":10,"bufnr":1})
prop_add(33,10,{"type":"type","id":2,"end_lnum":33,"end_col":16,"bufnr":1})
prop_add(33,18,{"type":"member","id":2,"end_lnum":33,"end_col":36,"bufnr":1})
prop_add(33,37,{"type":"parameter","id":2,"end_lnum":33,"end_col":40,"bufnr":1})
prop_add(33,41,{"type":"namespace","id":2,"end_lnum":33,"end_col":48,"bufnr":1})
prop_add(33,49,{"type":"type","id":2,"end_lnum":33,"end_col":56,"bufnr":1})
prop_add(33,58,{"type":"parameter","id":2,"end_lnum":33,"end_col":59,"bufnr":1})
prop_add(33,60,{"type":"operator","id":2,"end_lnum":33,"end_col":61,"bufnr":1})
prop_add(33,61,{"type":"namespace","id":2,"end_lnum":33,"end_col":69,"bufnr":1})
prop_add(33,70,{"type":"type","id":2,"end_lnum":33,"end_col":90,"bufnr":1})
prop_add(33,93,{"type":"operator","id":2,"end_lnum":33,"end_col":94,"bufnr":1})
prop_add(33,94,{"type":"namespace","id":2,"end_lnum":33,"end_col":102,"bufnr":1})
prop_add(33,103,{"type":"type","id":2,"end_lnum":33,"end_col":117,"bufnr":1})
prop_add(33,119,{"type":"type","id":2,"end_lnum":33,"end_col":124,"bufnr":1})
prop_add(34,2,{"type":"variable","id":2,"end_lnum":34,"end_col":5,"bufnr":1})
prop_add(34,7,{"type":"variable","id":2,"end_lnum":34,"end_col":10,"bufnr":1})
prop_add(34,11,{"type":"operator","id":2,"end_lnum":34,"end_col":13,"bufnr":1})
prop_add(34,14,{"type":"variable","id":2,"end_lnum":34,"end_col":15,"bufnr":1})
prop_add(34,16,{"type":"function","id":2,"end_lnum":34,"end_col":37,"bufnr":1})
prop_add(34,38,{"type":"variable","id":2,"end_lnum":34,"end_col":41,"bufnr":1})
prop_add(34,43,{"type":"variable","id":2,"end_lnum":34,"end_col":44,"bufnr":1})
prop_add(34,45,{"type":"variable","id":2,"end_lnum":34,"end_col":57,"bufnr":1})
prop_add(34,59,{"type":"variable","id":2,"end_lnum":34,"end_col":62,"bufnr":1})
I could call something like (just an example, might be better ways to design the parameters) :
prop_add_list({"type":"command":"id":2,"bufnr":1}, [1,1,1,55], [2,1,2,54], [3,1,3,50], [26,1,26,84], [27,1,27,84], [28,1,28,64], [30,1,30,55])
prop_add_list({"type":"keyword":"id":2,"bufnr":1}, [5,1,5,8], [7,1,7,7], [31,1,31,6], [33,1,33,5])
prop_add_list({"type":"namespace":"id":2,"bufnr":1}, [5,9,5,12], [8,3,8,8], [9,3,9,10], [10,3,10,6], [11,6,11,9], [12,6,12,11], [13,6,13,11], [14,8,14,16], [15,3,15,7], [16,3,16,10], [17,3,17,7], [19,31,19,36], [20,35,20,43], [21,35,21,41], [22,35,22,43], [23,2,23,8], [33,41,33,48], [33,61,33,69], [33,94,33,102])
prop_add_list({"type":"variable":"id":2,"bufnr":1}, [31,7,31,22], [33,7,33,8], [34,2,34,5], [34,7,34,10], [34,14,34,15], [34,38,34,41], [34,43,34,44], [34,45,34,57], [34,59,34,62])
prop_add_list({"type":"type":"id":2,"bufnr":1}, [31,23,31,26], [33,10,33,16], [33,49,33,56], [33,70,33,90], [33,103,33,117], [33,119,33,124])
prop_add_list({"type":"number":"id":2,"bufnr":1}, [31,29,31,35])
prop_add_list({"type":"operator":"id":2,"bufnr":1}, [33,9,33,10], [33,60,33,61], [33,93,33,94], [34,11,34,13])
prop_add_list({"type":"member":"id":2,"bufnr":1}, [33,18,33,36])
prop_add_list({"type":"parameter":"id":2,"bufnr":1}, [33,37,33,40], [33,58,33,59])
prop_add_list({"type":"function":"id":2,"bufnr":1}, [34,16,34,37])
The nature of syntax highlighting requires a lot of text properties added (and removed if you only want to show the visible lines).
Describe alternatives you've considered
None, I currently call prop_add multiple times each time the highlight changes. And it could be often as the user types or scrolls through the code in a buffer.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Martin Tournoij
I'm using the Lua LPeg module to apply highlights with text properties (unreleased plugin I've been working on this week), and I'm essentially doing the same kind of prop_add() calls.
I'm not sure if it's really a practical issue as such? As far as I can tell it's plenty fast enough.
Aside: the biggest issue I ran in to is that length can't span multiple lines and that I need to do some stuff to transform byte offsets to line/col; there's probably more overhead in all of that.
At any rate, the downside of the proposed API is that you first need to group properties by type, rather than by position. I think the latter is probably more common?
Yegappan Lakshmanan
Is your feature request about something that is currently impossible or hard to do? Please describe the problem.
As a part of govim, a Go development plugin for vim, I'm adding support for the LSP feature "semantic tokens".
In essence it lets an external process (language server) provide syntax highlighting that is more detailed than the one provided by vim (since it is has more knowledge about the specific language in detail).
I'm using text properties to apply these highlights to a buffer, by defining a couple of different text property types (like
comment,namespaceorkeyword; as defined by the language server protocol).Since a file often has many different highlights, this yields many calls to
prop_add(), even if we only highlight the visible part of a buffer (i.e. lines visible in a window).Describe the solution you'd like
I'd like to be able to add multiple text properties (of a specific type?) in a single call.So that I can, instead of calling (real life example only rendering the visible part of a buffer):
prop_add(1,1,{"type":"comment","id":2,"end_lnum":1,"end_col":55,"bufnr":1})
I could call something like (just an example, might be better ways to design the parameters) :
prop_add_list({"type":"command":"id":2,"bufnr":1}, [1,1,1,55], [2,1,2,54], [3,1,3,50], [26,1,26,84], [27,1,27,84], [28,1,28,64], [30,1,30,55])
vim-dev ML
On Fri, Jul 30, 2021 at 11:39 AM Pontus Leitzler ***@***.***>
wrote:
> *Is your feature request about something that is currently impossible or
> hard to do? Please describe the problem.*
>
> As a part of govim <http://github.com/govim/govim>, a Go development
> plugin for vim, I'm adding support for the LSP feature "semantic tokens"
> .
>
> In essence it lets an external process (language server) provide syntax
> highlighting that is more detailed than the one provided by vim (since it
> is has more knowledge about the specific language in detail).
>
> I'm using text properties to apply these highlights to a buffer, by
> defining a couple of different text property types (like comment,
> namespace or keyword; as defined by the language server protocol).
>
> Since a file often has many different highlights, this yields many calls
> to prop_add(), even if we only highlight the visible part of a buffer
> (i.e. lines visible in a window).
>
> I'd like to be able to add multiple text properties (of a specific type?)
> in a single call.
>
> So that I can, instead of calling (real life example only rendering the
> visible part of a buffer):
>
> prop_add(1,1,{"type":"comment","id":2,"end_lnum":1,"end_col":55,"bufnr":1})
>
>
> design the parameters) :
>
> prop_add_list({"type":"command":"id":2,"bufnr":1}, [1,1,1,55], [2,1,2,54], [3,1,3,50], [26,1,26,84], [27,1,27,84], [28,1,28,64], [30,1,30,55])
>
>
arguments accepted by prop_add(), will that work? i.e.
prop_add_list([[1,1 {'type': 'comment'}], [2, 1, {'type': 'variable'}],
[26, 1, {'type': 'variable'}]])
- Yegappan
> added (and removed if you only want to show the visible lines).
>
Pontus Leitzler
I'm not sure if it's really a practical issue as such? As far as I can tell it's plenty fast enough.
Aside: the biggest issue I ran in to is that
lengthcan't span multiple lines and that I need to do some stuff to transform byte offsets to line/col; there's probably more overhead in all of that.At any rate, the downside of the proposed API is that you first need to group properties by type, rather than by position. I think the latter is probably more common?
Thanks @arp242 for your comment! Let me add some more context. Even though performance is key when doing syntax highlighting, it isn't necessarily the pure invokation speed of one vs. multiple calls the proposal aims to improve.
The govim plugin uses a vim channel to communicate, so each call renders a network roundtrip & JSON parsing. We also want to express that each call should be surrounded with try/catch if we, for example, would like to suppress E964/E966 (invalid col/line).
Those things are mostly implementation details of my particular use case, and we already batch multiple calls into a single roundtrip for example.
I do however think that having a prop_add_list() make sense for others as well. As my example above shows there is a lot of redundant information that needs to be passed as arguments when using a single call per text property. If you, instead of applying highlights to the visible lines only, applies highlights to the entire buffer you could end up with several kilobytes of redundant "type":"foobar","bufnr":1,"id":2.
The arguments to prop_add_list() doesn't have to be as suggested above, the goal is just to have something that better fits the purpose of using text properties as syntax highlight replacement. In that use case you often get a bunch of highlights to apply to a single buffer where there is a limited set of different types.
That maybe answers your question too @yegappan:
If the argument to the new prop_add_list() function is a List of the
arguments accepted by prop_add(), will that work?
Such list would be an improvement to the number of calls, but we would still have all the overhead of having to define multiple type, bufnr and id for example. I don't think it brings enough benefits to be worth adding a new call? (i.e. prop_add_list()).
—
You are receiving this because you commented.
Yegappan Lakshmanan
vim-dev ML
On Sat, Jul 31, 2021 at 1:08 AM Pontus Leitzler ***@***.***>
wrote:
> I'm not sure if it's really a practical issue as such? As far as I can
> tell it's plenty fast enough.
>
> Aside: the biggest issue I ran in to is that length can't span multiple
> lines and that I need to do some stuff to transform byte offsets to
> line/col; there's probably more overhead in all of that.
>
> At any rate, the downside of the proposed API is that you first need to
> group properties by type, rather than by position. I think the latter is
> probably more common?
>
> some more context. Even though performance is key when doing syntax
> highlighting, it isn't necessarily the pure invokation speed of one vs.
> multiple calls the proposal aims to improve.
>
> The govim plugin uses a vim channel to communicate, so each call renders a
> network roundtrip & JSON parsing. We also want to express that each call
> should be surrounded with try/catch if we, for example, would like to
> suppress E964/E966 (invalid col/line).
>
> Those things are mostly implementation details of my particular use case,
> and we already batch multiple calls into a single roundtrip for example.
>
> I do however think that having a prop_add_list() make sense for others as
> well. As my example above shows there is a lot of redundant information
> that needs to be passed as arguments when using a single call per text
> property. If you, instead of applying highlights to the visible lines only,
> applies highlights to the entire buffer you could end up with several
> kilobytes of redundant "type":"foobar","bufnr":1,"id":2.
>
> The arguments to prop_add_list() doesn't have to be as suggested above,
> the goal is just to have something that better fits the purpose of using
> text properties as syntax highlight replacement. In that use case you often
> get a bunch of highlights to apply to a single buffer where there is a
> limited set of different types.
>
> That maybe answers your question too @yegappan
>
> If the argument to the new prop_add_list() function is a List of the
> arguments accepted by prop_add(), will that work?
>
> Such list would be an improvement to the number of calls, but we would
> still have all the overhead of having to define multiple type, bufnr and
> id for example. I don't think it brings enough benefits to be worth
> adding a new call? (i.e. prop_add_list()).
>
>
>
If prop_add_list() takes a List of the prop_add() arguments, then you
can use a single call to set all the text properties (instead of multiple
calls to prop_add_list() in your example). So I think the reduction in
the number of calls is a better tradeoff than passing less information
per call.
Regards,
Yegappan
Bram Moolenaar
I'm not against it. The second argument needs to be a list of lists though.
It appears all these are within a single line. We could just use [lnum, col, endcol].
In case a property spans more than one line, the prop_add() function can be used.
How do you remove the properties? I guess not by removing each individual property.
—
You are receiving this because you commented.
Pontus Leitzler
I'm not against it. The second argument needs to be a list of lists though.
It appears all these are within a single line. We could just use [lnum, col, endcol].
In case a property spans more than one line, the prop_add() function can be used.
Thanks for the input. Good idea with lnum, col, endcol, I haven't seen any semantic tokens yet that spawn multiple lines so I'm all in favour of falling back to prop_add() for multiline proprties.
How do you remove the properties? I guess not by removing each individual property.
Correct, I remove them via a single prop_remove using id where semantic token related highlights share the same id in the plugin.
—
You are receiving this because you commented.
bfrg
I think this can be closed. prop_add_list() was added in ccfb7c6.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
Bram Moolenaar
Closed #8675.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.