[vim/vim] feat: treesitter integration (PR #18869)
Foxe Chen
This PR doesn't actually do anything useful for the end user, all it does is just expose the treesitter C API through vimscript, which is what Neovim does except they use Lua. I suppose the end game plan is to create a vim9 script pack plugin that uses the API to actually do useful things.
To expose the treesitter API to vimscript, I created a new type called "opaque", which is similar to Lua's userdata if you know what that is. All they do is just bring reference counting to a block of memory + I also added properties to them (which are essentially the same as class members). They have full type checking at both runtime and compile time (for vim9 script), including the properties.
I haven't added tests yet so be mindful that it may be a little buggy/segfaulty.
Here's a demo script that highlights string literals using the C parser (just run g:RunParse(<buffer number>), also ignore the red highlighting on the function names, the syntax files have not been updated):
vim9script ts_load("c", "/usr/lib/tree_sitter/c.so") # Update to where the parser library is var parser: opaque<TSParser> = tsparser_new() tsparser_set_language(parser, "c") prop_type_add('literal', {'highlight': 'ErrorMsg'}) def g:RunParse(buf: number): opaque<TSQueryCursor> var res: opaque<TSTree> # There is a timeout limit for parsing (here 3 milliseconds), meaning we can implement async parsing: while true var tree: opaque<TSTree> = tsparser_parse_buf(parser, buf, 3) if tree == null_opaque continue endif res = tree break endwhile var node: opaque<TSNode> = tstree_root_node(res) var querystr: list<string> =<< trim END (string_literal) @string END var query: opaque<TSQuery> = tsquery_new("c", join(querystr, "\n")) var cursor: opaque<TSQueryCursor> = tsquerycursor_new() tsquerycursor_exec(cursor, query, node) while tsquerycursor_next_match(cursor) for i in range(0, len(cursor.match_captures) - 1) var capture: opaque<TSNode> = cursor.match_captures[i][0] var start: tuple<number, number> = capture.start_point var end: tuple<number, number> = capture.end_point prop_add(start[0] + 1, start[1] + 1, { type: 'literal', end_lnum: end[0] + 1, end_col: end[1] + 1 }) endfor endwhile return cursor enddef
TODO:
- Add ability to load WASM parser modules
- Add tests
- Add more API functions
You can view, comment on, or merge this pull request online at:
https://github.com/vim/vim/pull/18869
Commit Summary
- d2d92c6 initial commit
- 82e3a03 setup basic stuff
- 33cf973 refactor
- 01b94b1 iniitial parsing backend logic
- 4fae487 refactor
- 6eee682 basic working
- 0f4b855 initial vimscript api
- 5ee69f2 add more api functions
- 286fd77 add opaque data type to replace tsobject
- 29de65f port over to using opaques
- c83071d add more api functions
- 1bbfedc refactor
- 953afae add properties to opaque type
- e840e92 fix possible alignment issues
- cd08633 improve api
- 26c49b2 refactor
File Changes
(41 files)
- M src/Makefile (19)
- M src/auto/configure (80)
- M src/buffer.c (3)
- M src/config.h.in (3)
- M src/config.mk.in (6)
- M src/configure.ac (39)
- M src/dict.c (23)
- M src/errors.h (44)
- M src/eval.c (56)
- M src/evalfunc.c (163)
- M src/evalvars.c (3)
- M src/feature.h (11)
- M src/gc.c (1)
- M src/globals.h (218)
- M src/json.c (1)
- M src/main.c (7)
- A src/opaque.c (198)
- M src/option.c (7)
- M src/option.h (6)
- M src/optiondefs.h (12)
- M src/optionstr.c (3)
- M src/proto.h (4)
- M src/proto/dict.pro (1)
- A src/proto/opaque.pro (7)
- M src/proto/testing.pro (2)
- A src/proto/treesitter.pro (18)
- M src/proto/typval.pro (6)
- M src/proto/vim9instr.pro (3)
- M src/structs.h (55)
- M src/testing.c (85)
- A src/treesitter.c (1481)
- M src/typval.c (145)
- M src/vim.h (4)
- M src/vim9.h (13)
- M src/vim9compile.c (3)
- M src/vim9execute.c (121)
- M src/vim9expr.c (56)
- M src/vim9instr.c (92)
- M src/vim9script.c (1)
- M src/vim9type.c (169)
- M src/viminfo.c (1)
Patch Links:
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- e45791c refactor
—
View it on GitHub or unsubscribe.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 85d4c2c refactor
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- b926151 fix formatting
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- f41ece7 small fixes
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- b68cdb9 small fixes
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- ab0a676 small fixes
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- a869a02 small fixes
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 51013cd small fixes
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 5d14a86 small fixes
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- c2c7e0c small fixes
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- e44b17f small fixes
You are receiving this because you are subscribed to this thread.
Dominique Pelle
@DominiquePelle-TomTom commented on this pull request.
In src/vim9expr.c:
> + */
+ static int
+compile_opaque_index(cctx_T *cctx, char_u **arg, type_T *type)
+{
+ opaque_type_T *ot;
+ opaque_property_T *prop;
+ char_u *name;
+ char_u *name_end;
+ size_t len;
+ int idx;
+
+ if (VIM_ISWHITE((*arg)[1]))
+ {
+ semsg(_(e_no_white_space_allowed_after_str_str), ".", *arg);
+ return FAIL;
+ }
Indentation does not look right.
The same remark applies to several other places in this file and other files.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman commented on this pull request.
In src/vim9expr.c:
> + */
+ static int
+compile_opaque_index(cctx_T *cctx, char_u **arg, type_T *type)
+{
+ opaque_type_T *ot;
+ opaque_property_T *prop;
+ char_u *name;
+ char_u *name_end;
+ size_t len;
+ int idx;
+
+ if (VIM_ISWHITE((*arg)[1]))
+ {
+ semsg(_(e_no_white_space_allowed_after_str_str), ".", *arg);
+ return FAIL;
+ }
That is just github being weird with displaying code. It looks fine viewed otherwise
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 320fa1f small fixes
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 2b9ff2d add decode function for parsing
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 6d5ca9b add decode function for parsing
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 8373842 add decode function for parsing
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 72e3c18 fix build errros
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- e820967 fix ci errors
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 4df81f9 fix ci errors
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- f0f0353 fix ci errors
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 4f76e4a fix ci errors
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 59b93ce build treesitter library instead of distro pkg for ci
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- d59edd2 build treesitter library instead of distro pkg for ci
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 7b36492 build treesitter library instead of distro pkg for ci
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 65faa4d build treesitter library instead of distro pkg for ci
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- b64106c build treesitter library instead of distro pkg for ci
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 7fc285c build treesitter library instead of distro pkg for ci
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 2 commits.
You are receiving this because you are subscribed to this thread.
ubaldot
ubaldot left a comment (vim/vim#18869)So far I remember there have been discussions around treesitter and if I remember correctly it was a no-no. But I am aging badly and I may remember it wrongly.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Foxe Chen
64-bitman left a comment (vim/vim#18869)So far I remember there have been discussions around treesitter and if I remember correctly it was a no-no. But I am aging badly and I may remember it wrongly.
Treesitter has improved greatly since then.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 2 commits.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 372731f init decode callback to NULL
You are receiving this because you are subscribed to this thread.
Foxe Chen
64-bitman left a comment (vim/vim#18869)@chrisbra Would you agree to adding the treesitter library as a submodule? Treesitter packages for non rolling linux distros are too old (treesitter development moves fast) and we would have to statically link it for Windows I'm pretty sure. Or we could include the treesitter source code directly in the repo, but I don't think that is the best way. Thanks.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Clason
clason left a comment (vim/vim#18869)I could say something about tree-sitter development but will refrain from doing so here. My unsolicited (but hard-won) advice would be to
- Avoid submodules at all cost.
- Pin and statically link treesitter into Vim. (Tree-sitter uses semver but is at zerover, so patch updates should be safe but minor updates may be breaking.)
- Be prepared that distros will care little about what you do and link Vim against their shared system tree-sitter library at all costs.
@chrisbra In case you want to proceed with this; feel free to ping me with questions. I have some experience with the tree-sitter ecosystem...
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
chrisbra left a comment (vim/vim#18869)I'll take any advice I can 🙇♂️
I'd like to avoid submodules if possible, and your suggestions to link treesitter statically sounds good to me. Other than that, I am expecting a lot of issues from packaging side, especially on Windows systems and more exotic ones (like OpenVMS).
But I am basically blank with regard to tree-sitter. So any other suggestions from the other maintainers and power users are more than welcome!
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Christian Clason
clason left a comment (vim/vim#18869)Tree-sitter is battle-tested on Windows (although you may need CMake for that -- a fate worse than death; maybe Zig would be a more palatable approach due to its strong C interop). More "exotic" platforms are an issue, though; these are not officially supported, and no promises about the compiler requirements.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
BenYip
bennyyip left a comment (vim/vim#18869)Treesitter has improved greatly since then.
Can you elaborate this?
I am expecting a lot of issues from packaging side, especially on Windows systems and more exotic ones (like OpenVMS).
Treesitter is a C library with no external deps. They officially have a makefile that does amalgamated build. I suppose it's not very difficult to integrate.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
BenYip
bennyyip left a comment (vim/vim#18869)I don't agree with the "disadvantages" skywind3000 gave at the first place. Some of them are objectively false.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 25 commits.
- 027cad2 initial commit
- 474389b setup basic stuff
- 4cf9d11 refactor
- f1fcd60 iniitial parsing backend logic
- 6ca04f4 refactor
- 288cf7e basic working
- fc1d0d1 initial vimscript api
- 6c832b1 add more api functions
- 29e3643 add opaque data type to replace tsobject
- b640869 port over to using opaques
- 131f8ac add more api functions
- bea9a93 refactor
- 2f449ff add properties to opaque type
- 963334b fix possible alignment issues
- 775060b improve api
- d8d986a refactor
- 2d5559a fix formatting
- c18a6c8 small fixes
- 038e62f add decode function for parsing
- c9ea945 fix ci errors
- 7169525 build treesitter library instead of distro pkg for ci
- f31277a create test files
- 695086a remove unused 'tslanguage' option
- 444058e improve encoding detection
- c4c740b extra tests
You are receiving this because you are subscribed to this thread.
Foxe Chen
64-bitman left a comment (vim/vim#18869)Edit: Treesitter is build with
std=c11and vim is c99. But it still compiles under c99 with some warnings.% make -B cc -O3 -Wall -Wextra -Wshadow -Wpedantic -Werror=incompatible-pointer-types -std=c99 -fPIC -fvisibility=hidden -D_POSIX_C_SOURCE=200112L -D_DEFAULT_SOURCE -D_DARWIN_C_SOURCE -Ilib/src -Ilib/src/wasm -Ilib/include -c -o lib/src/alloc.o lib/src/alloc.c cc -O3 -Wall -Wextra -Wshadow -Wpedantic -Werror=incompatible-pointer-types -std=c99 -fPIC -fvisibility=hidden -D_POSIX_C_SOURCE=200112L -D_DEFAULT_SOURCE -D_DARWIN_C_SOURCE -Ilib/src -Ilib/src/wasm -Ilib/include -c -o lib/src/get_changed_ranges.o lib/src/get_changed_ranges.c In file included from lib/src/././tree_cursor.h:4, from lib/src/./get_changed_ranges.h:8, from lib/src/get_changed_ranges.c:1: lib/src/./././subtree.h:35:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 35 | }; | ^ lib/src/./././subtree.h:146:6: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 146 | }; | ^ lib/src/./././subtree.h:153:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 153 | }; | ^ cc -O3 -Wall -Wextra -Wshadow -Wpedantic -Werror=incompatible-pointer-types -std=c99 -fPIC -fvisibility=hidden -D_POSIX_C_SOURCE=200112L -D_DEFAULT_SOURCE -D_DARWIN_C_SOURCE -Ilib/src -Ilib/src/wasm -Ilib/include -c -o lib/src/language.o lib/src/language.c In file included from lib/src/./language.h:8, from lib/src/language.c:1: lib/src/././subtree.h:35:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 35 | }; | ^ lib/src/././subtree.h:146:6: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 146 | }; | ^ lib/src/././subtree.h:153:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 153 | }; | ^ cc -O3 -Wall -Wextra -Wshadow -Wpedantic -Werror=incompatible-pointer-types -std=c99 -fPIC -fvisibility=hidden -D_POSIX_C_SOURCE=200112L -D_DEFAULT_SOURCE -D_DARWIN_C_SOURCE -Ilib/src -Ilib/src/wasm -Ilib/include -c -o lib/src/lexer.o lib/src/lexer.c In file included from lib/src/./lexer.h:9, from lib/src/lexer.c:2: lib/src/././subtree.h:35:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 35 | }; | ^ lib/src/././subtree.h:146:6: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 146 | }; | ^ lib/src/././subtree.h:153:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 153 | }; | ^ cc -O3 -Wall -Wextra -Wshadow -Wpedantic -Werror=incompatible-pointer-types -std=c99 -fPIC -fvisibility=hidden -D_POSIX_C_SOURCE=200112L -D_DEFAULT_SOURCE -D_DARWIN_C_SOURCE -Ilib/src -Ilib/src/wasm -Ilib/include -c -o lib/src/node.o lib/src/node.c In file included from lib/src/node.c:3: lib/src/./subtree.h:35:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 35 | }; | ^ lib/src/./subtree.h:146:6: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 146 | }; | ^ lib/src/./subtree.h:153:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 153 | }; | ^ cc -O3 -Wall -Wextra -Wshadow -Wpedantic -Werror=incompatible-pointer-types -std=c99 -fPIC -fvisibility=hidden -D_POSIX_C_SOURCE=200112L -D_DEFAULT_SOURCE -D_DARWIN_C_SOURCE -Ilib/src -Ilib/src/wasm -Ilib/include -c -o lib/src/parser.o lib/src/parser.c In file included from lib/src/././tree_cursor.h:4, from lib/src/./get_changed_ranges.h:8, from lib/src/parser.c:9: lib/src/./././subtree.h:35:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 35 | }; | ^ lib/src/./././subtree.h:146:6: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 146 | }; | ^ lib/src/./././subtree.h:153:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 153 | }; | ^ cc -O3 -Wall -Wextra -Wshadow -Wpedantic -Werror=incompatible-pointer-types -std=c99 -fPIC -fvisibility=hidden -D_POSIX_C_SOURCE=200112L -D_DEFAULT_SOURCE -D_DARWIN_C_SOURCE -Ilib/src -Ilib/src/wasm -Ilib/include -c -o lib/src/point.o lib/src/point.c cc -O3 -Wall -Wextra -Wshadow -Wpedantic -Werror=incompatible-pointer-types -std=c99 -fPIC -fvisibility=hidden -D_POSIX_C_SOURCE=200112L -D_DEFAULT_SOURCE -D_DARWIN_C_SOURCE -Ilib/src -Ilib/src/wasm -Ilib/include -c -o lib/src/query.o lib/src/query.c In file included from lib/src/./language.h:8, from lib/src/query.c:13: lib/src/././subtree.h:35:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 35 | }; | ^ lib/src/././subtree.h:146:6: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 146 | }; | ^ lib/src/././subtree.h:153:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 153 | }; | ^ cc -O3 -Wall -Wextra -Wshadow -Wpedantic -Werror=incompatible-pointer-types -std=c99 -fPIC -fvisibility=hidden -D_POSIX_C_SOURCE=200112L -D_DEFAULT_SOURCE -D_DARWIN_C_SOURCE -Ilib/src -Ilib/src/wasm -Ilib/include -c -o lib/src/stack.o lib/src/stack.c In file included from lib/src/./language.h:8, from lib/src/stack.c:2: lib/src/././subtree.h:35:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 35 | }; | ^ lib/src/././subtree.h:146:6: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 146 | }; | ^ lib/src/././subtree.h:153:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 153 | }; | ^ cc -O3 -Wall -Wextra -Wshadow -Wpedantic -Werror=incompatible-pointer-types -std=c99 -fPIC -fvisibility=hidden -D_POSIX_C_SOURCE=200112L -D_DEFAULT_SOURCE -D_DARWIN_C_SOURCE -Ilib/src -Ilib/src/wasm -Ilib/include -c -o lib/src/subtree.o lib/src/subtree.c In file included from lib/src/subtree.c:9: lib/src/./subtree.h:35:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 35 | }; | ^ lib/src/./subtree.h:146:6: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 146 | }; | ^ lib/src/./subtree.h:153:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 153 | }; | ^ cc -O3 -Wall -Wextra -Wshadow -Wpedantic -Werror=incompatible-pointer-types -std=c99 -fPIC -fvisibility=hidden -D_POSIX_C_SOURCE=200112L -D_DEFAULT_SOURCE -D_DARWIN_C_SOURCE -Ilib/src -Ilib/src/wasm -Ilib/include -c -o lib/src/tree.o lib/src/tree.c In file included from lib/src/././tree_cursor.h:4, from lib/src/./get_changed_ranges.h:8, from lib/src/tree.c:3: lib/src/./././subtree.h:35:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 35 | }; | ^ lib/src/./././subtree.h:146:6: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 146 | }; | ^ lib/src/./././subtree.h:153:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 153 | }; | ^ cc -O3 -Wall -Wextra -Wshadow -Wpedantic -Werror=incompatible-pointer-types -std=c99 -fPIC -fvisibility=hidden -D_POSIX_C_SOURCE=200112L -D_DEFAULT_SOURCE -D_DARWIN_C_SOURCE -Ilib/src -Ilib/src/wasm -Ilib/include -c -o lib/src/tree_cursor.o lib/src/tree_cursor.c In file included from lib/src/./tree_cursor.h:4, from lib/src/tree_cursor.c:2: lib/src/././subtree.h:35:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 35 | }; | ^ lib/src/././subtree.h:146:6: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 146 | }; | ^ lib/src/././subtree.h:153:4: warning: ISO C99 doesn’t support unnamed structs/unions [-Wpedantic] 153 | }; | ^ cc -O3 -Wall -Wextra -Wshadow -Wpedantic -Werror=incompatible-pointer-types -std=c99 -fPIC -fvisibility=hidden -D_POSIX_C_SOURCE=200112L -D_DEFAULT_SOURCE -D_DARWIN_C_SOURCE -Ilib/src -Ilib/src/wasm -Ilib/include -c -o lib/src/wasm_store.o lib/src/wasm_store.c ar rcs libtree-sitter.a lib/src/alloc.o lib/src/get_changed_ranges.o lib/src/language.o lib/src/lexer.o lib/src/node.o lib/src/parser.o lib/src/point.o lib/src/query.o lib/src/stack.o lib/src/subtree.o lib/src/tree.o lib/src/tree_cursor.o lib/src/wasm_store.o cc -shared -Wl,-soname,libtree-sitter.so.0.26 lib/src/alloc.o lib/src/get_changed_ranges.o lib/src/language.o lib/src/lexer.o lib/src/node.o lib/src/parser.o lib/src/point.o lib/src/query.o lib/src/stack.o lib/src/subtree.o lib/src/tree.o lib/src/tree_cursor.o lib/src/wasm_store.o -o libtree-sitter.so sed -e 's|@PROJECT_VERSION@|0.26.2|' \ -e 's|@CMAKE_INSTALL_LIBDIR@|lib|' \ -e 's|@CMAKE_INSTALL_INCLUDEDIR@|include|' \ -e 's|@PROJECT_DESCRIPTION@|An incremental parsing system for programming tools|' \ -e 's|@PROJECT_HOMEPAGE_URL@|https://tree-sitter.github.io/tree-sitter/|' \ -e 's|@CMAKE_INSTALL_PREFIX@|/usr/local|' lib/tree-sitter.pc.in > tree-sitter.pc
That looks like a problem, I guess we just disable the treesitter feature on non C11 compliant compilers?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
BenYip
bennyyip left a comment (vim/vim#18869)It still compiles in C99. The only feature they use is unnamed structs/unions in a few structs. We can give them names to suppress warnings.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Foxe Chen
64-bitman left a comment (vim/vim#18869)I'm quite confused on how to go forward with statically linking Vim with Treesitter. The only thing I can think of is copying the Treesitter library files into src/treesitter/, and modifying our src/Makefile to build the library.
However, if we want to include WASM support, then it looks like we need rust (cargo) installed? I couldn't find any packages containing wasmtime development files on Arch Linux + how would this be done on Windows or MacOS? I'm not sure how accessible this wasmtime library is other then getting it through Cargo using this crate (from what I could find).
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- f56f14e add tsquery_inspect() function
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- a9942e6 add tsquery_inspect() function
You are receiving this because you are subscribed to this thread.
Christian Clason
clason left a comment (vim/vim#18869)Or we could use the Treesitter provided Makefile, but not sure if that is the best way.
Yes, that would be the best way. The lib itself is a fairly simple project, and so is the makefile (make static, and then include that in the linker step). Of course, if this is your first external dependency, this will require significant changes to your build process. (That's not something I can much help with; I can only say that the tree-sitter makefile is relatively footgun-free.)
However, if we want to include WASM support, then it looks like we need rust (cargo) installed?
Yes, for WASM parsers you (currently) need wasmtime-c-api, which is a rust dependency; see https://github.com/neovim/neovim/blob/master/cmake.deps/cmake/BuildWasmtime.cmake
However, we pretty much abandoned this for Neovim, not least because wasmtime is a fast-moving target (monthly major releases). I don't want to spoil things, but there may be a lighter-weight wasm runtime in Zig coming...
For now, I'd recommend ignoring WASM parsers and require native parsers.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 873390e add treesitter library in-tree
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- fbd8068 add treesitter library in-tree
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 27 commits.
- b18215e initial commit
- 8be8c43 setup basic stuff
- 88998cd refactor
- 2b9f3df iniitial parsing backend logic
- a5f4d46 refactor
- b00ef72 basic working
- 878f999 initial vimscript api
- 0bdf1c0 add more api functions
- c8cd95c add opaque data type to replace tsobject
- 3ced324 port over to using opaques
- e82be5f add more api functions
- d66ecf1 refactor
- b338eb7 add properties to opaque type
- c54543a fix possible alignment issues
- 1eba46a improve api
- d3adf7a refactor
- 4284514 fix formatting
- 27922d1 small fixes
- 4afd22e add decode function for parsing
- 5065dc6 fix ci errors
- c83513e build treesitter library instead of distro pkg for ci
- b37f191 create test files
- 6741982 remove unused 'tslanguage' option
- 0c6e29d improve encoding detection
- 56c98c4 extra tests
- 5aef695 add tsquery_inspect() function
- b160b0d add treesitter library in-tree
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 8d6bcb5 add treesitter library in-tree
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- cc9eb9a add treesitter library in-tree
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- ebfea7f add treesitter library in-tree
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 3b765f4 add treesitter library in-tree
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 781cd4a add treesitter library in-tree
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- e218f58 fix proto files
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 84e7297 opaque type fixes
You are receiving this because you are subscribed to this thread.
D. Ben Knoble
benknoble left a comment (vim/vim#18869)I think @ubaldot was thinking of #9087. I do think we still have to have some kind of plan to address concerns raised in that discussion about the build process for language parsers that need compilation. It doesn't have to be perfect, but I think we need at least some notion of a vision/goal for how this will come together.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Foxe Chen
64-bitman left a comment (vim/vim#18869)I think @ubaldot was thinking of #9087. I do think we still have to have some kind of plan to address concerns raised in that discussion about the build process for language parsers that need compilation. It doesn't have to be perfect, but I think we need at least some notion of a vision/goal for how this will come together.
I believe all language parsers need compilation? As for the problem on how users will somehow get these parsers, I guess that can be implemented in community plugins? I don't think some users would like Vim having a builtin plugin that accesses the internet. We already offload actual LSP support onto community plugins, so this sounds reasonable as well.
As for needing a compiler, that is straightforward on Linux (just install the distro's devel package whatever) and I'm guessing other Unix OSes too, but I'm not sure about Windows.
The solution to this would be precompiled parsers, but looking the parser repo's of major programming languages, all they seem to provide is the source code in their releases. I read somewhere that nvim-treesitter provides precompiled artifacts of parsers and uses those, but that is sorta an official plugin which I'm not sure is feasible with Vim's ecosystem.
I think we should start simple for the initial builtin Treesitter plugin. Maybe the first iteration should have:
- Async parsing
- Syntax highlighting
- Provided functions for for
foldexpr,indentexpr - Utility functions (execute queries, visualize AST tree, ...)
Performance optimizations/other stuff can come later. This should be enough for most users
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Christian Clason
clason left a comment (vim/vim#18869)I read somewhere that nvim-treesitter provides precompiled artifacts of parsers and uses those
You read that wrong. Long-term, the answer is WASM, but that is not yet feasible today. (Again, it's not like other editors haven't been working on this for 5+ years.)
I think we should start simple for the initial builtin Treesitter plugin. Maybe the first iteration should have:
I think you need to focus first and foremost on (efficiently) handling "language trees", i.e., injections (which is arguably the biggest selling point for tree-sitter over other highlighting solutions.)
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 2 commits.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 32 commits.
- d568545 initial commit
- 98fc4c4 setup basic stuff
- b5a0f77 refactor
- 4c96287 iniitial parsing backend logic
- bc75bfe refactor
- dcff49c basic working
- bdec1c6 initial vimscript api
- c8765e3 add more api functions
- 10a3d8b add opaque data type to replace tsobject
- b807a0b port over to using opaques
- 2435f03 add more api functions
- f5ef1a1 refactor
- 9c19c76 add properties to opaque type
- 1d0532e fix possible alignment issues
- fd37c59 improve api
- 1483956 refactor
- d891d4e fix formatting
- 23c9193 small fixes
- 8e7a7f4 add decode function for parsing
- 18b8c08 fix ci errors
- 5344e37 build treesitter library instead of distro pkg for ci
- 267a055 create test files
- 27efe9d remove unused 'tslanguage' option
- 227d547 improve encoding detection
- 3f5e278 extra tests
- 88d3c0b add tsquery_inspect() function
- 4c5f987 add treesitter library in-tree
- 98a096b fix proto files
- 41d0fa3 opaque type fixes
- 834ccde more opaque type tests
- 22a5f22 build treesitter library using src/Makefile
- e45b901 temp
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 3 commits.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 30d1dca refactor
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 87e8133 refactor
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 52dde4e more api functions
You are receiving this because you are subscribed to this thread.
D. Ben Knoble
benknoble left a comment (vim/vim#18869)Can we make tree-sitter a submodule? That might simplify vendoring changes (and makes it easier to verify this PR isn't sneaking in a close-but-not-actually tree-sitter source file that turns out to be some kind of backdoor or whatever).
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Clason
clason left a comment (vim/vim#18869)Have you looked at the tree-sitter repository? Let that man cook, sheesh.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
D. Ben Knoble
benknoble left a comment (vim/vim#18869)Have you looked at the tree-sitter repository? Let that man cook, sheesh.
I don't follow: I assume this was in reply to my comment, but I'm not sure what I'm supposed to get from this.
A brief glance at tree-sitter/tree-sitter didn't cause anything to jump out, except maybe that we're only taking the relevant part of the tree-sitter code (which is admittedly hard to do in an easy-to-verify way with Git).
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 923cbae refactor tsparser_parse_buf()
You are receiving this because you are subscribed to this thread.
Foxe Chen
64-bitman left a comment (vim/vim#18869)Can we make tree-sitter a submodule? That might simplify vendoring changes (and makes it easier to verify this PR isn't sneaking in a close-but-not-actually tree-sitter source file that turns out to be some kind of backdoor or whatever).
The same could be said for the xdiff dependency we have. I don't think a submodule is necessary, since its just a matter of copying the library source files from a Tree-sitter release tar into the src/treesitter. Additionally, the Tree-sitter Makefile doesn't seem to work with BSD make, so I had to use our src/Makefile
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 5ffaeda refactor tsparser_parse_buf()
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 2 commits.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 983e21d add ts_query_cursor_remove_match
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- b4d3b92 add ts_query_cursor_remove_match
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- de3179e add ts_query_cursor_remove_match
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 19a8216 add tsquerymatch_get_capture()
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 0502d02 refactor tsquerycursor_next_capture
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 0 commits.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 5b2d010 refactor tsquerycursor_next_capture
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 0 commits.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 85ca832 return false for value comparisons of tsquerymatch
You are receiving this because you are subscribed to this thread.
Christian Clason
clason left a comment (vim/vim#18869)Note that there are version compatibility constraints between the tree-sitter library and the tree-sitter CLI used to generate the individual parsers. This effectively prevents you from fully vendoring the code. And compared to xdiff, tree-sitter is still under heavy development and will see breaking changes.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
James McCoy
jamessan left a comment (vim/vim#18869)Would you agree to bundling the treesitter library in repo? Treesitter packages for non rolling linux distros are too old (treesitter development moves fast)
I don't see how "treesitter packages on old distro releases are old" leads to "bundle treesitter in Vim's repo". Vim packages are also old on old distro releases. If someone wants to build a newer Vim there, then they can do the work to also build the new Vim's dependencies.
Bundling a third-party dependency shouldn't be a light decision. It's fine to say "Vim depends on this specific range of versions for dependency X" and let packagers figure out how to do that. Don't try to solve their problem for them by bundling the code in Vim.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Foxe Chen
64-bitman left a comment (vim/vim#18869)@chrisbra What do you think? I'm fine with either way
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 3b40767 update tsquerymatch equal func
You are receiving this because you are subscribed to this thread.
BenYip
bennyyip left a comment (vim/vim#18869)Maybe we can handle it the same way as libsodium. For Linux user, they can use liblibtree-sitter.so from package manager. And for Windows user, we bundle it in https://github.com/vim/vim-win32-installer.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
chrisbra left a comment (vim/vim#18869)If possible, we should avoid vendoring tree-sitter into Vim. It is a lot of code, and I do not want to take on responsibility for maintaining it here - especially from a security perspective, since it is a fast-moving target and I already have plenty to do for Vim. Instead, we should keep this modular and have the configure script detect (or point) and link against an installed tree-sitter.
I guess we have to find out how to do this for Windows, but we had to do the same for libsodium back then. We don't have to solve everything at once.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- d093839 Revert "add treesitter library in-tree"
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 2 commits.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 6ac6222 make captures property of tsquerymatch return dict
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 3ff4a05 use tuple for tsquery_inspect patterns entry instead of dict
You are receiving this because you are subscribed to this thread.
D. Ben Knoble
benknoble left a comment (vim/vim#18869)(PS: I know most PRs get squashed here, so what I'm about to say is largely moot for the final result. It might be helpful for 64-bitman and collaborators as they develop anyway.
Reverting the vendor commit still requires Git to keep around all the object files corresponding to that code. By contrast, if the branch is rewritten so that the vendor commit never appears in the first place, Git can throw those objects away from local copies and they won't be present in new fetches or clones. It also keeps history cleaner for projects that don't squash.)
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 43 commits.
- 5e125ae initial commit
- 75e5406 setup basic stuff
- e5b9246 refactor
- fd4d0a6 iniitial parsing backend logic
- 89ec164 refactor
- 762f8ca basic working
- fa8bf35 initial vimscript api
- c9aef9e add more api functions
- c8bc85c add opaque data type to replace tsobject
- 4f6b047 port over to using opaques
- d47c17f add more api functions
- c2544a2 refactor
- f202d2b add properties to opaque type
- d52db1a fix possible alignment issues
- ed78983 improve api
- 9fb3e8f refactor
- 0f2b2e8 fix formatting
- a91104a small fixes
- afc7921 add decode function for parsing
- 4f60518 fix ci errors
- d7386d8 build treesitter library instead of distro pkg for ci
- 250fbd5 create test files
- a28fbe3 remove unused 'tslanguage' option
- 2e5aaad improve encoding detection
- a938ecf extra tests
- 517154d add tsquery_inspect() function
- fefd92c add treesitter library in-tree
- 323e19d fix proto files
- 5eb2de3 opaque type fixes
- 8978e52 more opaque type tests
- cce5143 build treesitter library using src/Makefile
- 96e223a temp
- d4f0368 set allocator functions
- 6b5b536 add tsparser_set_included_ranges()
- a905827 refactor
- 736c9fa more api functions
- 463f2b6 refactor tsparser_parse_buf()
- b47f8e3 refactor ts_query_cursor_next_match
- f193c70 add ts_query_cursor_remove_match
- 867335c update tsquerymatch equal func
- b117542 Revert "add treesitter library in-tree"
- c380771 make captures property of tsquerymatch return dict
- be66adf use tuple for tsquery_inspect patterns entry instead of dict
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 43 commits.
- f91f440 initial commit
- 4bbbdcd setup basic stuff
- c992256 refactor
- 7273fd5 iniitial parsing backend logic
- 4aa6e55 refactor
- 9d2d7c5 basic working
- 00f63df initial vimscript api
- 9590196 add more api functions
- 8ba70b7 add opaque data type to replace tsobject
- 049cec6 port over to using opaques
- 21914f5 add more api functions
- 7c1f990 refactor
- b4b37c1 add properties to opaque type
- fc99801 fix possible alignment issues
- 18e2505 improve api
- c6edaf0 refactor
- 1ef03ad fix formatting
- 42a94ea small fixes
- b19e188 add decode function for parsing
- 6f71dc3 fix ci errors
- ca928d3 build treesitter library instead of distro pkg for ci
- a84aa4f create test files
- d9f645f remove unused 'tslanguage' option
- 1e22793 improve encoding detection
- 3149be4 extra tests
- 532aef0 add tsquery_inspect() function
- 8383d91 add treesitter library in-tree
- b201d47 fix proto files
- 342bcbd opaque type fixes
- 775ec54 more opaque type tests
- f74d036 build treesitter library using src/Makefile
- 7c49136 temp
- 31bc4e7 set allocator functions
- 10b81ff add tsparser_set_included_ranges()
- 16993f5 refactor
- 02fb046 more api functions
- 5886797 refactor tsparser_parse_buf()
- c70876a refactor ts_query_cursor_next_match
- 1e30607 add ts_query_cursor_remove_match
- 4c5387f update tsquerymatch equal func
- 5527ec7 Revert "add treesitter library in-tree"
- 8741101 make captures property of tsquerymatch return dict
- 4dcf7b9 use tuple for tsquery_inspect patterns entry instead of dict
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 2 commits.
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 675717d add ts_language_is_loaded()
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 8e43c2f fix tsquery_inspect
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 6550695 fix tsquery_inspect
You are receiving this because you are subscribed to this thread.
Foxe Chen
@64-bitman pushed 1 commit.
- 0a94459 fix tsquery_inspect
You are receiving this because you are subscribed to this thread.