[vim/vim] Revamp `make proto` (PR #18045)
h_east
Instead of cproto, use clang.cindex from Python to create prototype declaration files (.pro) from source code.
I'm working on Ubuntu 24.04.
$ sudo apt-get update $ sudo apt-get install -y python3 python3-pip clang libclang-dev python3-clang
You can view, comment on, or merge this pull request online at:
https://github.com/vim/vim/pull/18045
Commit Summary
- c81eed3 Revamp `make proto`
- 0bef846 Add src/proto/gen_prototypes.py
- bac2a88 tweak around proto
- 9e3fedb tweak around proto
- ed0df52 tweak around proto
- 7a5188c tweak arounf proto
- 7806c05 tweak around proto
- f916ead Make array parameters print as `name[]` (e.g., `char *dirnames[]`) instead of `char *[]name`.
- ace6c5d - Top-level variadic: when the function type is variadic, change void → `...` or append `, ...`.
- 141ba6d show libclang diagnostics to stderr for easier troubleshooting, when env GENPROTO_SHOW_DIAGS is true.
- de34b92 - Respect argument types as defined.
File Changes
(19 files)
- M Filelist (1)
- M src/Makefile (78)
- M src/cmdexpand.c (4)
- M src/indent.c (2)
- M src/locale.c (2)
- M src/macros.h (2)
- M src/option.c (6)
- M src/optionstr.c (2)
- M src/os_unix.c (4)
- M src/os_win32.c (2)
- M src/proto/cmdexpand.pro (4)
- A src/proto/gen_prototypes.py (253)
- M src/proto/gui_gtk_x11.pro (2)
- M src/proto/if_python.pro (2)
- M src/proto/if_python3.pro (2)
- M src/protodef.h (14)
- M src/testing.c (4)
- M src/vim.h (8)
- M src/window.c (2)
Patch Links:
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#18045)@chrisbra and Vim members,
What do you think of this approach?
Although make proto depends on python and clang, I think it's better than continuing to use cproto.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
h_east
@h-east pushed 11 commits.
- 3c6de13 Revamp `make proto`
- e5f6433 Add src/proto/gen_prototypes.py
- 95cd8e2 tweak around proto
- 1ce7324 tweak around proto
- f6010c6 tweak around proto
- 8d72718 tweak arounf proto
- 6f1bab1 tweak around proto
- 20f4a28 Make array parameters print as `name[]` (e.g., `char *dirnames[]`) instead of `char *[]name`.
- 84ccf02 - Top-level variadic: when the function type is variadic, change void → `...` or append `, ...`.
- 295acf7 show libclang diagnostics to stderr for easier troubleshooting, when env GENPROTO_SHOW_DIAGS is true.
- c479d4b - Respect argument types as defined.
—
View it on GitHub or unsubscribe.
You are receiving this because you are subscribed to this thread.
h_east
@h-east pushed 8 commits.
- eea2c35 tweak around proto
- beab6b5 tweak script around pointer
- a951150 tweak around proto
- 153a902 tweak around proto
- a8d6c66 tweak around proto
- 0dcac87 tweak around proto
- 3915f78 tweak around proto
- a4dd71b tweak around proto
You are receiving this because you are subscribed to this thread.
Yegappan Lakshmanan
@yegappan commented on this pull request.
In src/indent.c:
> @@ -474,7 +474,7 @@ get_indent_str(
return count;
}
-#ifdef FEAT_VARTABS
+#if defined(FEAT_VARTABS) || defined(PROTO)
One suggestion: it might be easier to track the changes later, if you can separate out the changes related to the "PROTO" checks into one PR and the changes for the new script into another PR.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
h_east
@h-east commented on this pull request.
In src/indent.c:
> @@ -474,7 +474,7 @@ get_indent_str(
return count;
}
-#ifdef FEAT_VARTABS
+#if defined(FEAT_VARTABS) || defined(PROTO)
Around the same time you commented, I submitted a PR implementing your suggestion 👍
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Yee Cheng Chin
ychin left a comment (vim/vim#18045)Does this work in all platforms? And does it still require us to do all the typedef's in say os_mswin.c (e.g. typedef int BOOL;) or the declarations in struct.h (e.g. typedef long varnumber_T;).
If the answer is no, then I wonder what the point of switching to a separate Python script is?
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#18045)The answer is “Yes.” I’m still experimenting, but my goal is to get the same output in any environment where Python and Clang are available. It’s impossible to do this with a truly accurate parser, so I’m switching to an approach that detects “something like a function definition” and then generates its prototype declaration as-is.
If this works, we won’t need the PROTO definitions themselves nor the hacky consistency-forcing code for cproto that you mentioned.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
h_east
@h-east pushed 9 commits.
- c0ff97c Revamp `make proto`
- 4bb8af1 Add src/proto/gen_prototypes.py
- 104e29b Make array parameters print as `name[]` (e.g., `char *dirnames[]`) instead of `char *[]name`.
- dbb93ff - Top-level variadic: when the function type is variadic, change void → `...` or append `, ...`.
- 18239d4 show libclang diagnostics to stderr for easier troubleshooting, when env GENPROTO_SHOW_DIAGS is true.
- d77ff3d - Respect argument types as defined.
- 40881fc tweak script around pointer
- f2ffcf4 fix makefile
- 3863053 Switching to an approach that detects "something like a function definition" and then generates its prototype declaration as-is.
You are receiving this because you are subscribed to this thread.
h_east
@h-east pushed 4 commits.
You are receiving this because you are subscribed to this thread.
h_east
h_east
@h-east pushed 2 commits.
You are receiving this because you are subscribed to this thread.
h_east
h_east
@h-east pushed 9 commits.
- 215e9e9 Revamp `make proto`
- 76e8e94 support `#ifndef PROTO` and `#if !defined(PROTO)`
- 0770a36 Removed error workaround code for `cproto`
- 16f698a update script
- e2868cd update script
- 4027a9d Update script
- cbf8c73 New `make proto` checks the `#if` side by default, so the conditions and contents of `#if` and `#else` are swapped.
- f9c9d5f The code inside `#ifdef _DEBUG` is excluded.
- 5d7be04 update
You are receiving this because you are subscribed to this thread.
h_east
@h-east pushed 3 commits.
- dd6ed51 Removed error workaround code for `cproto`
- 4082af4 Removed unnecessary `defined(PROTO)`
- 9e42ba3 fix proto/insexpand.pro
You are receiving this because you are subscribed to this thread.
h_east
h_east
h-east left a comment (vim/vim#18045)I think it's almost finished.
I've confirmed that proto/*.pro outputs the same thing on Ubuntu 24.04 and Windows 11 (with Git Bash).
ALL>
I have a request for everyone:
Could you please let me know if there are any differences when running make proto in your environment?
The setup for the execution environment is written in src/Makefile. (See below)
I would be happy to gather information from various platforms. Thank you.
# Generate function prototypes. Previously, cproto was used, but since there
# was a lot of fiddling to do to generate them correctly, so we switched to a
# Python script using libclang. It has the following features:
# - Only the bare minimum of macro definitions is required.
# - Generates the same .pro file even across different operating systems.
# (maybe)
#
# How to set up environment to run `make proto`
# On Ubuntu 24.04:
# - Install libclang
# Use the package provided by Ubuntu.
# $ sudo apt install python3-clang
# Or, Create venv (if you want to use it freely with pip)
# $ python3 -m venv ~/venv
# $ source ~/venv/bin/activate
# $ pip install --upgrade pip
# $ pip install libclang
# On Windows 11 (using Git Bash):
# - Install GNU Make 3.81 or later.
# - Download and install Python from the Python official website.
# - Install libclang
# $ python -m pip install libclang
# - Change the following lines in this file:
# s/^PYTHON=python3/PYTHON=python/
# s/^tags TAGS: notags/tags: notags/
PYTHON=python3
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
chrisbra left a comment (vim/vim#18045)it seems to work better for me, does no longer depend on the current configure options, although the build dependencies are much heavier.
I have 2 questions:
- Can we make it true dependencies? So it only generates the proto files, if there are changes?
- I think we can get rid of this block here: https://github.com/vim/vim/blob/20e78ee807533d642ce4d6910a0718b854b7c405/src/Makefile#L1341-L1360
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
h_east
h_east
h-east left a comment (vim/vim#18045)although the build dependencies are much heavier.
What part of the behavior does this refer to specifically?
I have 2 questions:
* Can we make it true dependencies? So it only generates the proto files, if there are changes?
Does this mean, for example, that when alloc.c is updated, make should update proto/alloc.pro?
The original make proto was separate from make, so I gave it a similar structure.
The new make proto depends on Python and libclang. If I implement the above suggestion, even make will depend on Python and libclang. Is this okay?
Or does it mean that it only updates the .pro files that correspond to the .c files that are updated when make proto is executed?
Hmm, make proto isn't something that is executed that often, so I think it's fine to leave it as it is (updating all .pro files).
(It's just that writing the dependencies is a hassle. 😅)
* I think we can get rid of this block here:
Done. Thanks.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#18045)Or does it mean that it only updates the .pro files that correspond to the .c files that are updated when make proto is executed?
I'm working on above.
Final confirmation is underway.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
h_east
h_east
@h-east pushed 5 commits.
- 58f2dd5 Revamp `make proto`
- ce02786 Removed error workaround code for `cproto`
- 3747f92 Removed unnecessary `defined(PROTO)`
- af1eea0 Remove unnecessary blocks
- 09a45ae Added a mechanism to update only the target .pro by adding a dependency between .pro and .c
You are receiving this because you are subscribed to this thread.
h_east
@h-east pushed 5 commits.
- 9e01fe1 Revamp `make proto`
- 761e271 Removed error workaround code for `cproto`
- ae56f6a Removed unnecessary `defined(PROTO)`
- c4e4796 Remove unnecessary blocks
- 50e3fd2 Added a mechanism to update only the target .pro by adding a dependency between .pro and .c
You are receiving this because you are subscribed to this thread.
h_east
@h-east pushed 6 commits.
- 2c81725 Revamp `make proto`
- f9674e9 Removed error workaround code for `cproto`
- f8740b5 Removed unnecessary `defined(PROTO)`
- 81ad53d Remove unnecessary blocks
- 7dab2ed Added a mechanism to update only the target .pro by adding a dependency between .pro and .c
- 1a8616d Fix permission to 644
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#18045)Or does it mean that it only updates the .pro files that correspond to the .c files that are updated when make proto is executed?
I'm working on above. Final confirmation is underway.
Done !!
@ychin
Could you please tell me the results of make protoclean; make proto on macOS using the following as a reference?
#18045 (comment)
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
h_east
@h-east pushed 5 commits.
- 2184519 Revamp `make proto`
- 922801a Removed error workaround code for `cproto`
- 0d47408 Removed unnecessary `defined(PROTO)`
- bbc78cd Remove unnecessary blocks
- 63f746d Added a mechanism to update only the target .pro by adding a dependency between .pro and .c
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#18045)@chrisbra
No one has checked it, but if it works fine in your environment, I think it's fine to merge it in. At least you can run make proto on major/minor releases.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
chrisbra left a comment (vim/vim#18045)Thanks, it is quite nice. Can you add something like the following on top of it, so that it runs in CI automatically and verifies the proto files are always up to date?
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 004f4a622..b408d8569 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -84,7 +84,7 @@ jobs: architecture: arm64 - features: normal compiler: gcc - extra: [vimtags] + extra: [vimtags, proto] - features: huge compiler: gcc extra: [no_x11] @@ -182,6 +182,9 @@ jobs: libattr1-dev ) fi + if ${{ contains(matrix.extra, 'proto') }}; then + PKGS+=( python3-clang ) + fi sudo apt-get update && sudo apt-get upgrade -y --allow-downgrades && sudo apt-get install -y --allow-downgrades "${PKGS[@]}" - name: Install gcc-${{ env.GCC_VER }} @@ -270,6 +273,9 @@ jobs: if ${{ contains(matrix.extra, 'vimtags') }}; then echo "TEST=-C runtime/doc vimtags VIMEXE=../../${SRCDIR}/vim" fi + if ${{ contains(matrix.extra, 'proto') }}; then + echo "TEST=-C src proto" + fi echo "CFLAGS=${CFLAGS}" echo "CONFOPT=${CONFOPT}" # Disables GTK attempt to integrate with the accessibility service that does run in CI. @@ -354,6 +360,15 @@ jobs: true ) + - name: Generate Proto files + if: contains(matrix.extra, 'proto') + run: | + # This will exit with an error code if the generated proto files differ from source + ( + git diff --exit-code -- src/proto/ + true + ) + - name: Generate gcov files if: matrix.coverage run: |
(untested)
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
D. Ben Knoble
benknoble left a comment (vim/vim#18045)On my machine (macOS 12.7.6) after installing libclang in a Python virtual environment, make protoclean && make proto ran from src (but not the top-level…).
It reran the configure script, which was unexpected to me as a passive follower of this topic (I'm not sure which configure options were picked up or used).
The resulting working tree was clean (no files were modified according to git status).
I used commit 3b682f5 (Merge 63f746d into 3495936, 2025-10-02) generated automatically by GitHub as refs/pull/18045/merge from this PR at 63f746d (Added a mechanism to update only the target .pro by adding a dependency between .pro and .c, 2025-09-29).
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
D. Ben Knoble
benknoble left a comment (vim/vim#18045)It reran the
configurescript, which was unexpected to me as a passive follower of this topic (I'm not sure which configure options were picked up or used).I updated the path names for
TCL_PROandPERL_PROinsrc/configure.acandsrc/auto/configure, so it's normal for configure to run on the first attempt. (I'll also review the settings forLUA_PRO,MZSCHEME_PRO, andRUBY_PRO.)
Ah, gotcha, good to know!
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#18045)Thanks for confirming.
On my machine (macOS 12.7.6) after installing
libclangin a Python virtual environment,make protoclean && make protoran fromsrc(but not the top-level…).
It reran the
configurescript, which was unexpected to me as a passive follower of this topic (I'm not sure which configure options were picked up or used).
I updated the path names for TCL_PRO and PERL_PRO in src/configure.ac and src/auto/configure, so it's normal for configure to run on the first attempt. (I'll also review the settings for LUA_PRO, MZSCHEME_PRO, and RUBY_PRO.)
The resulting working tree was clean (no files were modified according to
git status).
Since everything worked without issues, I'm relieved.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
Drew Vogel
dvogel left a comment (vim/vim#18045)@chrisbra and Vim members, What do you think of this approach? Although
make protodepends on python and clang, I think it's better than continuing to usecproto.
@h-east can you elaborate on why you think this is better than using cproto? Which cproto shortcomings are you working to rid us of?
The python program appears to be using regular expressions to parse pre-processor directives, translating them to python code, so that it can eval the conditions. It is an impressive feat but it requires pretty substantial expertise with both C and python to work on. Also, it seems to me we already have a tool to evaluate those pre-processor conditions: the C pre-processor. Why not run the source files through cpp and parse that with the clang parser?
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#18045)Thank you for your question.
The main goal of gen_prototypes.py is to improve the consistency and maintainability of the generated .pro files.
There are several long-standing issues with the current cproto-based approach.
1. Limitations of cproto
- Development of
cprotohas long been discontinued, so no further updates or fixes are expected. - Without manually specifying a sufficient number of system include paths using
-I,cprotofails to parse and produces no.profiles. - Many workarounds exist in the Vim source and
src/Makefileto makecprotowork correctly.
See this commit - On macOS — one of Vim's major supported platforms —
make protocurrently does not work properly.
These issues make maintaining and reproducing consistent results with cproto increasingly difficult.
2. Advantages of using Clang
By using libclang, we can parse source code with the same accuracy as a real C compiler.
This allows us to reliably extract return types, argument types, and qualifiers, eliminating discrepancies between Vim's source code and the generated prototypes.
3. Why not just use cpp?
gen_prototypes.py implements its own lightweight handling of preprocessor directives only to keep the generated output consistent across all platforms.
- I wanted the output of
make prototo be identical regardless of the results of./configure. - Achieving that with
cppwould require defining all relevant macros for every platform, which is cumbersome. - Therefore, I implemented a minimal, self-contained interpretation of preprocessor conditions directly in Python,
reducing the need for external macro definitions while still ensuring consistent results.
In summary, combining Clang's precise parsing with minimal preprocessing logic in Python allows us to produce reliable, reproducible proto files across all platforms, without depending on cproto's legacy behavior.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
h_east
@h-east pushed 3 commits.
- 1c40b7e Revamp `make proto`
- 7de36f2 Removed error workaround code for `cproto`
- 1ccc488 Removed unnecessary `defined(PROTO)`
—
View it on GitHub or unsubscribe.
You are receiving this because you are subscribed to this thread.
Drew Vogel
@dvogel commented on this pull request.
Thanks for the full explanation of your motivations here @h-east. I did my best to condense it down into a comment at the top of the program because I think it will be very helpful to maintain momentum here.
I think some minimal level of automated tests will be needed to help others maintain this. cproto had a collection of tests that allowed it's output to be verified. Perhaps those could be ported over here.
In src/proto/gen_prototypes.py:
> @@ -0,0 +1,462 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- + +import os⬇️ Suggested change
-import os + +# The main goal of gen_prototypes.py is to provide .pro files that are generated consistently +# across platforms, thereby making them easier to maintain. This replaced the cproto project +# because (a) it became unmaintained, (b) had too many system-wide include path +# dependencies, and (c) did not run on macos. +# +# libclang is used to reliably parse the C source code with the same accuracy as the compiler. +# The parsed code is then scanned for pre-processor directives and dynamically translated to +# python code that can be evaluated. This approach allows the translation to be done without +# defining all macros (which would be required for evaluation with cpp) and without concern +# for the developer's most recently generated build configuration. + +import os
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#18045)Thank you.
Automated testing (CI) was also suggested by ChrisBra, so yesterday I committed a trial implementation. It's currently producing duplicate output, but I plan to fix it eventually.
#18045 (comment)
https://github.com/vim/vim/actions/runs/18284424132/job/52055491680?pr=18045#step:21:221
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
h_east
h_east
h-east left a comment (vim/vim#18045)Confirm that the expected proto error appears. OK
image.png (view on web)—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
h_east
@h-east pushed 8 commits.
- 1142850 Revamp `make proto`
- 7897c73 Removed error workaround code for `cproto`
- caaf039 Removed unnecessary `defined(PROTO)`
- f5da79e Add CI proto
- 864a50a Test: Commit the proto error intentionally
- cadc3c6 Fix peoro error
- a7d1d71 Tweak (W.I.P)
- d205beb Fix proto files
You are receiving this because you are subscribed to this thread.
h_east
@h-east pushed 6 commits.
- f773aca Revamp `make proto`
- 31a6b3d Removed error workaround code for `cproto`
- 42b50f1 Removed unnecessary `defined(PROTO)`
- f03b6bf Add CI proto
- c95f7d3 Test: Commit the proto error intentionally
- 7e32ec5 Tweak (W.I.P)
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#18045)@chrisbra Ready to merge 👍
I think it's fine to add various documentation and code comments later.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
h_east
@h-east pushed 1 commit.
- 5418fc2 Fix proto error
—
View it on GitHub or unsubscribe.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
chrisbra left a comment (vim/vim#18045)thanks
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Christian Brabandt
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Yee Cheng Chin
ychin left a comment (vim/vim#18045)Hi, I haven't been keeping up-to-date with this but I just pulled Vim and this doesn't even build…
The error message looks like this:
Traceback (most recent call last):
gcc -c -I. -Ilibvterm/include -Iproto -DHAVE_CONFIG_H -DMACOS_X -DMACOS_X_DARWIN -I/opt/homebrew/Cellar/libsodium/1.0.20/include -g -O2 -D_REENTRANT -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -DINLINE="" -DVSNPRINTF=vim_vsnprintf -DSNPRINTF=vim_snprintf -DIS_COMBINING_FUNCTION=utf_iscomposing_uint -DWCWIDTH_FUNCTION=utf_uint2cells -o objects/vterm_screen.o libvterm/src/screen.c
File "/Users/ychin/Dev/external/vim/src/proto/gen_prototypes.py", line 39, in <module>
from clang.cindex import Index, CursorKind, StorageClass, TokenKind
ModuleNotFoundError: No module named 'clang'
make[1]: *** [proto/gui_beval.pro] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [first] Error 2
For reference the build steps was make distclean && ./configure && make, the most bog-standard way to build Vim cleanly.
Seems like every single build is now calling gen_prototypes.py, which uses the clang dependency. This is a completely new build requirement. Is this intentional? I thought this is just run on CI to make sure the prototypes match? Why are we forcing a new build dependency on everyone who just wants to build Vim without prior notice?
Can we revert this if a fix cannot be found quickly?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#18045)@ychin Sorry, see below links.
#18523 (comment)
#18523 (comment)
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Yee Cheng Chin
ychin left a comment (vim/vim#18045)Thanks for fixing it.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.