[vim/vim] Concerns regarding undisclosed AI usage and redundant comments in PRs (Issue #21318)
h_east
h-east created an issue (vim/vim#21318)We should probably be a bit more cautious with PRs from people using AI
assistance without declaring it.
Also, even when it is disclosed, redundant code comments and similar
fluff should still be fixed. I believe that if redundant comments keep
getting merged, it will eventually hinder human maintenance. (Though
perhaps in the future, humans carefully reading code won't even be
valuable anymore... ;-)
Ever since I received "Claude for Open Source" back in March of this
year, I've also been relying heavily on AI assistance for my PRs.
However, I make sure to handle the concerns mentioned above very
carefully.
What are your thoughts on this?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
Christian Brabandt
chrisbra left a comment (vim/vim#21318)Yes makes sense. As the number of changes here has been growing, I often suspected the use of AI but did not check back with the contributor. Should be more careful in the future. Let me add a PR template to make this more important
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#21318)The template and AGENTS.md give two different forms for disclosure
AGENTS.md states this in the commit format rules:
Co-Authored-By:is allowed and is the accepted way to
acknowledge AI assistance transparently.
The template divides the same ground differently: Supported by AI for
help with writing or investigating, and Co-authored-by: only when AI
wrote the commit. A contributor who reads both files gets two answers
to the same question. CONTRIBUTING.md asks for disclosure without
naming a form, so the template and AGENTS.md are the only places that
define one.
Which form the project wants is still an open question. AGENTS.md was
added by #20039 in April as a single commit with no review and no
comments, and it has not been touched since. Supported by AI appears
in 67 commits in master, 49 of them from one author and the remaining
18 spread over 13 contributors. Neither the file nor the notation has
been discussed, so this PR is the place to settle on one form and to
update AGENTS.md and the "Using AI" section of CONTRIBUTING.md to match
it.
Co-authored-by: AI is not a trailer that GitHub picks up
A co-author is recognized only in the Name <email> form. Every AI
trailer in master uses it:
Co-Authored-By: Claude Opus 5 (1M context) <nor...@anthropic.com>
Co-Authored-By: OpenAI Codex <nor...@openai.com>
Co-authored-by: AI does not appear anywhere in the history. Showing
the real form in the comment block keeps the trailer functional.
The same bullet is ambiguous in another way: "If AI wrote the commit
itself" reads as either the code or the commit message. That
distinction decides which of the two forms applies, so it is worth
stating explicitly.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#21318)@jvoisin Regarding your reply in #21309 (comment):
The only (local) LLM usage was to run the benchmarks.
CONTRIBUTING.md asks for this:
When using AI for contributions, please disclose this.
The condition is "using AI for contributions", not "AI-generated code",
so the benchmarks a patch rests on fall under it as the file stands
today. Nothing beyond what you already described is needed: please
state that much in the pull requests where it applies.
Where the line runs is written down nowhere, which is why I opened this
issue instead of pressing the point in your pull request. Let us agree
here on what the disclosure covers, put it into the template #21321
adds, into CONTRIBUTING.md and into AGENTS.md, and follow it from there.
Once it is written down, a contributor knows what to write without being
asked, and a reviewer has a line to point at instead of guessing.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
Julien Voisin
jvoisin left a comment (vim/vim#21318)The condition is "using AI for contributions", not "AI-generated code",
so the benchmarks a patch rests on fall under it as the file stands
today. Nothing beyond what you already described is needed: please
state that much in the pull requests where it applies.
Oh, right, my bad; I'll make sure to add a note about this in future pull-request!
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#21318)To be honest, I don’t understand why people are so reluctant to disclose "AI assistance."
For the past six months, I’ve included Co-Authored-By: Claude on almost all of my PRs, but the actual level of AI contribution varies significantly from PR to PR:
- It only wrote the commit message for me.
- The AI did everything, and I only did the final review.
- The AI did everything, but I rejected it during my final review and had the AI fix it again.
- The AI did everything, I rejected it, had it retry, but it couldn't get it right, so I ended up fixing it myself.
I doubt anyone other than myself can judge these subtle differences on a per-PR basis.
As for whether disclosing AI support affects how others perceive you: personally, I hold someone in higher regard if they disclose AI support and successfully resolve the issue. You can easily tell by looking at the quality of the final output.
So, I really hope people won't get so obsessed over whether something was accomplished entirely without AI support.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
Christian Brabandt
chrisbra left a comment (vim/vim#21318)Co-Authored-By: is allowed and is the accepted way to
acknowledge AI assistance transparently.
The template divides the same ground differently: Supported by AI for
help with writing or investigating, and Co-authored-by: only when AI
wrote the commit. A contributor who reads both files gets two answers
to the same question. CONTRIBUTING.md asks for disclosure without
naming a form, so the template and AGENTS.md are the only places that
define one.
Which form the project wants is still an open question. AGENTS.md was
added by #20039 in April as a single commit with no review and no
comments, and it has not been touched since. Supported by AI appears
in 67 commits in master, 49 of them from one author and the remaining
18 spread over 13 contributors. Neither the file nor the notation has
been discussed, so this PR is the place to settle on one form and to
update AGENTS.md and the "Using AI" section of CONTRIBUTING.md to match
it.
The difference I wanted to express was to distinguish between "helped by AI" in any form and "created by AI" completely.
Happy to drop the distinction if you think it doesn't make sense and settle on a single trailer line.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
Shane Harper
shaneharper left a comment (vim/vim#21318)The difference I wanted to express was to distinguish between "helped by AI" in any form and "created by AI" completely.
Happy to drop the distinction if you think it doesn't make sense and settle on a single trailer line.
My 2 cents: I think the distinction is worth keeping somehow. How about adding an instruction to AGENTS.md that when all code changes in the patch were written by the AI tool, with no manual edits, the commit message should say so explicitly?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
Shay Hill
ShayHill left a comment (vim/vim#21318)To be honest, I don't understand why people are so reluctant to disclose "AI assistance." For the past six months, I've included
Co-Authored-By: Claudeon almost all of my 186 PRs, but the actual level of AI contribution varies significantly from PR to PR:
- It only wrote the commit message for me.
- The AI did everything, and I only did the final review.
- The AI did everything, but I rejected it during my final review and had the AI fix it again.
- The AI did everything, I rejected it, had it retry, but it couldn't get it right, so I ended up fixing it myself.
I doubt anyone other than myself can judge these subtle differences on a per-PR basis. As for whether disclosing AI support affects how others perceive you: personally, I hold someone in higher regard if they disclose AI support and successfully resolve the issue. You can easily tell by looking at the quality of the final output. So, I really hope people won't get so obsessed over whether something was accomplished entirely without AI support.
To be honest, I don't understand why people are so reluctant to disclose "AI assistance." For the past six months, I've included
Co-Authored-By: Claudeon almost all of my 186 PRs, but the actual level of AI contribution varies significantly from PR to PR:
- It only wrote the commit message for me.
- The AI did everything, and I only did the final review.
- The AI did everything, but I rejected it during my final review and had the AI fix it again.
- The AI did everything, I rejected it, had it retry, but it couldn't get it right, so I ended up fixing it myself.
I doubt anyone other than myself can judge these subtle differences on a per-PR basis. As for whether disclosing AI support affects how others perceive you: personally, I hold someone in higher regard if they disclose AI support and successfully resolve the issue. You can easily tell by looking at the quality of the final output. So, I really hope people won't get so obsessed over whether something was accomplished entirely without AI support.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
Shay Hill
ShayHill left a comment (vim/vim#21318)It would be easy to go too far in the other direction with this. Every commit I’ve made to anything since late 2021 (iirc first used Copilot) could be marked “helped by AI”, but that would be meaningless. We never littered commit messages with “helped by Stack Overflow” before.
The line should be as bold as possible: maybe “‘CoAuthored by AI’ if AI wrote full lines of non-idiomatic code.”
I write management systems for a living, and my rule about granularity is this: if users don’t understand it or aren’t using it, pay to train them or take it out. Conceivably, someone could make a training video or manual for contributors, but I think “take it out” is probably the way to go here.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#21318)Is the distinction based solely on whether the AI wrote the code?
For example, what happens if the AI writes the code, but a human reviews it
and rewrites the whole thing? What if only half of it is rewritten? Or what
if an AI reviews human-written code, suggests a fix, and that fix is applied
by the AI itself, or with git apply? What if the human types it out by
hand while reading the diff?
In my view, each of the following warrants a Co-Authored-By: line:
- Having the AI investigate the root cause of a bug
- Having the AI explore ways to fix it
- Having the AI write the fix
- Having the AI verify the behavior after the fix
This is wider than "the AI wrote the code", and on purpose: no reviewer can
tell these apart from the diff, so a line that has to be judged per patch is
a line nobody can apply. Stack Overflow was a reference; an AI writes the
text that ends up in the commit, which is why it gets a trailer at all. One
form for all of it, Co-Authored-By:, is what I would settle on.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
Julien Voisin
jvoisin left a comment (vim/vim#21318)Would you argue that this would also be the case if the AI was replaced with a human? Like, having my pal verify that the bug I fixed is indeed fixed, is enough to mandate them being added as Co-Authored-By: in the commit?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#21318)I have more than once added a Co-Authored-By: line after submitting a
PR, for someone who pointed out a contradiction with the spec or fixed a
bug in my code.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
Shay Hill
ShayHill left a comment (vim/vim#21318)@h-east , I would elect to not label many of your examples. This isn’t about having the most precise definition. It is purely a matter of intent and practicality.
The intent of the label is that some extra mechanism (additional review or even additional awareness) will be applied to commits with this label.
If the label applies to 80% or commits, that “extra” will have to be extremely thin. If the label only applies to 20% of commits, that extra can be more secure.
The idea with such triggers is that you have to decide how often you want them applied (that will be determined by the cost of the “extra” you’d like to engage) … then select a definition that aims at triggering the “extra” about that often.
A definitionally precise label might have the virtue of honestly or forthrightness, but it has little practical value if applied to 95% of commits.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#21318)@ShayHill
Co-Authored-By: is attribution, not a trigger for extra review. It
records who worked on the patch; how closely a patch is reviewed is
decided by the diff.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
Shay Hill
ShayHill left a comment (vim/vim#21318)If that’s the case, then why are we standardizing on a label at all?
I have no opinion on the purpose of the label. However, IF the label is meant as a caution, then we need to be wary of over-applying it. You can kill yourself my taking too many aspirin. You can also kill yourself by drinking too much water. If we want a label to act as a warning, we can’t label everything.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
h_east
h-east left a comment (vim/vim#21318)I looked into a few well-known projects.
It seems that very few projects use Co-authored-by: for AI.
Instead, I got the impression that many projects use Assisted-by: or Generated-by:.
After thinking it over, I'm withdrawing my proposal.
Assisted-by: seems like the best fit.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.
dkearns
dkearns left a comment (vim/vim#21318)It is generally settled law across multiple jurisdictions that tools cannot be authors and LLMs are tools. Irrespective, it should be common sense.
I'm not aware of any respected project enforcing the use of the co-authored-by trailer. There are probably exceptions to prove the rule. The only promoters of this use are the companies themselves who snuck them in and received general derision in response.
Why on earth do you want GitHub to pick up the trailer? Why do you want to provide free advertising for these companies?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.