PR descriptions being deleted

[Oscar Benjamin]

Hi all,

So far today 10 PRs have been closed (by Jason or by me) because the
author removed the PR description template. It has always been the
case that the release notes check would fail the PR if that part was
removed but I think more needs to be enforced. In particular the
current AI policy says that a PR should include an "AI Generation
Disclosure" and that is part of the template. It is quite common that
either just the AI part of the template is removed or that the whole
template is removed.

I think we maybe need a bit of a rethink about how the template looks
and what it says in the AI part. In the template it says:
```
#### AI Generation Disclosure

<!-- If this pull request includes AI-generated code or text, please disclose
the tool used and specify which lines were generated. Disclosure is not
required for minor assistive tasks, such as spell-checking or code reviewing,
in primarily human-authored work. Otherwise, leave this area blank. Read our
Policy on AI Generated Code and Communication at
https://docs.sympy.org/dev/contributing/ai-generated-code-policy.html. -->
```
The "leave this area blank" part now seems ambiguous to me given that
many people just delete it. What we want here is a clear statement
about how AI was used or not used. The reason that this is important
is because if someone says "No code was written by AI" but the code is
obviously written by AI then we can just close the PR for that reason.
We need an explicit statement though so I don't think it is enough to
let people leave the area blank but maybe the wording in the template
does not make this obvious.

More generally I think that the template is just too long so I am not
surprised that in some cases people would just delete it or not notice
what the important parts are. I also wonder if a lot of contributors
just don't even see the template. My suspicion is that many people now
open a PR by clicking an "open a PR" button (or an AI prompt?) in
their editor meaning that they don't go to the actual GitHub website
themselves and edit that text box directly.

We probably need to have a bot that closes PRs if they don't match the
template or maybe it could be just that sympy-bot checks more things
since it seems most people seem to get the message about the release
notes check from sympy-bot.

--
Oscar

[Aaron Meurer]

I'm sure people don't read it, but I suspect in 99% of the cases where it's "deleted" it's because someone opened the PR without using the web interface, and a majority of the time that's because they used some AI agent to do it (that doesn't necessarily mean autonomous though, it could just mean they promoted Claude code or codes to open the PR after writing it). It sucks a little bit because this is a legitimate way to open PRs, but if it correlates highly with slop, then I think we should just ban it. 

I believe we can replace the template with a form. Or is that only possible for issues?

We probably need to have a bot that closes PRs if they don't match the
template or maybe it could be just that sympy-bot checks more things
since it seems most people seem to get the message about the release
notes check from sympy-bot.

We can definitely get sympy-bot to do these checks. We can also have it do the closing if we want it to. 

Aaron Meurer 

--
Oscar

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sympy/CAHVvXxSsMFf06r0WN6mdhkb2%3DMBtrEV2xZ-DrN9h2F1H6wVEbw%40mail.gmail.com.

[Oscar Benjamin]

On Sun, 22 Feb 2026 at 15:41, Aaron Meurer <asme...@gmail.com> wrote:
>
> On Sun, Feb 22, 2026 at 7:03 AM Oscar Benjamin <oscar.j....@gmail.com> wrote:
>>
>> More generally I think that the template is just too long so I am not
>> surprised that in some cases people would just delete it or not notice
>> what the important parts are. I also wonder if a lot of contributors
>> just don't even see the template. My suspicion is that many people now
>> open a PR by clicking an "open a PR" button (or an AI prompt?) in
>> their editor meaning that they don't go to the actual GitHub website
>> themselves and edit that text box directly.
>
>
> I'm sure people don't read it, but I suspect in 99% of the cases where it's "deleted" it's because someone opened the PR without using the web interface, and a majority of the time that's because they used some AI agent to do it (that doesn't necessarily mean autonomous though, it could just mean they promoted Claude code or codes to open the PR after writing it). It sucks a little bit because this is a legitimate way to open PRs, but if it correlates highly with slop, then I think we should just ban it.

I would have expected Claude/codex to be smart enough to see that
there is a PULL_REQUEST_TEMPLATE.md file and use that. In fact I just
checked this for
https://github.com/sympy/sympy/pull/29127
I prompted codex with:
```
I want to make a PR with the python-flint typing related changes on
this branch. Can you write a description for the PR?
```
The first thing codex did was to read the pull request template and
then the commits and then it produced a PR description that matches
the template fully. A couple of notes about that though:

In the references to issues section it did not include a reference to
the associated issue. This is a problem that I have been seeing a lot
recently, that a PR is obviously related to an issue but no link is
given to that issue. It makes sense because I didn't tell it about the
issue just now when prompting for the description.

It also lied in the AI Generation Disclosure section and literally wrote:
```
#### AI Generation Disclosure

Used ChatGPT to help draft PR text only. No code changes were
AI-generated in this PR.
```
All code in the PR was generated by codex (GPT-5.3) but with heavy
prompting and guidance from me. I'm not sure if codex can figure that
out because I did this in a new session so it won't remember what
happened before. I didn't use ChatGPT at all so both sentences in that
disclosure are false and it made them up without me asking it to do
that.

It is very worrying that the AI will lie automatically about AI use on
behalf of the person using it.

> I believe we can replace the template with a form. Or is that only possible for issues?

I think it is only possible for issues.

>> We probably need to have a bot that closes PRs if they don't match the
>> template or maybe it could be just that sympy-bot checks more things
>> since it seems most people seem to get the message about the release
>> notes check from sympy-bot.
>
> We can definitely get sympy-bot to do these checks. We can also have it do the closing if we want it to.

That might be the best thing to do in terms of PR notification noise.
Then sympy-bot can keep complaining until the PR description is edited
to match the format.

--
Oscar

[Aaron Meurer]

On Sun, Feb 22, 2026 at 10:23 AM Oscar Benjamin

We can reword the template so that it is also a suitable prompt for
AI. Ironically enough, people probably won't read the template, but if
an AI agent does decide to pull it up, it will actually read what it
says. We can also add an AGENTS.md that asks agents to do something
like always add a co-author line in the commit message (Claude does
this automatically but I'm not sure if codex does). That also serves
as an effective disclosure of AI usage.

>
> > I believe we can replace the template with a form. Or is that only possible for issues?
>
> I think it is only possible for issues.
>
> >> We probably need to have a bot that closes PRs if they don't match the
> >> template or maybe it could be just that sympy-bot checks more things
> >> since it seems most people seem to get the message about the release
> >> notes check from sympy-bot.
> >
> > We can definitely get sympy-bot to do these checks. We can also have it do the closing if we want it to.
>
> That might be the best thing to do in terms of PR notification noise.
> Then sympy-bot can keep complaining until the PR description is edited
> to match the format.

It's pretty trivial to make sympy-bot do anything (and ironically
enough easy to change because of AI).

Although I should also note that Heroku is going to stop existing at
some point so we'll need to move it to something else eventually.

Aaron Meurer

>
> --
> Oscar
>
> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.

> To view this discussion visit https://groups.google.com/d/msgid/sympy/CAHVvXxR7UUu0NfQ1WRYpvMOyk5_O32FrhvfzRFhh%3DGr_37WLHw%40mail.gmail.com.