AI-assisted SymPy audit: human-verified bug reports

[Ernest Ryu]

Hello,

My name is Ernest Ryu, and I am a professor of applied mathematics at UCLA. Long-time SymPy user, first time posting here.

Recently, my students and I used OpenAI's Codex coding agent to audit parts of the SymPy codebase. To our surprise, we found many relatively simple correctness issues. So, wrote up a report:

https://github.com/xinjiegit/agentic-bug-hunt-sympy/blob/main/report.pdf

Interested to hear what the community thinks! If these reports are useful, we would be happy to turn them into Github PRs. But since these were AI-discovered/generated, though human-verified, I wanted to check here first to see what people think.

Sincerely,
Ernest

[Jason Moore]

Dear Ernest,

I find it a bit unusual that you write an AI generated report on such things. SymPy is 20 year old software with a long established method for reporting bugs. If you find a bug, you can report it at:

https://github.com/sympy/sympy/issues

You may even find that the bug is already there.

We actually have 4690 open issues and would love to have contributions to help fix those and any new ones. If you and your students would like to help improve SymPy here is how you can get started:

https://docs.sympy.org/dev/contributing/index.html

Sincerely,

Jason

moorepants.info
+01 530-601-9791

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sympy/56181e1d-0b48-4ee2-9f08-e3f072ea6fa4n%40googlegroups.com.

[Oscar Benjamin]

Hi Ernest,

Thanks for sharing this. I think many of these are likely already
known in existing issues. Some of them might be things that can be
fixed easily. Some of them will sound simple but will actually be
quite difficult to fix.

If you and your students want to make pull requests to fix some of
these then that could be useful if you can manage it to ensure that
they are high quality PRs that are easy to review. I'm happier to
review a PR if someone else has already reviewed or vouched for it and
if it fixes something important in an unambiguously correct way.

Note though that at the moment sympy development is struggling under
the weight of too many PRs. People are already opening more AI PRs
than reviewers can manage. Most PRs from new people pretty much just
get ignored right now as you can see if you look through the list of
recent ones:

https://github.com/sympy/sympy/pulls

I am not sure what is a good way to manage sympy development going
forwards but right now there are so many new PRs that I just skip over
most of them immediately. Other reviewers seem to do the same and now
most PRs don't even get a single reply comment.

So, yes turning those findings into PRs could be useful but it would
need to be done in some way that is more manageable for reviewers than
just a wave of random AI PRs from random new people.

Oscar

[Ernest Ryu]

Hi Jason,

Thank you for your response.

To clarify: the report was written by us (humans) and was not AI-generated. The bugs were discovered by AI, but we humans wrote up the report and verified each one of the bugs.

We also did dedup against
https://github.com/sympy/sympy/issues
and I believe many of the bugs we found are not already flagged there.

The report is also intended to serve as a case study on how AI can help improve open-source software. I understand that open-source projects are typically bottlenecked by contributors willing to do the actual fixing, which is why I wanted to check in before filing a batch of new issues to the issue tracker.

The purpose of my message here is to gauge whether the community would consider human-verified, AI-discovered bugs like these a productive form of engagement. If so, we're happy to file them in the issue tracker and open PRs.

Sincerely,

Ernest

[Jason Moore]

Dear Ernest,

We accept reports of any bugs in our issue tracker regardless of how they are discovered. We do expect human communication though, as per our AI policy: https://docs.sympy.org/dev/contributing/ai-generated-code-policy.html

Jason

moorepants.info
+01 530-601-9791

To view this discussion visit https://groups.google.com/d/msgid/sympy/5a20082b-58d9-4cf2-b36e-9fe2bcf581d9n%40googlegroups.com.