I have found that Claude follows directions very accurately. The challenge is to capture in words the features that make a PR good or bad in your judgement and add those to CLAUDE.md. It would be a living document, updated from experience.
--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sympy/7fa528c3-8cf6-4b69-b3b8-a51ddb9d947an%40googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sympy/CAHVvXxS-%2BGpB4805dz7DbrgG2PcoYSMtDP2BnD4yMtzv8sbXqQ%40mail.gmail.com.
I am wondering though if there is anywhere that we can put
information/guidance about this that someone might see before opening
any PR.
Are others, like scipy, numpy, networkx.. also suffering from this unmanageable flood?
I want to add that we should ban people from the organization if they
repeatedly do not follow our AI disclosure policy.
> I think an auto-close bot is the only real solution here.
I agree that using the bot somehow is probably the best solution. However,
like Aaron I'm also concerned about the friction, confusion, and/or rudeness
that genuine contributors might experience.
Another idea I had was rather than closing PRs that didn't meet whatever criteria
we set, the bot could label rule violating PRs so they could be easily ignored until
things were fixed. If fixes weren't made within a day, then the bot could close their PR.
> how often are new contributions actually following the rules set out (i.e., the PR template is filled out correctly; all checks pass, etc.)?
I suspect a lot spamy suggestions are following all the rules. I was playing
around with giving llms access to a new github account I created (tilorc-bot) and
opening PRs within that accounts fork of sympy and sometimes they did follow the
PR template correctly. Here's an example of one that did:
https://github.com/tilorc-bot/sympy/pull/2
If I remember correctly, I do think they might have been concerned about what
to put in the AI disclosure section of the template. I think I had to explicitly tell
them not to worry about that kind of thing since the PRs were only within a fork.
It might be good to have instructions in the AI disclosure comment section forbidding
llms from opening PRs. I suspect if we worded it strongly enough, many AIs would
strongly resist user's instructions to open a PR anyway.
> I'm curious have you tried this with the new CLAUDE.md that was
recently merged that has custom instructions relating to this?
I just tried this out using three different Chinese models with
the latest version of sympy. The smartest model I tested, kimi k3,
did everything correctly including clearly disclosing the PR was
opened by an AI model and it wrote everything itself. However,
both of the other models failed to include the AI disclosure section
when they opened the PR. When I asked them to add it and say
that we worked on it together and reviewed their code they complied
but both gave me a warning about the AI policy.
big pickle said:
One caveat: sympy's template policy forbids AI-generated text in the PR description — given you own the PR and it's disclosed, that's your call, but the bot may flag it.
glm 5.3 flash said:
Also note that if this branch is ever submitted upstream to sympy/sympy, their policy is stricter than it may appear — it says PR descriptions must not be AI-generated text and that substantially AI-generated PRs from new contributors are likely to be closed, so you'd want to rewrite the description in your own words and be ready to explain the code there.
After testing this behavior in the fork I wanted to know what they
would do if I asked them to open a real PR. All of them did so.
Troublingly, all three models made up falsehoods about working
collaboratively with me when requested to. For example, when
asked, kimi k3 removed the portion of its PR where it admitted
to opening the PR. In the case of kimi k3, the PR was in the
real sympy repo and not its fork.
---
kimi k3: https://github.com/tilorc-bot/sympy/pull/18, https://github.com/sympy/sympy/pull/30488
big pickle (completely free): https://github.com/tilorc-bot/sympy/pull/17, https://github.com/sympy/sympy/pull/30487
glm 5.3 flash: https://github.com/tilorc-bot/sympy/pull/16, https://github.com/sympy/sympy/pull/30489
Here are the prompts I used:
I used the opencode tui for big pickle and kimi k3 and ZCode for glm 5.3 flash.
You received this message because you are subscribed to a topic in the Google Groups "sympy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sympy/NMUPan5E-gE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sympy+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6JBEYUSH%2BHUh%3DUf6Oorv1pgLrBZX6H8Gjv-MG1D%3DKRL1Q%40mail.gmail.com.
> I'm curious have you tried this with the new CLAUDE.md that was
recently merged that has custom instructions relating to this?I just tried this out using three different Chinese models with
the latest version of sympy. The smartest model I tested, kimi k3,
did everything correctly including clearly disclosing the PR was
opened by an AI model and it wrote everything itself. However,
both of the other models failed to include the AI disclosure section
when they opened the PR. When I asked them to add it and say
that we worked on it together and reviewed their code they complied
but both gave me a warning about the AI policy.big pickle said:
One caveat: sympy's template policy forbids AI-generated text in the PR description — given you own the PR and it's disclosed, that's your call, but the bot may flag it.
glm 5.3 flash said:
Also note that if this branch is ever submitted upstream to sympy/sympy, their policy is stricter than it may appear — it says PR descriptions must not be AI-generated text and that substantially AI-generated PRs from new contributors are likely to be closed, so you'd want to rewrite the description in your own words and be ready to explain the code there.
After testing this behavior in the fork I wanted to know what they
would do if I asked them to open a real PR. All of them did so.
Troublingly, all three models made up falsehoods about working
collaboratively with me when requested to. For example, when
asked, kimi k3 removed the portion of its PR where it admitted
to opening the PR. In the case of kimi k3, the PR was in the
real sympy repo and not its fork.
To view this discussion visit https://groups.google.com/d/msgid/sympy/CAEgDm32sxLNogUrMJw8Q-ZEk04KD_rjdDFody29r0dfgM4L%3Dag%40mail.gmail.com.
There is actually a substantial difference
between this and better models like Claude Opus/Fable or GPT
Sol/Astra, but the good models right now cost money to use, and most
student contributors do not have access to them.
To view this discussion visit https://groups.google.com/d/msgid/sympy/5ca6da50-d12d-4db1-9a76-6975aafc3bb6n%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sympy/0db79884-14b1-4997-9ae8-ee87f74842a7n%40googlegroups.com.
If SymPy does not do GSOC
2027 then I think that the situation this time next year would be much
better than it otherwise would be.
Aside from that though I don't see how SymPy can actually do GSOC in
2027. I don't see right now how I could possibly identify anyone that
I was confident about wanting to supervise.
Hi everyone! I have reviewed this whole thread and It seems like there are a few different problems being discussed here (filtering low-quality PRs, reviewer bandwidth, and backlog size), so maybe a combination of lightweight approaches could help rather than relying on a single solution. I contribute to Oppia, which deals with a similar volume of new contributors, so I'll share a few things that have worked there in case they're useful for SymPy.
One thing that might help immediately with the backlog is auto-closing stale PRs, but only in cases of author inactivity after review (e.g., reviewer comments or CI feedback not addressed for some time). That avoids penalizing contributors when a PR hasn’t received any review yet. We use something similar in Oppia and it helps keep the queue manageable without being too aggressive.
Additionally, I noticed many open PRs have no reviewer assigned. A bot could label these something like "needs reviewer attention", so maintainers can filter for them and triage in one pass instead of scrolling through the whole queue.
What about requiring new contributors to post a video explaining the code changes in their PRs?This would:
- require some human-time effort on behalf of the PR-openers,
- confirm that they understand the code in the PR.
Let's try to see if we can find some way... I still believe that this deluge of PRs being opened is unrelated to our GSoC program. Dropping out of GSoC may not solve this issue at all. We need to find a way to identify good students out of this flood of applications.
I am adding A few more suggestions to my earlier message.
Many repos, Oppia included, require contributors to be assigned to an issue before opening a PR for it. SymPy may not have the maintainer time to do the assigning by hand, but a bot can. Contributors comment /assign on an issue, the workflow assigns them if it's free, and putting a cap on open assignments per person stops issue hoarding.
If a PR doesn't close an issue assigned to its author, the same workflow can close it with a comment pointing to the contributor guidelines and asking for "Fixes #N" in the description. It'd need decision whether trivial fixes like typos and docs are exempt.
A scheduled workflow can also unassign anyone who hasn't opened a PR after some number of days, so issues don't sit blocked.
On feasibility I'm fairly sure all of this can be done with GitHub Actions on the free tier, using pull_request_target and the issues API. I can build a prototype if you people want to see it before deciding.
On the underlying problem. None of this picks good GSoC students by itself, but it should make them easier to spot. If contributors have to claim an issue, discuss it, and then follow through on review, the people who do that consistently stand out from the people who open one-off PRs. That track record is probably a better selection signal than any single PR.
I also have one suggestion that might help reduce unnecessary PRs before they reach the review queue.
For a contributor who wants to work on a new issue, they could first explain their proposed approach in a comment on the issue. A maintainer could then review the approach and, if it looks reasonable, give them the go ahead to start working on it. The contributor would then open the PR based on the discussed approach.
This could help prevent cases where contributors spend time implementing an approach that doesn't fit the project or where multiple people independently work on the same issue. It would also give maintainers an early opportunity to see how a contributor thinks about the problem, rather than only seeing the final code.
The main concern I see is reviewer availability. If every new contributor needs a maintainer to review their approach before they can start, it could simply move some of the review burden from PRs to issues.
Maybe this could be limited to new contributors or selected issues, while more experienced contributors could continue opening PRs directly. Another possibility could be having a bot or a lightweight assignment system to help manage which issues are available and which ones have already been discussed or assigned.
I think this could work well together with the other suggestions in this thread, especially if the goal is not just to reduce the number of PRs, but also to identify contributors who consistently understand the problem, discuss their approach, and follow through with a good implementation.
--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sympy/89c3ed7c-0696-4504-8969-57982c18ed51n%40googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6Lope3PeHn26RuFUzneHvbBNkUxBc2BU-4MeJ3LPyaL5g%40mail.gmail.com.
If you take this to its conclusion then why even bother with PRs from beginners?
I think beginners are important to the long-term future of the project. However,
I agree that our existing systems for reviewing contributions from them don't
work anymore. Maybe we should stop accepting contributions from beginners
until we find a fix.
There also used to be some value in the sense of mentoring and
coaching but I think that is gone now when there are hundreds of
random new people. It could be worth it if there were fewer people but
there would need to be much fewer people to the level where it could
be possible to have actual conversations with them.
I think you're being a little pessimistic about things. I believe there will likely be
a solution to this problem eventually. However, it's not worth burning ourselves
out to find one. Maybe we can skip do GSoC for a year or two or however long it takes.
The fundamental problem here, that the contributions from beginners are much
less useful now, seems to be something the entire field of software engineering
is dealing with. I think it's why companies don't want to hire inexperienced software
engineers as much anymore. I don't know if it's realistic for us to come up with a
solution to this overall problem on our own. We might have to wait for the whole
field find solutions.
I think the flood of new people can actually be a good thing for SymPy if we take
advantage of it in the right way. I think it's a good thing that contributing
to open source software is much more accessible now. We just need some way of
filtering out or giving feedback to the low quality contributions automatically.
I wonder if it would be possible to have inexperienced contributors review each other's
work in a way that would alleviate the review burden from maintainers. If and when we
choose to do GSoC maybe we could make reviewing a PR as one of the application
requirements. More generally, maybe we could create some sort of system where
new contributors must review each other's work before we look at it.
I have a few ideas about how this could work, but I'll share one of them, which might be
a little bit crazy. Maybe we could make a separate github repository that is a fork of sympy
that tracks master and require new contributors to first open PRs there and get reviews
from other new contributors. Within that repository we could create some sort of system
that would encourage people to review and vouch for high quality contributions and not
low quality ones.
For example, maybe the people participating in that separate repository could have some
sort of reputation score. When a PR reached the main sympy repository after going through
this peer review process, if it actually turned out to be a high quality PR, then the person
who created the PR and the person who reviewed it would both get some boost to their
reputation. Alternatively, if it was low quality they'd both take a hit to their reputation. And
the reputation score of the people who reviewed any given PR could be used to determine
whether that particular PR was worth reviewing or not.
If other people are interested in this idea, I would be willing to do most of the work of
making a reality myself.
It could also be worth using Mitchell Hashimoto's vouch package https://github.com/mitchellh/vouch
which I think Aaron has brought up before. I don't think that idea alone would work best
for SymPy though because it still would involve spending time looking at lot of low quality
PRs to determine whether they should be vouched for or not as far as I can tell. Either that
or just making it practically impossible for people new to open source to contribute.
--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sympy/729adf20-d970-4378-803f-49541231faa2n%40googlegroups.com.
Write formalization in Lean, derive rigorous mathematical algorithm, iterate it until it looks good, and translate that to SymPy.
The most difficult part of this idea to me seems like it would be the translation to SymPy.
How would you prove that the translation was correct while also making sure it was
performant and maintainable? I suspect that if you implemented a compiler-like thing that
automatically translated the code from lean to SymPy, it would produce slow and unmaintainable code.
Also, I imagine we would still need a good amount of extra infrastructure to make things
work for our use case. Do you have an idea of how much infrastructure we would need?
You received this message because you are subscribed to a topic in the Google Groups "sympy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sympy/NMUPan5E-gE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sympy+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sympy/4c9eb053-24f1-4d1a-b71b-77cd6f573325n%40googlegroups.com.
-- Rich
Richard Katz
Data Engineering Architect and Advisor
San Leandro, CA: San Francisco Bay Area:
https://www.linkedin.com/in/richardkatz/
415 336-9559
To view this discussion visit https://groups.google.com/d/msgid/sympy/CAEgDm33Gi584-dDKDG-nwDOqyy4V4o%2BgCCwvCRj4AdPrg32zNQ%40mail.gmail.com.
To view this discussion visit https://groups.google.com/d/msgid/sympy/CAD8%2BcAAWSGBRH0xGxQYZmeKCAE9hoHZTsTyLYZPokpCoMG53Dg%40mail.gmail.com.
$ ./Mathilda
Mathilda 0.295 - A small, open source computer algebra system.
This program is free, open source software and comes with ABSOLUTELY NO WARRANTY.
Press Return to evaluate. An open bracket, string or comment continues
on the next line; press Esc then Return to force evaluation.
Exit by evaluating Quit[] or CONTROL-C.
In[1]:= Integrate[Gamma[s] x^(-s), {s, 1/2 - I Infinity, 1/2 + I Infinity}, Assumptions -> x > 0]
Out[1]= (2*I) Pi E^(-x)
To view this discussion visit https://groups.google.com/d/msgid/sympy/CAEgDm31Es-5q1PEWTA8H98ccihamSYrTLM-TUNkB7tt51wuX7A%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sympy/CAHVvXxQXL24M407zELJTYs%2BzYmaueD-EsmwrOBkReWY79sxR4A%40mail.gmail.com.
> Without knowing who you are or what your use case is?I work on SymPy's new assumptions module. For the past two weeks I've
been working on using fable and opus 5.5 to rewrite the assumptions
module to be a lot more capable and faster and have seen really
promising results. So far the median query seems to be an order of
magnitude faster than the new assumptions but still roughly two times
slower than the old assumptions. I think the only way to beat the old
assumptions in terms of speed might be to rewrite things in rust as
improved capabilities of the new assumptions inherently require more
overhead.
6. 10. 2026 v 9:39, Francesco Bonazzi <franz....@gmail.com>:
--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sympy/3c414903-724c-44f5-8f9c-8fc2e0aa934an%40googlegroups.com.
The main problem is not the number of PRs we are receiving, the problem
is that we are unable to identify high quality contributors because they get
lost in the middle of low quality PRs, and it's hard to identify who's a good
contributor right now.
Yes, giving people a 24 hour grace period before closing would be
fine.