Blog post on NUMFOCUS concerns

[Jason Moore]

Hi,

I was browsing Paul Invanov's blog today and came across this article:

https://pirsquared.org/blog/numfocus-concerns.html

We are part of NUMFOCUS, so I'd say it is important to at least be aware of this. I do not have an opinion yet myself, but wanted to share.

Also, this is what attracted me to his blog today: https://pirsquared.org/blog/current-challenges-in-free-software-and-open-source-development.html and is food for thought about whether we should have some policy to not accept AI generated code due to the likelhiood of OSS licenses being violated. There are examples of open source projects implementing such rules.

Jason

moorepants.info
+01 530-601-9791

[Aaron Meurer]

On Sun, Jan 26, 2025 at 1:40 AM Jason Moore <moore...@gmail.com> wrote:
>
> Hi,
>
> I was browsing Paul Invanov's blog today and came across this article:
>
> https://pirsquared.org/blog/numfocus-concerns.html
>
> We are part of NUMFOCUS, so I'd say it is important to at least be aware of this. I do not have an opinion yet myself, but wanted to share.

I love Paul, but I think that blog post is mostly FUD and these
concerns about the 501c6 are not something to be worried about. I had
many discussions with various people about this and related issues at
the NumFOCUS summit last year and I'm confident that everything is OK.
The 501c6 is more or less just a way for NumFOCUS to raise more money,
as it makes it easier for some types or organizations to give. But the
whole thing is being set up so that it does not affect the
relationship with the projects (like SymPy). Unfortunately the summit
was several months ago so I don't remember all the details, but maybe
some more details have been posted publicly since then. But the
biggest high level takeaway I had from the summit is that NumFOCUS
really does care about the open source projects and has their best
interests as community run projects at heart, and also that it is
probably the only fiscal sponsorship organization that fits that
description (i.e., moving away from NumFOCUS would be a bad idea).

If you're still concerned about this, I would suggest emailing Andy
Terrel about this (or maybe we can get him to respond here). He is on
the NumFOCUS board and also is a (from a long time ago) contributor to
SymPy.

>
> Also, this is what attracted me to his blog today: https://pirsquared.org/blog/current-challenges-in-free-software-and-open-source-development.html and is food for thought about whether we should have some policy to not accept AI generated code due to the likelhiood of OSS licenses being violated. There are examples of open source projects implementing such rules.

I personally don't think LLM outputs violate OSS licenses. The closest
something might come to being an issue is if an LLM generated a
significant block of code that is verbatim copied from something else.
That's not only unlikely in general due to the way LLMs work, but it's
unlikely for SymPy because most code that would be written for SymPy
is not something that would already have appeared somewhere else.

At any rate, the ship has basically sailed on this. I would expect a
large fraction of SymPy contributors already make use of LLMs in some
form or other, whether it's using code completion from something like
GitHub copilot or prompting a tool like ChatGPT or Cursor to help
refactor or write a function. Frankly if you're not using LLMs at all
to help you code you should because they are very useful tools.

Looking at some other projects, scikit-image added "no ai
contributions" policy and they ended up having to remove it
https://github.com/scikit-image/scikit-image/pull/7429. scikit-learn
has a policy disallowing completely automated contributions
(contributions that have no human in the loop)
https://github.com/scikit-learn/scikit-learn/blob/main/doc/developers/contributing.rst#automated-contributions-policy.
I think that's a good policy, but also I don't know if it's something
we need to write down unless it starts to become an issue (has it?).

There's also, separately, the question of the quality of LLM generated
code. I think that we need to use the GitHub review process we have
always been using to ensure the SymPy code remains high quality
regardless of its source. This means the usual things: good, thorough
tests that check for correctness, readable code, avoiding various
antipatterns, etc. LLM generated code won't always fit these
parameters, especially if not prompted correctly.

I think the biggest concern here is contributors (especially newer
contributors) contributing code that exclusively comes from an LLM
without any thought from the contributor themselves. This is
especially likely from potential GSoC applicants. This we should
disallow, because LLMs are not good enough to do this right now, and
in the case of a GSoC applicant, it tells us nothing about their
coding ability. Basically, any contributor to SymPy should be
responsible for all the code they contribute. This especially makes it
harder to evaluate GSoC applicants, but that's unfortunately the world
we live in and we just need to learn how to evaluate people better
(happy to discuss ideas for this. Should we do video call interviews
with top GSoC applicants?)

Aaron Meurer



Aaron Meurer

>
> Jason
> moorepants.info
> +01 530-601-9791
>
> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/sympy/CAP7f1AjsFmZv%2BZGB2RVH9%3DS4KcaR%2B%2B0QtG8hJ1hwKYKLOXg%3D9w%40mail.gmail.com.

[Jason Moore]

Dear Aaron,

Paul does raise numerous specific issues in his post and just saying "there is nothing to worry about" doesn't allay any concerns that may form from reading his post, at least not for me. If you look at existing organizations that are 501(c)(6) orgs, none really give off any warm fuzzy feelings nor do they come off as altruistic. So that does warrant concern.

And, as for the legality of generative AI tools, that will be decided by courts around the world at some point. I just listened to this recent Freakonoimcs: https://freakonomics.com/podcast/how-to-poison-an-a-i-machine/ and the professor pointed out the standard strategy of many companies, which is to do something that is legally ambiguous, but do it fast and broadly so people are hooked on it before the ligation system can even evaluate it, then by the time the courts do get to it, they rule in the company's favor because it is too ingrained to undo. The software licenses I apply to my open source code say that my license must be carried along with any copied and reworked versions of that code. I don't think the companies or people using the tools are following these licenses. If they don't have to follow them, then why does anyone at all have to follow them?

Jason

moorepants.info
+01 530-601-9791

To view this discussion visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6J97CtZNyN4qkx_%3DyEaj67yGB%3Dx%2BLLXx9gS2KKmtwKjcg%40mail.gmail.com.

[Oscar Benjamin]

On Mon, 27 Jan 2025 at 22:21, Aaron Meurer <asme...@gmail.com> wrote:
>
> There's also, separately, the question of the quality of LLM generated
> code. I think that we need to use the GitHub review process we have
> always been using to ensure the SymPy code remains high quality
> regardless of its source. This means the usual things: good, thorough
> tests that check for correctness, readable code, avoiding various
> antipatterns, etc. LLM generated code won't always fit these
> parameters, especially if not prompted correctly.

It would be great if we could have an LLM that could review basic PRs
rather than being used by the PR author to write the code. Much of the
back and forth in some PRs is about very basic things like "there
should be some tests" that could be easily handled. I have seen some
other projects using LLMs like this but it didn't seem to be very
useful.

> I think the biggest concern here is contributors (especially newer
> contributors) contributing code that exclusively comes from an LLM
> without any thought from the contributor themselves. This is
> especially likely from potential GSoC applicants. This we should
> disallow, because LLMs are not good enough to do this right now, and
> in the case of a GSoC applicant, it tells us nothing about their
> coding ability. Basically, any contributor to SymPy should be
> responsible for all the code they contribute. This especially makes it
> harder to evaluate GSoC applicants, but that's unfortunately the world
> we live in and we just need to learn how to evaluate people better
> (happy to discuss ideas for this. Should we do video call interviews
> with top GSoC applicants?)

This is where we are heading with students in my University.
Programming is increasingly assessed by interview. It just isn't
reasonable any more to take the raw code submitted and evaluate it at
face value for anything that is remotely near beginner level. There
always was the possibility that a student might have paid someone else
to do their homework but now any reasonable beginner programming
assignment can be done in seconds by LLMs that are available for free.

--
Oscar

[Aaron Meurer]

On Mon, Jan 27, 2025 at 3:09 PM Jason Moore <moore...@gmail.com> wrote:
>
> Dear Aaron,
>
> Paul does raise numerous specific issues in his post and just saying "there is nothing to worry about" doesn't allay any concerns that may form from reading his post, at least not for me. If you look at existing organizations that are 501(c)(6) orgs, none really give off any warm fuzzy feelings nor do they come off as altruistic. So that does warrant concern.

I'm sorry for not being more specific. LIke I said, the summit was
several months ago so I don't remember details. I can say there I had
some of the same concerns as you and had a lot of conversations about
these things there with various people, and based on those
conversations, my personal concerns were assuaged. You can take that
for whatever it's worth. I can't blame you if you if me not being
concerned doesn't allay your own concerns. That's why I suggested you
should reach out to someone from NumFOCUS who knows more about the
details (it's also possible that things have changed since September).
I suggested Andy Terrel because I know him and trust him, and he's an
old SymPy contributor so I know he cares about this project. But do
reach out to others as well if there is someone else deeply involved
in NumFOCUS right now who you trust. I think as long as we are
relying on hearsay (including my own hearsay) we are just spreading
FUD, so it would be better to get the facts.

Aaron Meurer

> To view this discussion visit https://groups.google.com/d/msgid/sympy/CAP7f1AgYpWdQE%3DbYXi0Mq2Fqq4aGF_yCcqfsKQydvG3MX4ww8w%40mail.gmail.com.

[Aaron Meurer]

On Mon, Jan 27, 2025 at 3:18 PM Oscar Benjamin
<oscar.j....@gmail.com> wrote:
>
> On Mon, 27 Jan 2025 at 22:21, Aaron Meurer <asme...@gmail.com> wrote:
> >
> > There's also, separately, the question of the quality of LLM generated
> > code. I think that we need to use the GitHub review process we have
> > always been using to ensure the SymPy code remains high quality
> > regardless of its source. This means the usual things: good, thorough
> > tests that check for correctness, readable code, avoiding various
> > antipatterns, etc. LLM generated code won't always fit these
> > parameters, especially if not prompted correctly.
>
> It would be great if we could have an LLM that could review basic PRs
> rather than being used by the PR author to write the code. Much of the
> back and forth in some PRs is about very basic things like "there
> should be some tests" that could be easily handled. I have seen some
> other projects using LLMs like this but it didn't seem to be very
> useful.

The GitHub copilot review thing that's built-in to PRs is actually not
terrible (you have to join this waitlist to use it
https://github.com/github-copilot/code-review-waitlist). I was a
little surprised because I thought it would be mostly noise, but they
seem to have heavily biased it towards not commenting on something
unless it's actually a real issue (so it will miss a lot of stuff). I
don't know if there's any way to tell it about project-specific coding
standards, though, which is probably something we'd want. We could
experiment with cursorrules, which are a way to tell the AI in the
Cursor editor project-specific guidelines, but I don't know if those
apply to any PR reviewing tools. It would help any contributor who
uses Cursor. Honestly this is something where tooling is still not
quite developed as well as it could be yet.

>
> > I think the biggest concern here is contributors (especially newer
> > contributors) contributing code that exclusively comes from an LLM
> > without any thought from the contributor themselves. This is
> > especially likely from potential GSoC applicants. This we should
> > disallow, because LLMs are not good enough to do this right now, and
> > in the case of a GSoC applicant, it tells us nothing about their
> > coding ability. Basically, any contributor to SymPy should be
> > responsible for all the code they contribute. This especially makes it
> > harder to evaluate GSoC applicants, but that's unfortunately the world
> > we live in and we just need to learn how to evaluate people better
> > (happy to discuss ideas for this. Should we do video call interviews
> > with top GSoC applicants?)
>
> This is where we are heading with students in my University.
> Programming is increasingly assessed by interview. It just isn't
> reasonable any more to take the raw code submitted and evaluate it at
> face value for anything that is remotely near beginner level. There
> always was the possibility that a student might have paid someone else
> to do their homework but now any reasonable beginner programming
> assignment can be done in seconds by LLMs that are available for free.

The reason I mentioned interviews is because some people from other
projects mentioned that they do it at the GSoC mentor summit.

Aaron Meurer

>
> --
> Oscar

>
> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.

> To view this discussion visit https://groups.google.com/d/msgid/sympy/CAHVvXxTcT%3Dnf87h1NT1sY9DXTJrOsX1Xj7arqc3ZP1Snv8Ck7A%40mail.gmail.com.