One year of Sage development on GitHub

[Matthias Koeppe]

1 year ago, we announced the completion of our migration from Trac to GitHub.

This was a big and scary change. Many thanks to all who helped with the migration, to all who helped with updating our developer's guide and refining the development processes in the months after the migration, and to all developers for their patience with adapting to the changes.

To our friends and colleagues who have been taking a break from Sage development: It's safe to come back and re-discover the joy of contributing to our project. Take a look at the current version of the developer guide.

We seem to be on a good trajectory. As of today, there are 329 forks of the repository, there's a steady flow of pull requests, and we will soon recover from the dramatic sacrifice of leaving our repo stars with the old GitHub repository. 

Let's also use this anniversary as an opportunity to discuss what still needs improving in our development workflows. I'll start:

1. We have a low development velocity. For example, some simple PRs sit for weeks or months before receiving any review comments. What can we do to improve this?

[Dima Pasechnik]

If Sage didn't include hundreds of unnecessarily vendored packages (mainly related to Jupyter and Sphinx but also more general Python and not only scaffolding, generally known as Sage the distributuoin) it would be considerably less PRs to take care of in total.

We should not try to compete, in effect, with Conda etc, yet we do. This is the primary reason for slowness.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/71ce080f-ad3a-4d71-afad-521c31e732f3n%40googlegroups.com.

[Kwankyu Lee]

1. We have a low development velocity. For example, some simple PRs sit for weeks or months before receiving any review comments. What can we do to improve this?

There have always been PRs or trac tickets with no comment for long time. I guess most of us had such tickets in trac era. I doubt if there is anything for us to do with it, except sage itself attracting more developers as it grows. Sorry for being negative.

[Michael Orlitzky]

On Thu, 2024-02-08 at 11:30 +0000, Dima Pasechnik wrote:
>
> We should not try to compete, in effect, with Conda etc, yet we do. This is
> the primary reason for slowness.
>

My personal stats for the year 2023-02-08 through 2024-02-08:

Commits: 423
Reviews: 38

Zero of those have anything to do with mathematics.

[Matthias Koeppe]

On Wednesday, February 7, 2024 at 1:11:14 PM UTC-8 Matthias Koeppe wrote:

Let's also use this anniversary as an opportunity to discuss what still needs improving in our development workflows. I'll start:

1. We have a low development velocity. For example, some simple PRs sit for weeks or months before receiving any review comments. What can we do to improve this?

Thanks to all who have already reacted. I'll not respond here immediately, as the main purpose of this thread is to collect thoughts from the whole community. 

I'd like to encourage everyone to share their personal take on these questions even after authoritative-sounding responses have come in.

Here's my next prompt:

2. Is our community aware of the sagemath/sage GitHub wiki? https://github.com/sagemath/sage/wiki

- Are the contents of the wiki front page useful?

- Are the links to Issue and Pull Request queries helpful? 

- ...

 

[Matthias Koeppe]

On Thursday, February 8, 2024 at 10:16:58 AM UTC-8 Matthias Koeppe wrote:

2. Is our community aware of the sagemath/sage GitHub wiki? https://github.com/sagemath/sage/wiki

- Are the contents of the wiki front page useful?

Meanwhile I have edited it a bit to offer "suggested activities".

(This is based on a version of the Trac wiki front page just before the transition to GitHub.)

 

[Martin R]

I suggest to remove "no dark arts required", because, at least to me, many of the issues I struggle with actually do require very little knowledge of mathematics but a considerable amount of knowledge of the way sage works.

Apart from that, I was looking for the tutorial telling me how to convert a trac branch into a github PR, it would be good to have it around.

Finally, "Explore the open meta-tickets on larger tasks or topical pages on Algebra, Coding Theory, Combinatorics, Manifolds, Optimization, Polyhedral Geometry, Symbolics." leads to non existing pages for Coding theory, Polyhedral Geometry and Combinatorics.  I don't know whether these pages exist, but I am also slightly sceptical whether there is anybody keeping them up-to-date if they do.  In general, I think that labelling hygiene is a better way to achieve the same goal.

I really like the quick link to pull requests that  involve me!

Best wishes,

Martin

[Martin R]

Found the page: https://github.com/sagemath/trac-to-github/blob/master/docs/Migration-Trac-to-Github.md

[David Lowry-Duda]

On 22:42 Sun 11 Feb 2024, Matthias Koeppe wrote:
>*2. Is our community aware of the sagemath/sage GitHub wiki?*

>https://github.com/sagemath/sage/wiki
>- Are the contents of the wiki front page useful?

I think the existence of two wikis (i.e. the github wiki and https://wiki.sagemath.org/) is confusing.

- DLD

[Dima Pasechnik]

that's due to the Great GitHub Schism

>
> - DLD

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/ZcpeOwLOBAUzDUOL%40icerm-dld.

[Matthias Koeppe]

I agree, and that one needs a separate account to edit https://wiki.sagemath.org/ is also unnecessary friction.

https://github.com/sagemath/sage/issues/33725 has a detailed discussion for possible ways of resolving this, please take a look.

 

[Vincent Delecroix]

I fully second the observation of Michael though it might have few to
do with the github switch. Sage development nowadays does not seem to
be anymore about math research and efficient computations but mostly
about "dependencies", "infrastructure" and "maintenance". I am always
depressed by reading the change logs. It might have been a reasonable
thing if sage was a "stable core Computer Algebra System" on which
further specialized math research libraries would depend on. But the
latter is not the official nor advised way of doing things.

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/665f0fd945ca6301e75489e1922fb03462cda021.camel%40orlitzky.com.

[Matthias Koeppe]

On Monday, February 12, 2024 at 10:55:37 AM UTC-8 Vincent Delecroix wrote:

Sage development nowadays does not seem to
be anymore about math research and efficient computations but mostly
about "dependencies", "infrastructure" and "maintenance". I am always
depressed by reading the change logs. It might have been a reasonable
thing if sage was a "stable core Computer Algebra System" on which
further specialized math research libraries would depend on.

Isn't your https://github.com/flatsurf/sage-flatsurf project doing exactly that?

But the
latter is not the official nor advised way of doing things.

What are you talking about? It's the intentional result of over a decade of the project's policy of sending potential contributors away to go do their contributions in separately maintained projects. 

We have a working mechanism to advertise such projects in our manual.

- We have an index of packages: https://deploy-livedoc--sagemath.netlify.app/html/en/reference/spkg/#optional-packages

- We have a page for each package: https://deploy-livedoc--sagemath.netlify.app/html/en/reference/spkg/sage_flatsurf#spkg-sage-flatsurf

- We have a process for adding package:  https://github.com/sagemath/sage/issues/31164

What we don't have is a clear process to include news about the projects in our release notes. https://github.com/sagemath/sage/wiki/Sage-10.3-Release-Tour

Such contributions to the release notes would be very welcome!

That's by the way a downside of including the project in Sage only as a "pip" package without pinning the version: There's never a PR that bumps the version, hence it does not appear in the Github-generated changelog, and hence there is nothing that prompts anyone to add a narrative to our release notes. 

[Dima Pasechnik]

On Mon, Feb 12, 2024 at 6:55 PM Vincent Delecroix

<20100.d...@gmail.com> wrote:
>
> I fully second the observation of Michael though it might have few to
> do with the github switch. Sage development nowadays does not seem to
> be anymore about math research and efficient computations but mostly
> about "dependencies", "infrastructure" and "maintenance". I am always
> depressed by reading the change logs. It might have been a reasonable
> thing if sage was a "stable core Computer Algebra System" on which
> further specialized math research libraries would depend on.

Indeed, it doesn't have much to do with switching to GitHub. The
problem is that we don't have a
"stable core", we have a rotting, decaying core, which is in no way
helped by spending time on
micromanaging versions of every tiny piece of Jupyter, Sphinx, Python
build infrastructure, etc.
Something that other maths Python projects (apart from Anaconda) just
accept as given.

What's rotten and decaying - well, the most obvious points are:


* pynac (memory leaks, bugs, sketchy or no docs, authors left long time ago)

* commutative algebra, in particular Singular-based (memory leaks,
bugs, no docs, authors either left or are not willing to look into it
much), etc.

* maxima (bugs, bugs, bugs)

* broken optional packages, e.g. p_group_cohomology

> But the
> latter is not the official nor advised way of doing things.
>
> On Thu, 8 Feb 2024 at 14:02, Michael Orlitzky <mic...@orlitzky.com> wrote:
> >
> > On Thu, 2024-02-08 at 11:30 +0000, Dima Pasechnik wrote:
> > >
> > > We should not try to compete, in effect, with Conda etc, yet we do. This is
> > > the primary reason for slowness.
> > >
> >
> > My personal stats for the year 2023-02-08 through 2024-02-08:
> >
> > Commits: 423
> > Reviews: 38
> >
> > Zero of those have anything to do with mathematics.
> >
> > --
> > You received this message because you are subscribed to the Google Groups "sage-devel" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/665f0fd945ca6301e75489e1922fb03462cda021.camel%40orlitzky.com.
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAGEwAAmCkF%2Bj1XUQEuAK2B%3DD6WDT8Ejk9qrVBZ6fVvxg8H19EQ%40mail.gmail.com.

[Nils Bruin]

On Monday 12 February 2024 at 15:58:11 UTC-8 Dima Pasechnik wrote:

What's rotten and decaying - well, the most obvious points are:

* pynac (memory leaks, bugs, sketchy or no docs, authors left long time ago)

* commutative algebra, in particular Singular-based (memory leaks,
bugs, no docs, authors either left or are not willing to look into it
much), etc.

* maxima (bugs, bugs, bugs)

* broken optional packages, e.g. p_group_cohomology

Each of those components could definitely use attention. However, the skill set required to work on those components is quite different from that on working on (re)packaging existing, maintained python projects. People choose what they work on. I think we have a problem if we don't have anyone willing/able to work on pynac or singular or maxima. But I'm not sure this has very much to do with people working on (re)packaging other software.

The discussion about whether a sage-the-distribution should exist and how it relates to sage-the-library definitely needs to be resolved at some point and, whatever decision is made, people need to accept that's the consensus and move on until the next time it needs to be reconsidered, but I don't think that the resolution of that issue will alleviate the lack of maintainers of pynac etc.

[Matthias Koeppe]

On Monday, February 12, 2024 at 4:58:05 PM UTC-8 Nils Bruin wrote:

Each of those components could definitely use attention. However, the skill set required to work on those components is quite different from that on working on (re)packaging existing, maintained python projects. People choose what they work on. I think we have a problem if we don't have anyone willing/able to work on pynac or singular or maxima. But I'm not sure this has very much to do with people working on (re)packaging other software.

Exactly!

[Martin R]

For me - personally - the problem that Vincent pointed out has an emotional flavour: most of the threads on sage-devel are on (lcertainly important) technicalities.  The discussion about how to implement math has mostly moved to github, and split into very very many issues that are hard to find and hard to follow.

The efforts to sort issues by topic are therefore very important to me.  It is a bit sad that search on github is even worse than on google groups - there is no way to seach for partial words.

Martin

[Kwankyu Lee]

On Monday, February 12, 2024 at 3:42:55 PM UTC+9 Matthias Koeppe wrote:

On Thursday, February 8, 2024 at 10:16:58 AM UTC-8 Matthias Koeppe wrote:

2. Is our community aware of the sagemath/sage GitHub wiki? https://github.com/sagemath/sage/wiki

- Are the contents of the wiki front page useful?

Meanwhile I have edited it a bit to offer "suggested activities".

(This is based on a version of the Trac wiki front page just before the transition to GitHub.

The content of the added section does not give new info and seems just visual noise to me...

 

[Dima Pasechnik]

On 13 February 2024 00:58:04 GMT, Nils Bruin <nbr...@sfu.ca> wrote:
>On Monday 12 February 2024 at 15:58:11 UTC-8 Dima Pasechnik wrote:
>
>What's rotten and decaying - well, the most obvious points are:
>
>* pynac (memory leaks, bugs, sketchy or no docs, authors left long time
>ago)
>
>* commutative algebra, in particular Singular-based (memory leaks,
>bugs, no docs, authors either left or are not willing to look into it
>much), etc.
>
>* maxima (bugs, bugs, bugs)
>
>* broken optional packages, e.g. p_group_cohomology
>
>
>Each of those components could definitely use attention. However, the skill
>set required to work on those components is quite different from that on
>working on (re)packaging existing, maintained python projects. People
>choose what they work on.

This is not quite correct. People have a sense of duty. If e.g. Sage docs don't build, someone has to fix this? It's a higher priority task than fixing a particular maths-related bug.

Certainly some people are happy to ignore pleas for help with re-packaging and maintaining packages, and use their maths skillsets, but it's hard for many to do so.
One does not need a maths degree to cut and paste lists of files, or write autoconf macros, yet that's what I was doing for quite a part of my contributions. Out of sense of duty, cause it was something needed, or it looked like something needed.
And it causes quite a riot now when I dare to question the need for a large part of these packaging business...

> I think we have a problem if we don't have anyone
>willing/able to work on pynac or singular or maxima. But I'm not sure this
>has very much to do with people working on (re)packaging other software.

One can alleviate parts of these problems by connecting other potential backends to sage, e.g. symengine to become a replacement for pynac.

>
>The discussion about whether a sage-the-distribution should exist and how
>it relates to sage-the-library definitely needs to be resolved at some
>point and, whatever decision is made, people need to accept that's the
>consensus and move on until the next time it needs to be reconsidered, but
>I don't think that the resolution of that issue will alleviate the lack of
>maintainers of pynac etc.

Pynac is beyond salvation.
We need a collective momentum and a sense of urgency to fix it in a good way. And the same for the rest of the issues I listed.
But right now we have 300 irrelevant to maths standard packages with versions controlled in 5 different ways which sits in our way and hard to ignore.


>

[Martin R]

The "math" activity might be useful, but the "documentation" label is currently also used for issues that have nothing to do with math.  So either relabel those or exclude "documentation".

[Matthias Koeppe]

Thanks all for suggestions and edits regarding the "suggested activities" section! 

I think it is now in a nice balance of being actively inviting/guiding and not too verbose.

[Matthias Koeppe]

On Thursday, February 8, 2024 at 10:16:58 AM UTC-8 Matthias Koeppe wrote:

On Wednesday, February 7, 2024 at 1:11:14 PM UTC-8 Matthias Koeppe wrote:

Let's also use this anniversary as an opportunity to discuss what still needs improving in our development workflows. 

1. We have a low development velocity. For example, some simple PRs sit for weeks or months before receiving any review comments. What can we do to improve this?

2. Is our community aware of the sagemath/sage GitHub wiki? https://github.com/sagemath/sage/wiki

- Are the contents of the wiki front page useful?

- Are the links to Issue and Pull Request queries helpful? 

3. Are the labels on GitHub Issues / PRs helpful?

- Note that new contributors who are not in the Triage team cannot set/change labels! 

- This includes component labels, but also status labels such as "needs review".

- Is it time for the next step with syncing status labels (https://github.com/sagemath/sage/issues/35927)?

- Wishlist item: Component auto-labeler for GitHub PRs (https://github.com/sagemath/sage/issues/37373)

- Wishlist item: PR size labeler (https://github.com/sagemath/sage/pull/37262)

[seb....@gmail.com]

 > Is it time for the next step with syncing status labels (https://github.com/sagemath/sage/issues/35927)?

The reason this is blocked is because there is a bug in the GitHub web interface that might cause confusion when labels are added or removed by the bot. More precisely, the panel with labels is not updated immediately after such an action. I have opened two bug reports (2448092, 2573072). Both were closed without a satisfactory answer, only informing that it is a known bug that will be fixed one day.

[Dima Pasechnik]

On Mon, Feb 19, 2024 at 9:20 PM seb....@gmail.com <seb....@gmail.com> wrote:

 > Is it time for the next step with syncing status labels (https://github.com/sagemath/sage/issues/35927)?

The reason this is blocked is because there is a bug in the GitHub web interface that might cause confusion when labels are added or removed by the bot. More precisely, the panel with labels is not updated immediately after such an action. I have opened two bug reports (2448092, 2573072). Both were closed without a satisfactory answer, only informing that it is a known bug that will be fixed one day.

As far as I know you have an API to manipulate github labels, e.g. it's supported by gh.

Is this what's used by the bot?

(by the way, your bug reports are not visible - probably only visible to you.

Matthias Koeppe schrieb am Freitag, 16. Februar 2024 um 19:00:52 UTC+1:

On Thursday, February 8, 2024 at 10:16:58 AM UTC-8 Matthias Koeppe wrote:

On Wednesday, February 7, 2024 at 1:11:14 PM UTC-8 Matthias Koeppe wrote:

Let's also use this anniversary as an opportunity to discuss what still needs improving in our development workflows. 

1. We have a low development velocity. For example, some simple PRs sit for weeks or months before receiving any review comments. What can we do to improve this?

2. Is our community aware of the sagemath/sage GitHub wiki? https://github.com/sagemath/sage/wiki

- Are the contents of the wiki front page useful?

- Are the links to Issue and Pull Request queries helpful? 

3. Are the labels on GitHub Issues / PRs helpful?

- Note that new contributors who are not in the Triage team cannot set/change labels! 

- This includes component labels, but also status labels such as "needs review".

- Is it time for the next step with syncing status labels (https://github.com/sagemath/sage/issues/35927)?

- Wishlist item: Component auto-labeler for GitHub PRs (https://github.com/sagemath/sage/issues/37373)

- Wishlist item: PR size labeler (https://github.com/sagemath/sage/pull/37262)

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/6745500d-2077-42b7-9406-d38c3e280407n%40googlegroups.com.

[Kwankyu Lee]

3. Are the labels on GitHub Issues / PRs helpful?

- Note that new contributors who are not in the Triage team cannot set/change labels! 

- This includes component labels, but also status labels such as "needs review".

We may drop "needs review" label, and start to use github "draft" functionality. This will remove one major obstacle for new contributors.

 

[Matthias Koeppe]

I would support this.

Can the status label sync workflow help with this transition, without getting in the way? For example, when the _author_ removes the "needs review" label (without setting "positive review"), set the PR to "draft"? 

[seb....@gmail.com]

The bot uses gh edit to manipulate the labels (see the code here). AFAIR I also tried with gh api leading to the same behavior in the web interface. In principle these commands work, but the result is not visible until you add or edit a comment or refresh the web-page.

> (by the way, your bug reports are not visible - probably only visible to you.

Thanks for the hint. I've uploaded screen-shots in the header of a corresponding PR 36292. Here some citations of the respond:

December 4th:

The good news is, our team is already working on a suite of improvements to the issues and pull requests experience and this will be resolved as result of those changes.
Unfortunately I can't share too much about the details as it's still being tested internally, but you can keep an eye on our changelog for any updates.
Thanks for your feedback and I hope this quirk doesn't bother your team too much in the meantime.

February 8th (from the second report where I asked for a status report):

Thanks for following up. Our engineering team are working on a complete overhaul of the issues experience - I've confirmed that this bug is already resolved in the new version.

I can understand it's frustrating seeing this bug persist, but given its upcoming replacement, bug fixes for the current issues experience are limited to critical issues that have a significant impact. The overhaul is in progress and unfortunately hasn't been publicly announced which limits how much detail I can share. I'd suggest keeping an eye on our roadmap and changelog for updates.

I'm sorry for any inconvenience, and please let me know if you have any questions.

[seb....@gmail.com]

This is currently not implemented, but of course possible. But the converse can be activated immediately: If a user converts a ready PR to a draft, all status labels will be removed. If a draft is marked as ready for review the s: needs review label is added. Also implemented (but not activated): If the s: needs review label is added a draft PR, it is marked as ready for review.

[Matthias Koeppe]

On Tuesday, February 20, 2024 at 10:11:14 AM UTC-8 seb....@gmail.com wrote:

can be activated immediately: If a user converts a ready PR to a draft, all status labels will be removed. If a draft is marked as ready for review the s: needs review label is added. Also implemented (but not activated): If the s: needs review label is added a draft PR, it is marked as ready for review.

Sounds good to me, let's do this.

 

[seb....@gmail.com]

It must be done by a maintainer of the repository (see the desription of SYNC_LABELS_IGNORE_EVENTS in  #35172). However, the restriction regarding the error in the web interface must then be accepted.

[Kwankyu Lee]

On Wednesday, February 21, 2024 at 3:11:14 AM UTC+9 seb....@gmail.com wrote:

This is currently not implemented, but of course possible. But the converse can be activated immediately:

 

If a user converts a ready PR to a draft, all status labels will be removed.

Do not activate because of the github bug

 

If a draft is marked as ready for review the s: needs review label is added.

Activate immediately. 

Also implemented (but not activated): If the s: needs review label is added a draft PR, it is marked as ready for review.

Activate immediately.

Is this selective activation possible?  

If the syncing script should be modified to implement this, please open a PR.

 

[Kwankyu Lee]

If a draft is marked as ready for review the s: needs review label is added.

Activate immediately. 

Sorry. This one is also affected by the github bug. Right? 

If so, not activate. 

[seb....@gmail.com]

> Is this selective activation possible?

No! Activating the labelled trigger effects all state and priority labels. Even if restriced to the label s: needs review it could be affected by the GitHub bug, since the bot will remove a previously selected state label.

> Sorry. This one is also affected by the github bug. Right?

Yes!

[Kwankyu Lee]

On Tuesday, February 20, 2024 at 12:26:50 PM UTC+9 Matthias Koeppe wrote:

Can the status label sync workflow help with this transition, without getting in the way? For example, when the _author_ removes the "needs review" label (without setting "positive review"), set the PR to "draft"? 

sync workflow seems not yet ready for the transition because of the bug.

By the way, an author of a PR needs also the ability to remove "needs work". Hence the author needs to be in the Triage team anyway in our workflow.

 

[Sebastian Oehms]

> By the way, an author of a PR needs also the ability to remove "needs work". Hence the author needs to be in the Triage team anyway in our workflow.

Not necessarily! If the synchronizing trigger is enabled then the bot would change needs work to  needs review on a non draft PR each time a new commit is pushed to the branch.

--
You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/sulCa-6EZRA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/268e5d1c-5ff4-4d88-8ea4-7aaef77973cfn%40googlegroups.com.