DISCUSS: move Sage development to Github

[David Roe]

Dear Sage developers,

As announced in a parallel thread, we are voting to move Sage development from Trac to Github.  Several of us have created a wiki page attempting to summarize arguments in favor of each system, and this thread can serve as a space for people to make clear their own reasoning for favoring one option over the other.  This discussion has gotten heated at times, so remember to be considerate, respectful and polite: we are all aiming to make Sage better.

David

[Dima Pasechnik]

Do we require everyone willing to vote to do so on sage-devel, or it
could be done elsewhere (not every contributor to Sage or its
dependencies/packages is there) ?

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAChs6_mZzK6v7wWP2vS5JzHYVNd%3DPoaCDDTdQF-u8VTgE-ONcg%40mail.gmail.com.

[David Roe]

Sage has a tradition of public voting, but I'm fine if people want to email me separately with their vote and I can forward it along.

David

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq0bagggnCks_SH6x-TFrkvK9Wm2Ypn9_eyqRTRmaYD3mg%40mail.gmail.com.

[Clemens Heuberger]

I did not chime in in the long thread leading to the vote, but I am quite used
to working with Gitlab (hosted at my university) and would be more comfortable
with a Gitlab solution because I have the impression that it gives us more
freedom (we currently run our own patchbots, so running our own Gitlab runners
should not be that difficult to achieve).

While working with trac seemed to be fine when I first contributed to SageMath
in 2014, it now feels completely anachronistic to me; reviewing code there
without any real support just feels painful. Adding the maintenance issues, I am
happy to vote for leaving trac rather sooner than later. Given the current vote,
this means that I'll vote for Github in the other thread.

Regards,

Clemens

Am 21.09.22 um 19:23 schrieb David Roe:

> Dear Sage developers,
> As announced in a parallel thread, we are voting to move Sage development from
> Trac to Github.  Several of us have created a wiki page

> <https://github.com/sagemath/sage/wiki/Github-vs-Gitlab-vs-trac> attempting to

> summarize arguments in favor of each system, and this thread can serve as a
> space for people to make clear their own reasoning for favoring one option over
> the other.  This discussion has gotten heated at times, so remember to be
> considerate, respectful and polite

> <https://github.com/sagemath/sage/blob/develop/CODE_OF_CONDUCT.md>: we are all

[Kwankyu Lee]

> personally I still prefer Trac, but the bus factor argument and recruitment of new contributors are more important

+1

[kcrisman]

Not sure where to ask this - here?  On the GH page documenting the transition and new workflow proposal, I don't see a way to have multiple AUTHORs in the way we usually kept track of it.  Note that often there were people who were authors who didn't show up on a specific commit, but which the discussion on a ticket made clear there was a consensus they should be.  (For instance, one might propose some code on sage-devel or sage-support, which then someone else puts on a branch.)  Do we have a proposal for a mechanism to keep track of this beyond PR-associated emails?  I realize that it's possible to put a different author than committer, but I'm not sure how that would work with multiple of both.  

If not, hat would not be an insurmountable difference for contributors, but would be a change from our historic practice which assigned credit "liberally", as it were.  For context, I thought of this as a potential contributor was asking about credit issues (not this precise issue, more generally) on a ticket.

[David Roe]

This is Github's documentation for doing it, but it's pretty annoying:

https://docs.github.com/en/pull-requests/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors

Maybe we could create some automation to add these kinds of comments to the merge commit created when a PR is merged, or as a comment on the PR when it's closed....

David

On Fri, Sep 23, 2022 at 10:51 AM kcrisman <kcri...@gmail.com> wrote:

Not sure where to ask this - here?  On the GH page documenting the transition and new workflow proposal, I don't see a way to have multiple AUTHORs in the way we usually kept track of it.  Note that often there were people who were authors who didn't show up on a specific commit, but which the discussion on a ticket made clear there was a consensus they should be.  (For instance, one might propose some code on sage-devel or sage-support, which then someone else puts on a branch.)  Do we have a proposal for a mechanism to keep track of this beyond PR-associated emails?  I realize that it's possible to put a different author than committer, but I'm not sure how that would work with multiple of both.  

If not, hat would not be an insurmountable difference for contributors, but would be a change from our historic practice which assigned credit "liberally", as it were.  For context, I thought of this as a potential contributor was asking about credit issues (not this precise issue, more generally) on a ticket.

--

You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/87584831-2a22-4f43-be96-046840ac480an%40googlegroups.com.

[Matthias Koeppe]

On Friday, September 23, 2022 at 7:51:24 AM UTC-7 kcrisman wrote:

On the GH page documenting the transition and new workflow proposal, I don't see a way to have multiple AUTHORs in the way we usually kept track of it.

I agree, this still needs to be specified. (Related: https://groups.google.com/g/sage-devel/c/ayOL8_bzOfk/m/pPqbu13fAAAJ)

 

[Matthias Koeppe]

I think part of a solution could be PR templates, which add structure to the PR description (= the first comment). That could be a way of adding Authors (and Reviewers) to a PR.

https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/about-issue-and-pull-request-templates

https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/creating-a-pull-request-template-for-your-repository

[kcrisman]

I think part of a solution could be PR templates, which add structure to the PR description (= the first comment). That could be a way of adding Authors (and Reviewers) to a PR.

If there's a way to (lightly) enforce that via some kind of bot, that sounds very reasonable.

On another note, I realize that the comment I made 6 years ago after Volker's comment is still relevant:  

"There's also the non-trivial (though not blocker, probably) issue that zillions of links to trac.sagemath.org would instantly be obsolete"

How long do we want to have Trac still exist, but be read-only?  Obviously we wouldn't take it down right away, but presumably eventually we would need to do so.  

[Matthias Koeppe]

On Saturday, September 24, 2022 at 4:57:36 AM UTC-7 kcrisman wrote:

On another note, I realize that the comment I made 6 years ago after Volker's comment is still relevant:  

"There's also the non-trivial (though not blocker, probably) issue that zillions of links to trac.sagemath.org would instantly be obsolete"

How long do we want to have Trac still exist, but be read-only?  Obviously we wouldn't take it down right away, but presumably eventually we would need to do so.  

The solutions for this are discussed in the ticket description of https://trac.sagemath.org/ticket/30363 - we can keep the external links to trac.sagemath.org working by means of redirects to the equivalent github issues.

 

[William Stein]

More precisely, we could download a complete static copy of all of
trac to static html (e.g., using recursive wget), then publish that
static html using github pages, and modify our DNS to point there.
The result would be free, fast, robust, and can be there indefinitely,
and would provide a read-only copy of all the pages on trac. We
could also automatically edit each static page to have a clear banner
stating that it is an archive, and include a link to the corresponding
GitHub issue.

I'm not volunteering to do that. I'm just saying it is technically
possible, and probably not that hard.

William

>
>
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/37e506cc-efc8-49d8-a5e3-2bb3e858d8f5n%40googlegroups.com.



--
William (http://wstein.org)

[TB]

Is it possible to choose the issue numbers in GH when making a migration? Then, setting a redirect of the form "https://trac.sagemath.org/ticket/$TICKET_NUMBER -> https://github.com/sagemath/sage/issues/$TICKET_NUMBER" will make the lion's share of the links still relevant. This does not preserve fragments like "#comment:7", which is useful in long ticket discussions.

Regards,

TB

[Matthias Koeppe]

On Saturday, September 24, 2022 at 9:27:46 AM UTC-7 mathzeta2 wrote:

Is it possible to choose the issue numbers in GH when making a migration? Then, setting a redirect of the form "https://trac.sagemath.org/ticket/$TICKET_NUMBER -> https://github.com/sagemath/sage/issues/$TICKET_NUMBER" will make the lion's share of the links still relevant.

Yes, to map it like this is the plan.

 

This does not preserve fragments like "#comment:7", which is useful in long ticket discussions.

Thanks, I've opened https://github.com/sagemath/trac-to-github/issues/7 for this.

[John H Palmieri]

I think I'm missing part of this. What is the actual path to switching to GitHub? I've seen pages describing how individual development tasks will be converted from trac to GitHub, but what does the overall transition look like?

- Do we just say, before November 1 (or whenever) we're doing everything on trac, and after we're doing everything on GitHub?

- Or is there a period where both are active and people can slowly transition? We have a GitHub page now; if the transition is approved, do people start creating issues and pull requests right away?

- Or some other option?

I apologize if this has been discussed and I missed it.

[Matthias Koeppe]

On Saturday, September 24, 2022 at 5:15:11 PM UTC-7 John H Palmieri wrote:

I think I'm missing part of this. What is the actual path to switching to GitHub? I've seen pages describing how individual development tasks will be converted from trac to GitHub, but what does the overall transition look like?

- Do we just say, before November 1 (or whenever) we're doing everything on trac, and after we're doing everything on GitHub?

Yes, exactly (although we have not discussed the date yet).

On the switchover day, it would look like this: 

1. We take Trac offline, reconfigure it to be read-only, bring it online.

2. Convert all tickets to Issues in a new repo. (This preserves the ticket numbers as Issue numbers.)

3. Final check that the new repo is OK.

4. Replace sagemath/sage by the new repo.

5. Announce that sagemath/sage is now open for Issues and PRs.

[Matthias Koeppe]

I've added this to https://github.com/sagemath/sage/wiki/migration-from-trac-to-Git**b#conversion-of-trac-tickets-and-the-trac-wiki-to-github now

[John H Palmieri]

Will the changeover also mark Sage 10.0? Is there a vision for what Sage 10.0 means?

[Matthias Koeppe]

I don't think it's useful to reflect this change in a version number. I think it can just happen in the middle of the 9.8 release cycle.

[Kwankyu Lee]

On Monday, September 26, 2022 at 12:21:41 PM UTC+9 John H Palmieri wrote:

Is there a vision for what Sage 10.0 means?

Sage X :) 

[Kwankyu Lee]

On Monday, September 26, 2022 at 3:17:03 PM UTC+9 Kwankyu Lee wrote:

On Monday, September 26, 2022 at 12:21:41 PM UTC+9 John H Palmieri wrote:

Is there a vision for what Sage 10.0 means?

 Yes. See the headline of 

https://trac.sagemath.org/ticket/29705

[Eric Gourgoulhon]

Le samedi 24 septembre 2022 à 13:57:36 UTC+2, kcrisman a écrit :

On another note, I realize that the comment I made 6 years ago after Volker's comment is still relevant:  

"There's also the non-trivial (though not blocker, probably) issue that zillions of links to trac.sagemath.org would instantly be obsolete"

This is a good point IMHO. In particular, there are zillions of links to Trac from the sage-devel threads.

 

How long do we want to have Trac still exist, but be read-only?  Obviously we wouldn't take it down right away, but presumably eventually we would need to do so.  

As I said earlier, discussions on Trac constitute a very valuable database for Sage development. As far as I am concerned, I learned a lot from them. It would be a pity to lose them. 

Eric.

[Tobias Diez]

2. Convert all tickets to Issues in a new repo. (This preserves the ticket numbers as Issue numbers.)

Would it make sense to convert tickets with branches directly to pull-requests? Since most of them probably already contain quite a bit of discussion about the implementation, which you would like to have as a easy reference if you would review the PR later.

 

4. Replace sagemath/sage by the new repo.

 If you rename a repo (in our case sage -> sage-old), then github adds redirects for issues etc in sage to sage_old. So it should be double-checked that you can later indeed add a new repo with the name sage. 

[Dima Pasechnik]

On Mon, Sep 26, 2022 at 10:26 AM Tobias Diez <tobias...@gmail.com> wrote:
>
>
>
>> 2. Convert all tickets to Issues in a new repo. (This preserves the ticket numbers as Issue numbers.)
>
>
> Would it make sense to convert tickets with branches directly to pull-requests?

only open tickets.
Closed tickets are served well enough by changing links to branches on
sagetrac-mirror.

> Since most of them probably already contain quite a bit of discussion about the implementation, which you would like to have as a easy reference if you would review the PR later.
>
>>
>> 4. Replace sagemath/sage by the new repo.
>
>
> If you rename a repo (in our case sage -> sage-old), then github adds redirects for issues etc in sage to sage_old. So it should be double-checked that you can later indeed add a new repo with the name sage.
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/6418ac37-a858-447a-be9f-f7d7f4da9461n%40googlegroups.com.

[Matthias Koeppe]

On Monday, September 26, 2022 at 1:35:04 AM UTC-7 Eric Gourgoulhon wrote:

 

How long do we want to have Trac still exist, but be read-only?  Obviously we wouldn't take it down right away, but presumably eventually we would need to do so.  

As I said earlier, discussions on Trac constitute a very valuable database for Sage development. As far as I am concerned, I learned a lot from them. It would be a pity to lose them.

Yes, of course, that's why the plan is to convert all tickets to Issues! That includes all closed tickets and all comments on the tickets.

Preview of converted tickets: https://github.com/sagemath/trac_to_gh/issues

 

[seb....@gmail.com]

Don’t we need an issue for the first point, as well? The example #26 corresponds to #34110 which is not easy to recover from the migrated information.

Furthermore, it isn’t still clear to me how dependencies between PRs will be visible (like in the Trac dependencies field). In the above example you have to recover this from the history of commit messages (which may not be clear enough in general). Shouldn’t the migration put something into the header fields milestone, assignees, …, as well (if possible)? How will authors and reviewers be visible?

​

[Tobias Diez]

One more question: The current plan is to use the sagetrac-mirror repo as the base for creating PRs but also to archived it. However, if I'm not mistaken, that makes all branches in sagetrac-mirror readonly and thus one cannot continue working on existing PRs by pushing to the corresponding branch in sagetrac-mirror.

[Dima Pasechnik]

On Tue, Sep 27, 2022 at 11:29 AM Tobias Diez <tobias...@gmail.com> wrote:
>
> One more question: The current plan is to use the sagetrac-mirror repo as the base for creating PRs but also to archived it. However, if I'm not mistaken, that makes all branches in sagetrac-mirror readonly and thus one cannot continue working on existing PRs by pushing to the corresponding branch in sagetrac-mirror.

IMHO the plan is to create new PRs in sagemath/sage, not in
sagemath/sagetrac-mirror
There won't be "existing" PRs, only issues, pointing to branches on
sagetrac-mirror

> On Tuesday, 27 September 2022 at 10:02:06 UTC+2 seb....@gmail.com wrote:
>>
>> Matthias Koeppe schrieb am Samstag, 24. September 2022 um 19:09:46 UTC+2:
>>>
>>> On Saturday, September 24, 2022 at 9:27:46 AM UTC-7 mathzeta2 wrote:
>>>>
>>>> Is it possible to choose the issue numbers in GH when making a migration? Then, setting a redirect of the form "https://trac.sagemath.org/ticket/$TICKET_NUMBER -> https://github.com/sagemath/sage/issues/$TICKET_NUMBER" will make the lion's share of the links still relevant.
>>>
>>>
>>> Yes, to map it like this is the plan.
>>>
>>>>
>>>> This does not preserve fragments like "#comment:7", which is useful in long ticket discussions.
>>>
>>>
>>> Thanks, I've opened https://github.com/sagemath/trac-to-github/issues/7 for this.
>>
>> Don’t we need an issue for the first point, as well? The example #26 corresponds to #34110 which is not easy to recover from the migrated information.
>>
>> Furthermore, it isn’t still clear to me how dependencies between PRs will be visible (like in the Trac dependencies field). In the above example you have to recover this from the history of commit messages (which may not be clear enough in general). Shouldn’t the migration put something into the header fields milestone, assignees, …, as well (if possible)? How will authors and reviewers be visible?
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/d815783e-fd5c-4aa3-ab27-7024b18b299dn%40googlegroups.com.

[Tobias Diez]

Yes, the target repo of these PRs will be the (new) sagemath/sage, but the source will be sagemath/sagetrac-mirror, right? So in order to update the pull request one needs to push the changes to sagemath/sagetrac-mirror (it is not possible to update a PR by pushing to /refs/pull/xyz, because this is readonly). Thus, if sagetrac-mirror is archived (and thus readonly), the only way to work on existing tickets/branches would be to checkout the existing branch (from either sagetrac-mirror or sage/refs/pull), make changes, push to a new fork, create a new PR, close the old PR (essentially the workflow https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally#modifying-an-inactive-pull-request-locally).

[Dima Pasechnik]

On Tue, 27 Sep 2022, 14:08 Tobias Diez, <tobias...@gmail.com> wrote:

Yes, the target repo of these PRs will be the (new) sagemath/sage, but the source will be sagemath/sagetrac-mirror, right?

Hmm, I might have missed something - what is the need to have 2 repos here, if 1 is sufficient?

Any fork of sagemath/sage may be a source of a PR, not only sagetrac-mirror

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/6df40198-0d1a-45f4-ac1f-2bee6e07d313n%40googlegroups.com.

[Matthias Koeppe]

On Tuesday, September 27, 2022 at 1:02:06 AM UTC-7 seb....@gmail.com wrote:

Furthermore, it isn’t still clear to me how dependencies between PRs will be visible (like in the Trac dependencies field).

This is an important point. See https://trac.sagemath.org/ticket/30363#comment:91

 

​

[Matthias Koeppe]

On Tuesday, September 27, 2022 at 3:29:10 AM UTC-7 tobias...@gmail.com wrote:

One more question: The current plan is to use the sagetrac-mirror repo as the base for creating PRs but also to archived it. However, if I'm not mistaken, that makes all branches in sagetrac-mirror readonly and thus one cannot continue working on existing PRs by pushing to the corresponding branch in sagetrac-mirror.

Thanks for catching this. I have revised https://github.com/sagemath/sage/wiki/migration-from-trac-to-Git**b#conversion-of-trac-tickets-and-the-trac-wiki-to-github as follows:

1. Make a fork of sagemath/sagetrac-mirror called sagemath/sagetrac-archive and archive it (= set it to readonly).
2. Open PRs from sagemath/sagetrac-mirror to the new repo for all open tickets with attached branches.

 

[Matthias Koeppe]

I've created https://github.com/sagemath/sage-gh-templates-sandbox for playing with Issue and PR templates.

[Matthias Koeppe]

The first proposed Issue template can be seen here: https://github.com/sagemath/sage-gh-templates-sandbox/issues/new/choose

See https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/configuring-issue-templates-for-your-repository 

[Tobias Diez]

Just to make sure we are talking about the same thing. Imagine a currently open ticket with a linked branch. How is this going to be migrated? My assumption has been that this will create a PR from sagemath/sagetrac-mirror/branch into sagemath/sage.

If thats indeed the plan (which I find is a good plan), then there are the following issues:

- sagetrac-mirror is not a fork of sage, thus it might not be possible to create a PR from it (at leas from the web interface its not possible, not sure about the API)

- sagetrac-mirror cannot be archived otherwise it will be readonly (this is taken care of my Matthias recent edit to the migration wiki page)

- devs might not have the permission to push to sagetrac-mirror (currently there is a branch protection rule in place that prevents any direct commits, but even if that's removed I'm not sure if everyone can just push to it)

[Dima Pasechnik]

On Tue, 27 Sep 2022, 21:12 Tobias Diez, <tobias...@gmail.com> wrote:

Just to make sure we are talking about the same thing. Imagine a currently open ticket with a linked branch. How is this going to be migrated? My assumption has been that this will create a PR from sagemath/sagetrac-mirror/branch into sagemath/sage.

No, there will be an issue on sagemath/sage, no PR. There will be a link to a branch on sagetrac-mirror (which will be readonly). 

To proceed, just push this branch to your personal fork of sagemath/sage and make a PR from there.

At this point it becomes a usual github workflow.

If thats indeed the plan (which I find is a good plan), then there are the following issues:

- sagetrac-mirror is not a fork of sage, thus it might not be possible to create a PR from it (at leas from the web interface its not possible, not sure about the API)

- sagetrac-mirror cannot be archived otherwise it will be readonly (this is taken care of my Matthias recent edit to the migration wiki page)

- devs might not have the permission to push to sagetrac-mirror (currently there is a branch protection rule in place that prevents any direct commits, but even if that's removed I'm not sure if everyone can just push to it)

all this is avoided if done as I described above 

Dima

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/cb7d6705-09e1-41ee-9f9a-1543c4b097a9n%40googlegroups.com.

[Matthias Koeppe]

On Tuesday, September 27, 2022 at 1:12:55 PM UTC-7 tobias...@gmail.com wrote:

Just to make sure we are talking about the same thing. Imagine a currently open ticket with a linked branch. How is this going to be migrated? My assumption has been that this will create a PR from sagemath/sagetrac-mirror/branch into sagemath/sage.

If thats indeed the plan (which I find is a good plan), then there are the following issues:

- sagetrac-mirror is not a fork of sage, thus it might not be possible to create a PR from it (at leas from the web interface its not possible, not sure about the API)

Good point about the "is-fork-of" relation. I've made another refinement in  https://github.com/sagemath/sage/wiki/migration-from-trac-to-Git**b#conversion-of-trac-tickets-and-the-trac-wiki-to-github :

1. Convert all tickets to Issues in a new repo sagemath/sage-temp. (This preserves the ticket numbers as Issue numbers.)
2. Rename sagemath/sagetrac-mirror to sagemath/sagetrac-archive and archive it (= set it to readonly).
3. Create a single-branch fork of sagemath/sage-temp called sagemath/sagetrac-mirror and push all branches (or all branches of open tickets) from sagemath/sagetrac-archive to it.
4. Open PRs from sagemath/sagetrac-mirror to sagemath/sage-temp for all open tickets with attached branches.

[Tobias Diez]

Okay, fair enough! Then it's a bit more work to get tickets into PRs (for devs) but maybe its a good idea to start with a clean slate.

[Nils Bruin]

A fair point made: an "exit strategy" from Github should exist and should ideally take into account that this exit may need to happen at a time where github is no longer able/willing to cooperate in this exit: in other words, we should ideally *back up* our issues and pull-request histories. The APIs are there; writing the scripts to pull this stuff (incrementally?) would be quite a bit of work, but then running it shouldn't be so bad.

This is just common sense data security policy: to us github is a single point-of-failure. You want to store with some frequency snapshots of the data there.

[John H Palmieri]

You would think that this would be a solved problem: others in the open source community must have be in the practice of backing up their GitHub info.

[Matthias Koeppe]

I would say that in general, projects are not concerned that the https://en.wikipedia.org/wiki/BitKeeper situation with the Linux kernel from 20 years ago would be repeated by Microsoft/GitHub. 

[Marc Mezzarobba]

John H Palmieri wrote:
> You would think that this would be a solved problem: others in the
> open source community must have be in the practice of backing up their
> GitHub info.

The following tools seem fairly complete:

- https://github-backup.branchable.com/ (but I'm getting timeouts with
it),

- https://github.com/josegonzalez/python-github-backup (not tested).

IMO we should at the very least have something like that running before
making the switch. We should also refrain from using features of github
not supported by our backup tool.

--
Marc

[Dima Pasechnik]

While migrating to github, we can get json records for each issue we
created to replace tickets.
(they are complete records, everything may be recreated from them)

Then we can set up a GitHub actions to produce such jsons for
created/updated issues and PRs,
see e.g. https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#issue_comment

I think this is more robust than running tools which make global copies.
(These updated jsons need to be be stored somewhere)

Dima



>
> --
> Marc

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/th67k9%246eq%241%40ciao.gmane.io.

[Tobias Diez]

Github as their own open archive program (https://archiveprogram.github.com/) and works together with known software / archive websites. In particular, the code itself is archived via the EU-funded Software Heritage foundation and issue/PR metadata for all public repos are in GHTorrent / GHArchive. Thus, in the unlikely situation of massive policy changes of GH including a complete removal/block of their API, the data is still there and the community will probably develop migration tools to whatever platform will become popular then.

In addition, one can download a "migration archive" from github which should include all data (https://docs.github.com/en/enterprise-server@3.6/admin/user-management/migrating-data-to-and-from-your-enterprise/exporting-migration-data-from-githubcom#generating-a-migration-archive) or use one of the third-party solutions that keep backups per repo (https://github.com/marketplace?category=backup-utilities&query=sort%3Apopularity-desc&type=&verification=).

[Matthias Koeppe]

Thanks for the pointers! 

I see that the Cython project uses python-github-backup to back up their repo to https://github.com/cython/cython-issues

I would propose that we set up such backups for the projects that are already hosted at https://github.com/sagemath/:

In particular:

- https://github.com/sagemath/cypari2

- https://github.com/sagemath/sagetex

- https://github.com/sagemath/pplpy

- https://github.com/sagemath/cysignals

- https://github.com/sagemath/memory_allocator

[Matthias Koeppe]

I've opened https://trac.sagemath.org/ticket/34624 to organize the discussion on backup and coordinate work on it.

[Nils Bruin]

Speaking of backups ... do we backup the sage-devel, sage-support news groups? It contains a lot of stuff that loses relevance with time, but every now and again there are discussions that contain important bits of information. In fact, they are sometimes referred to on trac, via super-opaque URLs. So I suspect google groups going down (which is probably a less unlikely event than github ending) would actually damage those links irretrievably.

I recognize that we have limited resources for backing up stuff. We should be using our resources and time in a smart way to back up things that will have most effect. But it's probably worth it for someone to spend a bit of to determine how to back up sage-devel (that's easy going forward since it can just send email digests; and someone may have been storing those already) and how to refer to threads -- perhaps a date is better than a URL -- as long as we can have a list archive that allows searching by date (I'm not so sure google's web interface allows it), and then perhaps draw the conclusion that it's not worth the effort to archive?

[kcrisman]

On Sunday, October 2, 2022 at 2:16:00 PM UTC-4 Nils Bruin wrote:

Speaking of backups ... do we backup the sage-devel, sage-support news groups? It contains a lot of stuff that loses relevance with time, but every now and again there are discussions that contain important bits of information. In fact, they are sometimes referred to on trac, via super-opaque URLs. So I suspect google groups going down (which is probably a less unlikely event than github ending) would actually damage those links irretrievably.

I guess they are "backed up" by  other news aggregators?  Sometimes those are the hits that come up first in Google search, ironically.  E.g. https://www.mail-archive.com/sage-...@googlegroups.com/msg105313.html

Which also points out that we already rely on proprietary software for something fairly vital ... not judging that either way, but it's worth mentioning.

[Marc Mezzarobba]

Nils Bruin wrote:
> Speaking of backups ... do we backup the sage-devel, sage-support news
> groups?

They are on gmane, and presumably some sage developers have more or less
complete archives on their own computers...

> In fact, they are sometimes referred to on trac, via
> super-opaque URLs.

I suppose the "right" way to refer to ML posts would be by Message-Id,
but nobody does that, and many people apparently do not understand such
references or have no convenient tools to follow them... (I once had
someone complain when I used a Message-Id to refer to sage-devel on
trac.)

--
Marc