Forked repositories in GitHub

[Daniel Beck]

Hi everyone,

I'd like to propose a cleanup of 'fork' relationships of the repositories in the jenkinsci GitHub organization.

Background:
For many years, the plugin hosting process has forked existing repositories. The expectation was always that the new repo in jenkinsci was the canonical 'main' repository, but that wasn't enforced. For the past year or two, we've even asked maintainers to delete their repository after forking unless there were useful PRs and issues in there already, so that the jenkinsci repo became the 'main' repo (with occasional mishaps if someone else had forked before us).

Some people enjoy the "branding" effect that having the source repository creates. But this comes with downsides: Sometimes GitHub code search doesn't work, depending on the popularity of the repository. Links to create pull requests sometimes don't work quite right, and INFRA-2697 notes that the GitHub CLI cannot really handle networks where a fork is the "main" repo, probably for the same reason. Having a different repo than what we consider canonical as the "root" repository confuses users trying to file pull requests or issues on GitHub. It'll get worse once GitHub adds repo-level discussions[1]. Basically, the more stuff is attached to a repository that isn't trivially cloned/mirrored to forks, the worse it gets.

In terms of security, GitHub for quite some time did not support security warnings for forks. LGTM.com / GitHub Security Labs still does not recognize forked repositories. Earlier this year a security researcher recently used its CodeQL functionality to identify and submit fixes to pom.xml files referencing plain HTTP Maven repositories, but couldn't do that for forked repos. In many cases, the source repositories are much less active than the repo in jenkinsci, or the maintainers have moved on entirely, making this feature unavailable to (other) current maintainers, or the Jenkins security team.

The way we create forks is simply not a well-supported use case.

My proposal therefore is to "unfork" plugin and similar repositories in the jenkinsci organization. Only repositories that clearly are forks (e.g. some libraries not maintained by us) would remain forks.

After checking with GitHub support, the following options exist:

1. It is possible to invert the fork relationship. This requires approval from both repo owners (i.e. jenkinsci and whoever we forked from).
2. It is possible to cut the fork relationship. This requires approval from the forked repo owner (i.e. jenkinsci).

And while it is technically possible to re-attach repos to a network / merge networks, GH support would rather not do that.

Therefore I propose we implement the following steps:

1. We try to contact, wherever possible, whoever we forked from, and ask them to contact GitHub support. I'll grant blanket permission on behalf of jenkinsci and will tell everyone the support ticket number to reference so this goes as smoothly as possible.
2. We wait a while while folks ask GH support for an inversion of the fork relationship.
3. We ask GitHub support to cut the fork relationship of everything that's left over.

Additionally, we should change the hosting process to work with repo transfers, or creation of repos without the fork relationship. That can be done at any time though; as even now we don't really want that fork relationship we create to exist.

To understand the scope of this, I've written a script that periodically updates a list of forked repositories in jenkinsci, you can see the result at https://www.jenkins.io/doc/developer/publishing/source-code-hosting/forks/

One potential problem are plugins that are actively maintained outside the jenkinsci organization and only have an outdated fork in jenkinsci that isn't being used. I think it makes sense to ask maintainers to move their activity into jenkinsci (including perhaps a complete repo transfer to retain issues and PRs). If they refuse, rather than cut the fork relationship, we could just delete our unused fork. (While this touches on plugins maintained exclusively outside jenkinsci, I consider that general topic to be a separate conversation. Please keep this thread focused on this proposal.)

Thoughts?

Daniel

1: https://github.blog/2020-05-06-new-from-satellite-2020-github-codespaces-github-discussions-securing-code-in-private-repositories-and-more/#discussions

[Tim Jacomb]

Hi Daniel

Completely agree with this, I've accidentally submitted PRs to the wrong forks many times.

Both because of GitHub CLI and also just the web UI picking the wrong one.

For the record I've had GitHub support break the fork relationship from all the components that I maintain as far as I know.

and it all went fine :)

Thanks

Tim

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/6D96DA83-2AE0-4C87-92D6-4CCC8DFE1E57%40beckweb.net.

[Arnaud Héritier]

+1 for the proposed plan

Something is strange in your export.

For example I am supposed to host https://github.com/aheritier/build-flow-plugin (origin) which should be forked to https://github.com/jenkinsci/jenkinsci/build-flow-plugin ( doesn't exist)

We probably had such repo in the past and it was deleted after I forked it but maybe you could exclude from the list the repos when they aren't existing anymore in the jenkinsci side (not sure how many repos could be like this)

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/6D96DA83-2AE0-4C87-92D6-4CCC8DFE1E57%40beckweb.net.

--

Arnaud Héritier

Twitter/Skype : aheritier

[Daniel Beck]

The repo exists, there's just an additional "jenkinsci/" in the link. I have no idea why the GH API behaves inconsistently there.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFNCU-_vuzGEO_u18SkF43t1vSbZouZm7yq61-m9BCvj3dizMg%40mail.gmail.com.

[Arnaud Héritier]

ok I missed :( 

It doesn't make sense to have my repo as primary. I didn't create it and never committed to it. 

There is probably a bug in GitHub with forks which were created a long time ago

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAMo7PtKTB1QCVTd-c1ABxBi3pf%2Bo8w-ODJu1Poq2vWjKX4Ot8g%40mail.gmail.com.

[Arnaud Héritier]

and I received a PR https://github.com/aheritier/build-flow-plugin/pull/2

😭

+1000 for the proposal

[Mark Waite]

+1 from me.

[Matt Sicker]

+1 here, especially due to GitHub tooling and apps.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/50ad23a4-abe8-4a69-ab09-2419d227e830n%40googlegroups.com.

--

Matt Sicker
Senior Software Engineer, CloudBees

[Ullrich Hafner]

+1 from me as well

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAEot4oz_CHTEH257FqacEOChDxEHTWj0SPOVTbt3%2BKKCSxnj0A%40mail.gmail.com.

[Liam Newman]

+1000

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/48B64CEC-02A5-4797-95A2-6969F5F28C93%40gmail.com.

[Oleg Nenashev]

+1 from me.

Thanks for driving it, Daniel!

On Wednesday, August 19, 2020 at 12:37:15 AM UTC+2, Liam Newman wrote:

+1000

On Tue, Aug 18, 2020 at 3:20 PM Ullrich Hafner <ullric...@gmail.com> wrote:

+1 from me as well

Am 18.08.2020 um 16:30 schrieb Matt Sicker <msi...@cloudbees.com>:

+1 here, especially due to GitHub tooling and apps.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/50ad23a4-abe8-4a69-ab09-2419d227e830n%40googlegroups.com.

--

Matt Sicker
Senior Software Engineer, CloudBees

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAEot4oz_CHTEH257FqacEOChDxEHTWj0SPOVTbt3%2BKKCSxnj0A%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.

[Baptiste Mathus]

+1. All for it. 

[timja...@gmail.com]

Seem's like GitHub won't break the fork relationship anymore without both sides agreeing to it?

They've done it in the past for me but wouldn't do it yesterday.

GH> In the two scenarios you have:

DB>> It is possible to invert the fork relationship. This requires approval from both repo owners (i.e. jenkinsci and whoever we forked from).

GH> This is correct.

DB> It is possible to cut the fork relationship. This requires approval from the forked repo owner (i.e. jenkinsci).

GH>>This would only be the case of the fork network is private. However, the fork network in question is public, so we'll need the repository owner to contact us and approve this request.

GH>>I hope this helps clarify.

Thanks

Tim

[Daniel Beck]

Could you ask why they no longer do that when they did in the past (or at least offered to, see my old support request)?

[Slide]

Transfers are much less automatable. There are APIs, but they have to be initiated by the person who is transferring the repo and then ack'd by the org receiving the transfer. So, the bot would not be able to do everything anymore.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/a2b655fb-f32e-4868-b1b8-ace38c280026n%40googlegroups.com.

[Jesse Glick]

On Wed, Jul 28, 2021 at 2:48 AM timja...@gmail.com <timja...@gmail.com> wrote:

Seem's like GitHub won't break the fork relationship anymore without both sides agreeing to it?

This could be a real problem for us. We still have a bunch of repos with misleading fork status, like https://github.com/jenkinsci/plugin-pom rather prominently. There should be some provision by which a “fork” which has way more activity than the “origin” can unilaterally break the relationship. Of course you can delete and recreate the repo with Git history intact but you will lose all PR history, which is intolerable.

[Tim Jacomb]

Wouldn't creating a repo and pushing the code instead of forking work during the hosting process?

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/SkKoCccPrOc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1XZbUvJER_5-LMFd8i1d1RTaFy6JPLhwq9pG9uLE_ZVg%40mail.gmail.com.

[Slide]

It would be super ideal to do transferring if we could figure out a good process for it, we lose GitHub Issues and some other things when we fork as is.

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAH-3Bie4r6MsTyNum4R57DSe3eMO7yNSVt1qbndVZviE%3D57Eug%40mail.gmail.com.

--

Website: http://earl-of-code.com

[Tim Jacomb]

I replied with Daniel's support ticket number (569994)

and got this back:

Hi Tim,

Thanks for your patience here. I checked internally once more and found that we can, in fact, do this for you. We've gone ahead and detached the repository into its own fork network.

Feel free to reach out if you have any additional questions about this!

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPiUgVf5HMgWmy%2BrQbnTH1p02Ngo9dXfqtR19SG%3DcxR_NLyWxw%40mail.gmail.com.

[Tim Jacomb]

GitHub support have offered to make the jenkinsci organisation the root of all of the fork networks

See

https://github.com/jenkins-infra/helpdesk/issues/2272#issuecomment-1216867119

Any objections or +1s?

Thanks

Tim

[Mark Waite]

+1 from me

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAH-3Bid6aSGqdh_JCMjvmANiin4TVbvOnMRLKGbF%2B89uM5iDFg%40mail.gmail.com.

[Gavin Mogan]

This is a one time operation for all past repos? What happens to all the new ones? Is the script pushing a new repo now (I forgot)?

Totally on board with fixing all the old repos, not being able to search them kinds sucks.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtHBOeuV4ZFPJXsRSfbiKZebUExEdyuyU%2B0EKjTr09nZWw%40mail.gmail.com.

[Tim Jacomb]

It currently still forks although there’s instructions telling people to delete their repo or raise a support ticket if it gets messed up

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAG%3D_DuuRwO2B1eV4RiQb_%2B1oyzjuZdoxmzq%3DLTw_R-D6Tpq5zw%40mail.gmail.com.

[Gavin Mogan]

So it's just all the old repos at once they have issues with? They are still okay with one off requests to fix a repo?

I wonder if they have a private API we could use that would prevent us from having to download then upload to prevent the fork 

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAH-3Bidzg-4X3X_J%3DYiPiZLZzLWZ%2BvrgTSWY%2B2FneEyKYOuUoQ%40mail.gmail.com.

[Alexander Brandes]

+1 on Tim's proposal.