speed your Jenkins by an adapter of Jenkins update center

[Rick]

Hi all,

An adapter of Jenkins update center.

## Why

Jenkins downloads the plugins by parsing the JSON file which comes from a update center.
One thing that you might already noticed is about the speed of downloading.
Accessing a global storage for everyone could be slow although 
there're many [mirror sites](http://mirrors.jenkins-ci.org/status.html).

If you deep into the file [update-center.json](https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json).
Almost every mirror file is base on `http://updates.jenkins-ci.org/download/plugins`.
So, the result is that these mirror sites can only speed up the process of downloading file update-center.json.
Jenkins can download the `.hpi` from your target mirror site.

## Background

It's might not be a good idea to change the `update-center.json` file directly. keeping sync all files
should be simple without other logic.

Second, you need to provide a certificate file if you changed the `update-center.json`. Because Jenkins
will validate the file before parsing it. It's necessary due to the safety reason.

## Design

* Create an adapter to replace the base URL
* Provide a certificate file

## How to

Here's a prototype implement which added into [localization-zh-cn-plugin](https://github.com/jenkinsci/localization-zh-cn-plugin/pull/115).
You Just need to take three steps if you want to use a real mirror of update center:

* install localization-zh-cn-plugin 1.0.10
* use the new certificate file
* change the update center URL

## Feedback

Please don't hesitate to tell us your thoughts.

You can find the codes from https://github.com/jenkins-zh/mirror-adapter.

Best regards,

Rick

--

Zhao Xiaojie (Rick)

Blog: https://github.com/LinuxSuRen

Twitter: https://twitter.com/suren69811254

[Baptiste Mathus]

Hey Rick,

FWIW I've read twice the Why, and am still unsure what problem this would be trying to solve. Care to maybe rephrase?

Thank you!

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAMM7nTF0UDz17%3Dcd95cPN9Om_FmyZY0OK1VPNVO%2BFASykkMnpQ%40mail.gmail.com.

[Xiaojie Zhao (Rick)]

Hi Baptiste,

Thanks for your response. Jenkins download the plugin by the URL which comes from update-center.json. For example this plugin http://updates.jenkins-ci.org/download/plugins/AnchorChain/1.0/AnchorChain.hpi.

The problem is that it’s too slow for me. I hope jenkins can download it from https://mirrors.tuna.tsinghua.edu.cn/jenkins/plugins/AnchorChain/1.0/AnchorChain.hpi.

On 14 Nov 2019, at 20:39, Baptiste Mathus <bma...@cloudbees.com> wrote:

FWIW

[Björn Pedersen]

Hi,

So the idea is to omit the round-trip to the main update center on each download. Community-wise this has one downside: all those downloads then do not show up in the download statistics.

But the same is true for all fully private mirrors behind firewalls that serve updatecenter.json  with internal urls.

Björn

[Rick]

Hi Björn,

It's true that those download statistics cannot be count. What about building a third party statistic site. And we can publish this data besides the official one as a reference. What do you think about this?

Best,

Rick

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/4e01b094-d310-448c-9706-6ed526a36e52%40googlegroups.com.

[Oliver Gondža]

I suspect few people would bother to report (accurate) numbers there.
Setting up internal mirror is the part that needs to be done and this is
the kind of bonus task that always gets outprioritized. IMO.

On 15/11/2019 12.14, Rick wrote:
> Hi Björn,
>
> It's true that those download statistics cannot be count. What about
> building a third party statistic site. And we can publish this data
> besides the official one as a reference. What do you think about this?
>
> Best,
> Rick
>
> On Fri, Nov 15, 2019 at 6:52 PM 'Björn Pedersen' via Jenkins Developers

> <jenkin...@googlegroups.com <mailto:jenkin...@googlegroups.com>>

> wrote:
>
> Hi,
>
> So the idea is to omit the round-trip to the main update center on
> each download. Community-wise this has one downside: all those
> downloads then do not show up in the download statistics.
> But the same is true for all fully private mirrors behind firewalls
> that serve updatecenter.json  with internal urls.
>
>
> Björn
>
> Am Donnerstag, 14. November 2019 13:45:11 UTC+1 schrieb Rick:
>
> Hi Baptiste,
>
> Thanks for your response. Jenkins download the plugin by the URL
> which comes from update-center.json. For example this plugin
> http://updates.jenkins-ci.org/download/plugins/AnchorChain/1.0/AnchorChain.hpi.
>
> The problem is that it’s too slow for me. I hope jenkins can
> download it from
> https://mirrors.tuna.tsinghua.edu.cn/jenkins/plugins/AnchorChain/1.0/AnchorChain.hpi.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to jenkinsci-de...@googlegroups.com

> <mailto:jenkinsci-de...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-dev/4e01b094-d310-448c-9706-6ed526a36e52%40googlegroups.com

> <https://groups.google.com/d/msgid/jenkinsci-dev/4e01b094-d310-448c-9706-6ed526a36e52%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
>
>
> --
> Zhao Xiaojie (Rick)
> Blog: https://github.com/LinuxSuRen
> Twitter: https://twitter.com/suren69811254
>

> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jenkinsci-de...@googlegroups.com

> <mailto:jenkinsci-de...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jenkinsci-dev/CAMM7nTF%3Ddk69exkvWN0Tv6rsdDa0cbcCLDSHaf8KxTsqKdgdRg%40mail.gmail.com
> <https://groups.google.com/d/msgid/jenkinsci-dev/CAMM7nTF%3Ddk69exkvWN0Tv6rsdDa0cbcCLDSHaf8KxTsqKdgdRg%40mail.gmail.com?utm_medium=email&utm_source=footer>.


--
oliver

[Rick]

What I'm talking about is not the internal mirror. The mirror is public. I just set up one mirror on GitHub which is https://github.com/jenkins-zh/update-center-mirror/tree/master/tsinghua.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/2e072c53-8421-a371-f9d5-4e53e5c88cd3%40gmail.com.

--

Zhao Xiaojie (Rick)

Blog: https://github.com/LinuxSuRen

Twitter: https://twitter.com/suren69811254

[Olblak]

Hey,

My two cents here, regarding update-center

I wanted to work on it quite a long time ago, in short that service is running in a deprecated cloud account, and I was thinking of refactoring it before moving it on Azure.

I created the ticket INFRA-1656 to track the work but unfortunately, I had to de-prioritized it, so if you have some thoughts, feel free to add them there as well.

---

gpg --keyserver keys.gnupg.net --recv-key 52210D3D

---

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAMM7nTGUKxgoWERq%2B1Cyx9bSUdiNDRwSskLc32h-FBt2W7%3DB%3DQ%40mail.gmail.com.