Broken latest symlinks in mirrors of jenkins plugins

[Michael Prokop]

Hi,

The "latest" permalinks for the Jenkins plugins work fine from the
main site:

| % wget https://updates.jenkins.io/latest/swarm.hpi
| [...]
| % unzip swarm.hpi
| [...]
| % grep Plugin-Version META-INF/MANIFEST.MF
| Plugin-Version: 3.42

But all the mirrors seem to have stuck and outdated "latest"
symlinks for all the plugins, all dating back to 2022-08-12 and
pointing to outdated version (version 3.34 instead of 3.42 for the
swarm plugin as an example):

| % rsync -av --dry-run rsync://ftp-nyc.osuosl.org/jenkins/plugins/swarm/
| [...]
| receiving incremental file list
| drwxrwxr-x 4,096 2023/12/13 17:47:08 .
| lrwxrwxrwx 4 2022/08/12 11:35:14 latest -> 3.34
| drwxrwxr-x 30 2017/06/29 21:35:47 1.0
| [...]
| drwxrwxr-x 30 2022/03/08 08:35:55 3.31
| -rw-rw-r-- 8,598,985 2022/03/08 08:35:55 3.31/swarm.hpi
| drwxrwxr-x 30 2022/03/18 20:13:56 3.32
| -rw-rw-r-- 8,605,755 2022/03/18 20:13:56 3.32/swarm.hpi
| drwxrwxr-x 30 2022/06/18 02:13:11 3.33
| -rw-rw-r-- 8,657,915 2022/06/18 02:13:11 3.33/swarm.hpi
| drwxrwxr-x 30 2022/07/11 06:56:58 3.34
| -rw-rw-r-- 8,661,891 2022/07/11 06:56:58 3.34/swarm.hpi
| drwxrwxr-x 30 2022/09/08 05:22:58 3.35
| -rw-rw-r-- 8,781,226 2022/09/08 05:22:58 3.35/swarm.hpi
| drwxrwxr-x 30 2022/09/10 23:15:51 3.36
| -rw-rw-r-- 8,483,861 2022/09/10 23:15:51 3.36/swarm.hpi
| drwxrwxr-x 30 2022/10/31 16:59:39 3.37
| -rw-rw-r-- 2,664,239 2022/10/31 16:59:39 3.37/swarm.hpi
| drwxrwxr-x 30 2022/11/29 18:17:04 3.38
| -rw-rw-r-- 2,840,505 2022/11/29 18:17:04 3.38/swarm.hpi
| drwxrwxr-x 30 2022/12/01 18:20:48 3.39
| -rw-rw-r-- 2,840,475 2022/12/01 18:20:48 3.39/swarm.hpi
| drwxrwxr-x 30 2020/05/16 15:44:28 3.4
| -rw-rw-r-- 15,977 2017/04/10 15:00:25 3.4/swarm.hpi
| drwxrwxr-x 30 2023/04/09 22:19:45 3.40
| -rw-rw-r-- 2,781,049 2023/04/09 22:19:45 3.40/swarm.hpi
| drwxrwxr-x 30 2023/10/25 03:20:06 3.41
| -rw-rw-r-- 2,851,644 2023/10/25 03:20:06 3.41/swarm.hpi
| drwxrwxr-x 30 2023/12/13 17:47:08 3.42
| -rw-rw-r-- 2,867,243 2023/12/13 17:47:08 3.42/swarm.hpi
| [...]

Same for other mirrors as well:

| % rsync -av --dry-run rsync://ftp.belnet.be/jenkins/plugins/swarm/
| receiving incremental file list
| drwxr-xr-x 4,096 2023/12/13 17:47:08 .
| lrwxrwxrwx 4 2022/08/12 11:35:14 latest -> 3.34
| drwxr-xr-x 31 2017/06/29 21:35:47 1.0
| -rw-r--r-- 10,649 2016/04/20 02:35:11 1.0/swarm.hpi
| drwxr-xr-x 31 2017/06/29 21:35:47 1.1
| -rw-r--r-- 10,664 2016/04/20 02:35:10 1.1/swarm.hpi
| drwxr-xr-x 31 2020/05/16 15:44:28 1.10
| -rw-r--r-- 12,564 2016/04/20 02:35:10 1.10/swarm.hpi
| [...]
| drwxr-xr-x 31 2022/06/18 02:13:11 3.33
| -rw-r--r-- 8,657,915 2022/06/18 02:13:11 3.33/swarm.hpi
| drwxr-xr-x 31 2022/07/11 06:56:58 3.34
| -rw-r--r-- 8,661,891 2022/07/11 06:56:58 3.34/swarm.hpi
| drwxr-xr-x 31 2022/09/08 05:22:58 3.35
| -rw-r--r-- 8,781,226 2022/09/08 05:22:58 3.35/swarm.hpi
| drwxr-xr-x 31 2022/09/10 23:15:51 3.36
| -rw-r--r-- 8,483,861 2022/09/10 23:15:51 3.36/swarm.hpi
| drwxr-xr-x 31 2022/10/31 16:59:39 3.37
| -rw-r--r-- 2,664,239 2022/10/31 16:59:39 3.37/swarm.hpi
| drwxr-xr-x 31 2022/11/29 18:17:04 3.38
| -rw-r--r-- 2,840,505 2022/11/29 18:17:04 3.38/swarm.hpi
| drwxr-xr-x 31 2022/12/01 18:20:48 3.39
| -rw-r--r-- 2,840,475 2022/12/01 18:20:48 3.39/swarm.hpi
| drwxr-xr-x 31 2020/05/16 15:44:28 3.4
| -rw-r--r-- 15,977 2017/04/10 15:00:25 3.4/swarm.hpi
| drwxr-xr-x 31 2023/04/09 22:19:45 3.40
| -rw-r--r-- 2,781,049 2023/04/09 22:19:45 3.40/swarm.hpi
| [...]

Looks like something in the mirror architecture might have changed
in August 2022 and/or broke then?

regards
-mika-

[Michael Prokop]

Hi,

* Michael Prokop [Thu Dec 14, 2023 at 07:42:35AM +0100]:

> The "latest" permalinks for the Jenkins plugins work fine from the
> main site:
>
> | % wget https://updates.jenkins.io/latest/swarm.hpi
> | [...]
> | % unzip swarm.hpi
> | [...]
> | % grep Plugin-Version META-INF/MANIFEST.MF
> | Plugin-Version: 3.42
>
> But all the mirrors seem to have stuck and outdated "latest"
> symlinks for all the plugins, all dating back to 2022-08-12 and
> pointing to outdated version (version 3.34 instead of 3.42 for the
> swarm plugin as an example):

[...]

This is still on-topic and buggy. The latest symlinks are also
missing for all the plugins that got released after 2022-08-12.

See e.g. json-api + asm-api plugins, which both are missing
according "latest" symlinks on all the Jenkins mirrors:

http://ftp-nyc.osuosl.org/pub/jenkins/plugins/json-api/
http://ftp-nyc.osuosl.org/pub/jenkins/plugins/asm-api/

http://ftp.belnet.be/mirror/jenkins/plugins/json-api/
http://ftp.belnet.be/mirror/jenkins/plugins/asm-api/

https://ftp.halifax.rwth-aachen.de/jenkins/plugins/json-api/
https://ftp.halifax.rwth-aachen.de/jenkins/plugins/asm-api/

Might this be related to the removal of mirrorbrain at
https://github.com/jenkins-infra/helpdesk/issues/2888
?

regards
-mika-

[Mark Waite]

I don't think it is buggy.  The latest symlinks are no longer maintained.  Please use the Jenkins update center to obtain the most recent releases of the plugins rather than relying on symbolic links.

Mark Waite

[Michael Prokop]

Hi,

* Mark Waite [Mon Mar 18, 2024 at 01:20:10AM -0700]:

> I don't think it is buggy. The latest symlinks are no longer maintained.
> Please use the Jenkins update center to obtain the most recent releases of
> the plugins rather than relying on symbolic links.

Ok, but then I'm pretty sure we're not the only ones deploying
Jenkins plugins via puppet and rtyler's "jenkins::plugin"? See e.g.
https://github.com/rtyler/puppet-jenkins/blob/master/manifests/plugin.pp +
https://github.com/voxpupuli/puppet-jenkins/blob/master/manifests/plugin.pp
which relies on the "latest" to avoid having to pin plugins to a
specific version. Or is there any good API or something like that to
deploy plugins with configuration management?

regards
-mika-

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Infrastructure" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkins-infr...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/12255db2-3607-40f8-8ff3-cda93eaf814cn%40googlegroups.com.

[Mark Waite]

On Monday, March 18, 2024 at 2:31:25 AM UTC-6 Michael Prokop wrote:

Hi,

* Mark Waite [Mon Mar 18, 2024 at 01:20:10AM -0700]:
> I don't think it is buggy. The latest symlinks are no longer maintained.
> Please use the Jenkins update center to obtain the most recent releases of
> the plugins rather than relying on symbolic links.

Ok, but then I'm pretty sure we're not the only ones deploying
Jenkins plugins via puppet and rtyler's "jenkins::plugin"? See e.g.
https://github.com/rtyler/puppet-jenkins/blob/master/manifests/plugin.pp +
https://github.com/voxpupuli/puppet-jenkins/blob/master/manifests/plugin.pp
which relies on the "latest" to avoid having to pin plugins to a
specific version. Or is there any good API or something like that to
deploy plugins with configuration management?

We use the plugin installation manager tool quite widely in the Jenkins project infrastructure to download plugins.  When a plugin is listed without a specific version number, then the latest release of that plugin is downloaded.

I prefer to manage plugin versions explicitly so that I can always return to a precise configuration.  You can see my techniques at https://github.com/MarkEWaite/docker-lfs/blob/lts-with-plugins/update_plugins.py 

Mark Waite

[Michael Prokop]

Hi,

* Mark Waite [Mon Mar 18, 2024 at 08:47:22AM -0700]:

> On Monday, March 18, 2024 at 2:31:25 AM UTC-6 Michael Prokop wrote:
> > * Mark Waite [Mon Mar 18, 2024 at 01:20:10AM -0700]:

> > > I don't think it is buggy. The latest symlinks are no longer maintained.
> > > Please use the Jenkins update center to obtain the most recent releases
> > > of
> > > the plugins rather than relying on symbolic links.

> > Ok, but then I'm pretty sure we're not the only ones deploying
> > Jenkins plugins via puppet and rtyler's "jenkins::plugin"? See e.g.
> > https://github.com/rtyler/puppet-jenkins/blob/master/manifests/plugin.pp +
> > https://github.com/voxpupuli/puppet-jenkins/blob/master/manifests/plugin.pp
> > which relies on the "latest" to avoid having to pin plugins to a
> > specific version. Or is there any good API or something like that to
> > deploy plugins with configuration management?

> We use the plugin installation manager tool

> <https://github.com/jenkinsci/plugin-installation-manager-tool/> quite

> widely in the Jenkins project infrastructure to download plugins. When a
> plugin is listed without a specific version number, then the latest release
> of that plugin is downloaded.

Ah, thanks for the pointer, I'll see whether the puppet module can
be reasonably be adjusted to use that underneath instead.

> I prefer to manage plugin versions explicitly so that I can always return
> to a precise configuration.

FTR, we have a local mirror of all the relevant plugins available
and pin specific versions only if there's a good reason to do so
(e.g. known regression/bug/…).

But now having to keep up with all the dependencies of plugins we
rely on became quite annoying (we make sure to not have any plugins
installed that aren't managed through configuration management). Now
with the plugins not being available through the "latest" symlinks
that were available so far, but having to pin them to specific
versions, this tended to become a maintenance nightmare. :(

> You can see my techniques
> at https://github.com/MarkEWaite/docker-lfs/blob/lts-with-plugins/update_plugins.py

Thank you, appreciated!

regards
-mika-