[ANN] Change to how the update center JSON files are created

[Daniel Beck]

Hi everyone,

I pushed a change to the update center today.

TLDR: Wiki edits could have inadvertently removed plugins from the update site, breaking the suggested plugin installation experience for new Jenkins users. We fixed that by no longer relying on wiki data during update site generation. We also dropped the wiki URL requirement for the pom.xml.

----

You've probably seen these messages scroll by on this mailing list: "My plugin isn't on the update center, why?", or, worse, "My plugin disappeared from the update center, why?". While well-intentioned, the wiki URL requirement meant that plugins with no wiki page (e.g. because it was renamed), or a wiki page with the plugin-deprecated label, would not be distributed. Coupled with the setup wizard relying on the update site to get a useful initial set of plugins, and we have a potential disaster when someone with bad intentions figures out that anyone can rename or re-label wiki pages.

While we'd typically discuss changes that impact plugin developers like this in advance, as security officer I decided to prepare this in private, giving Jenkins CERT members and a number of other long time contributors the chance to review the proposal and request changes. Otherwise we'd have opened ourselves up to someone actually changing things around in the wiki, deliberately causing damage, and potentially break new Jenkins installs for everyone.

Since we now cannot rely on the wiki pages to exist, how can be accomplish the goals of having some minimal information for all plugins? Well, since October or so, we have plugins.jenkins.io, with a simple URL for every plugin. It shows all the metadata the plugin-info macro in the wiki did, just nicer -- and if there's a wiki URL, it gets scraped and shown right there. So the update center will now link to plugins.jenkins.io URLs for all plugins.

What other features did the wiki provide that we now cannot rely on anymore?

1. Deprecating plugins
The update center generator has a blacklist, and I've recently migrated over all plugins whose wiki pages were marked plugin-deprecated. Plugins will from now on be deprecated here as the sole source of truth:
https://github.com/jenkins-infra/backend-update-center2/blob/master/src/main/resources/artifact-ignores.properties

2. Titles and descriptions
The update center had a feature by which the name of the wiki page would be used if no <name/> for the plugin was defined in the pom.xml. Similarly, it actually preferred the wiki page {excerpt} to the <description/> in the pom.xml. These features are now gone, and whatever is in the pom.xml as <name/> and <description/> will be used.

3. Labels
Wiki page labels were used to categorize plugins in Jenkins. So I exported all labels as of a few days ago, and created this file based on that:
https://github.com/jenkins-infra/backend-update-center2/blob/master/src/main/resources/label-definitions.properties
The downside: No longer can plugin maintainers change the labels just by editing the wiki page, but need to file a PR.
The upside: No longer can some random people change the labels just by editing the wiki page; and this allows for easier large-scale changes. A new technology gains some rapid adoption and there's no label? It's much easier to relabel a bunch of things in a coordinated manner.

----

Contributor documentation (probably mostly the 'Hosting Plugins' wiki page and IRC bot generated Jira comments) will be updated accordingly.

As a side effect, this means that Tyler will now finally be able to update the wiki, as we don't have to fear that'll break update site generation.

I'd be happy to discuss proposals how to improve this. I think this solution improves the situation for developers, users, and the Jenkins project, and I'm looking forward to you telling me how wrong I am ;-)

Daniel

[Daniel Beck]

> On 07.03.2017, at 23:12, Daniel Beck <m...@beckweb.net> wrote:
>
> Since we now cannot rely on the wiki pages to exist, how can be accomplish the goals of having some minimal information for all plugins? Well, since October or so, we have plugins.jenkins.io, with a simple URL for every plugin. It shows all the metadata the plugin-info macro in the wiki did, just nicer -- and if there's a wiki URL, it gets scraped and shown right there. So the update center will now link to plugins.jenkins.io URLs for all plugins.

One thing I failed to explain is how we deal with actual plugin URLs now:

- If it's a wiki page, and it exists, that's what plugins.jenkins.io will scrape as before
- If there's a URL, and not the above, then a link there is displayed.
- If there's nothing, then a message saying so is displayed.

So users will still be able to get to your actual documentation with just as many clicks as before. If it's a wiki page, one click. If it's elsewhere, and your wiki page just pointed there, it's still two clicks (with the potential for fewer as we can add more scraping to the plugin site as appropriate).

[Andrew Bayer]

Nice improvement, I'd say.

A.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/6064DA7E-B3E0-46C4-A8F7-58FBEBE63422%40beckweb.net.

For more options, visit https://groups.google.com/d/optout.

[Jesse Glick]

On Tue, Mar 7, 2017 at 5:12 PM, Daniel Beck <m...@beckweb.net> wrote:
> The update center had a feature by which the name of the wiki page would be used if no <name/> for the plugin was defined in the pom.xml. Similarly, it actually preferred the wiki page {excerpt} to the <description/> in the pom.xml. These features are now gone, and whatever is in the pom.xml as <name/> and <description/> will be used.

Ah, finally.

Thanks for taking care of all this.

[Raphael Pionke]

Thanks for doing all the background work!

BTW: Can we adjust the links from https://twitter.com/jenkins_release to use the new plugins.jenkins.io page?

[Daniel Beck]

> On 10.03.2017, at 00:18, Raphael Pionke <raphael...@gmail.com> wrote:
>
> BTW: Can we adjust the links from https://twitter.com/jenkins_release to use the new plugins.jenkins.io page?

Right, that would be nice.

https://github.com/jenkins-infra/backend-update-center2/pull/118
https://github.com/jenkins-infra/jenkins.io/pull/736

[Gavin Mogan]

This is so awesome. Thanks for doing all this work.

Is the wiki going away (you probably did but it's hard to multi window on the phone)? Should any external links be going to the plugin site?

[Fritz Elfert]

Looks great!

However: The link generator apparently needs some more work.

Links on the left top ("GitHub", "Open Issues", "Latest rolling changes"
"Changes in x.x release") seem to be broken with some plugins:

E.g. broken on:

https://plugins.jenkins.io/jclouds-jenkins

but correct in the old wiki:

https://wiki.jenkins-ci.org/display/JENKINS/JClouds+Plugin


-Fritz

[Daniel Beck]

> On 10.03.2017, at 08:53, Gavin Mogan <ga...@gavinmogan.com> wrote:
>
> Is the wiki going away

No.

We cut the wiki dependency to allow the wiki to be updated etc. without worrying about breaking the update center, not to get rid of it.

http://lists.jenkins-ci.org/pipermail/jenkins-infra/2017-March/001050.html

> Should any external links be going to the plugin site?

Depends. The plugin (pom.xml) itself should continue linking to wherever its documentation actually lives (wiki or GitHub, typically), otherwise the plugin site has nowhere to link to/scrape.

Otherwise I prefer linking to the plugin site. It's nicer to look at than the wiki, and the URL for a plugin won't change, no matter where the documentation happens to live -- as long as we publish a plugin, it will have the same URL there. So that's nice.

[Daniel Beck]

> On 10.03.2017, at 12:00, Fritz Elfert <fr...@fritz-elfert.de> wrote:
>
> However: The link generator apparently needs some more work.
>
> Links on the left top ("GitHub", "Open Issues", "Latest rolling changes"
> "Changes in x.x release") seem to be broken with some plugins:
>
> E.g. broken on:
>
> https://plugins.jenkins.io/jclouds-jenkins
>
> but correct in the old wiki:
>
> https://wiki.jenkins-ci.org/display/JENKINS/JClouds+Plugin

Good find, I'll talk to Michael. The plugin site is his first contribution to the Jenkins project so he's not yet familiar with the inconsistent mess that's the jenkinsci org.

I think for now we can't have these links, as there's nowhere to adapt them for non-default cases (i.e. when it's not ${artifactId}-plugin). I'll look into that, perhaps as an addition to the otherwise useless 'scm' element in the update center (we've shut down Subversion, so those computed links from the wiki macro make no sense anymore).