[JIRA] (JENKINS-53767) Offer plugin management tooling

Issue Type:	Epic
Assignee:	Unassigned
Components:	core
Created:	2018-09-25 13:42
Priority:	Major
Reporter:	Nicolas De Loof

Jenkins do offer a web UI to manage plugin installation from a live master instance, but in many deployment administrator would like to control installed plugins by version and by tooling, before jenkins master starts.

docker official image provides an install-plugins.sh script which evolved far beyond it's initial "help script" scope.
configuration-as-code implemented plugin installation hack based on PluginSite.Plugin#deploy.
open-source update center do expose plugins-version.json on casc request so one can manage versions for all hosted plugins, not just latest
custom war packager do support plugin installation on his own
jenkins evergreen do manage plugin installation based on updates from metadata server, using his own logic implemented by nodejs evergreen client.
jenkinsfile-runner do manage plugin installation implemented in Go
coreOS provides a groovy script to handle required plugins
cloudbees DEV@Cloud uses a set of Chef Ruby recipes for the same purpose
etc

Everybody is re-inventing the wheel, partially implementing the "details" of plugin management (signed metadata, artifacts checksums, plugins detached from core, ...). It becomes obvious Jenkins should provide adequate tooling for plugin installation outside jenkins live instance.

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

o.v.nenashev@gmail.com (JIRA)

unread,

Sep 25, 2018, 10:32:02 AM9/25/18

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on

Thanks for submitting it Nicolas De Loof! I totally agree we should streamline it at some point and offer a single solution

batmat@batmat.net (JIRA)

unread,

Sep 27, 2018, 4:10:04 PM9/27/18

to jenkinsc...@googlegroups.com

Baptiste Mathus updated an issue

Jenkins /

Change By:	Baptiste Mathus

Jenkins do does offer a web UI to manage plugin installation from a live master instance, but in many deployment deployments administrator would like to control installed plugins by version and by tooling, before jenkins master starts.
* docker official image provides an install-plugins.sh script which evolved far beyond it's its initial "help script" scope.
* configuration-as-code implemented plugin installation hack based on PluginSite.Plugin#deploy.
* open-source update center do does expose plugins-version.json on casc request so one can manage versions for all hosted plugins, not just latest
* custom war packager do does support plugin installation on his own
* jenkins evergreen do does manage plugin installation based on updates from metadata server, using his own logic implemented by nodejs evergreen client.
* jenkinsfile-runner do does manage plugin installation implemented in Go
* coreOS provides a [groovy script|https://github.com/coreos/jenkins-os/blob/master/init.groovy] to handle required plugins
* cloudbees DEV@Cloud uses a set of Chef Ruby recipes for the same purpose
* etc

Everybody is re-inventing the wheel, partially implementing the "details" of plugin management (signed metadata, artifacts checksums, plugins detached from core, ...). It becomes obvious Jenkins should provide adequate tooling for plugin installation _outside_ jenkins live instance.

james.strachan@gmail.com (JIRA)

unread,

Oct 4, 2018, 4:09:11 AM10/4/18

to jenkinsc...@googlegroups.com

James Strachan commented on

https://github.com/garethjevans/jenkins-quickstart01/tree/master/jenkinsConfig/departmentFoo/teamA

I'm a big fan of GitOps and using source code to define what version of jenkins core, dockerfile, plugins and their versions etc that folks wanna use. Then to upgrade a plugin/core its a Pull Request.

It'd be nice to enumerate all the different ways folks configure jenkins core + plugins + versions to generate/setup their Jenkins server.

e.g. there's the classic Dockerfile + plugins.txt approach like this...

or there's Custom War Packager style:

https://github.com/garethjevans/jenkins-cwp-quickstart01/blob/master/packager-config.yml#L11

then there's CasC YAML too.

I'd like a way to use something like updatebot to automate the generation of PRs for folks using GitOps to upgrade jenkins and/or plugins using either Evergreen, incrementals: https://repo.jenkins-ci.org/incrementals/ or releases: https://repo.jenkins-ci.org/releases/

james.strachan@gmail.com (JIRA)

unread,

Oct 4, 2018, 4:10:02 AM10/4/18

to jenkinsc...@googlegroups.com

James Strachan assigned an issue to James Strachan

Jenkins /

Change By:	James Strachan
Assignee:	James Strachan

james.strachan@gmail.com (JIRA)

unread,

Oct 4, 2018, 4:11:02 AM10/4/18

to jenkinsc...@googlegroups.com

James Strachan edited a comment on

I'm a big fan of GitOps and using source code to define what version of jenkins core, dockerfile, plugins and their versions etc that folks wanna use. Then to upgrade a plugin/core its a Pull Request.

It'd be nice to enumerate all the different ways folks configure jenkins core + plugins + versions via source code to generate/setup their Jenkins server.

e.g. there's the classic Dockerfile + plugins.txt approach like this...

[ https://github.com/garethjevans/jenkins-quickstart01/tree/master/jenkinsConfig/departmentFoo/teamA ]

or there's Custom War Packager style:

[https://github.com/garethjevans/jenkins-cwp-quickstart01/blob/master/packager-config.yml#L11]

then there's CasC YAML too.

I It shouldn ' d like a way t be too hard to use something like handle those 3 source code formats.

Then we could create a plugin for updatebot [https://github.com/jenkins-x/updatebot/] to automate the generation of PRs for folks using GitOps to upgrade jenkins and/or plugins using either Evergreen, incrementals: [https://repo.jenkins-ci.org/incrementals/] or releases: [ https://repo.jenkins-ci.org/releases/ ]

nicolas.deloof@gmail.com (JIRA)

unread,

Oct 4, 2018, 4:19:02 AM10/4/18

to jenkinsc...@googlegroups.com

Nicolas De Loof commented on

Part of the problem here is that those tools rely on the Update Center layout to bake download URL for version of plugins to install, while this layout is not defined by a spec.

So we need

define|document the plugin update center mechanism (metadata signature, how to retrieve hosted plugin versions and download URL, ...)
offer an implementation that can be used in various installation context

For (2) I'd like we extract PluginManager into a standalone executable jar so

it can run as a CLI jar and been used in docker images as a replacement for install-plugins script
it can be used by custom war packager as an utility library to implement the same need
the exact same bits are used by jenkins PluginManager (to enforce consistency)

batmat@batmat.net (JIRA)

unread,

Oct 4, 2018, 4:32:04 AM10/4/18

to jenkinsc...@googlegroups.com

Baptiste Mathus commented on

FWIW, CWP and Evergreen are using (no enforcement/checker yet, so there's probably smallish discrepancies) both use the Bill Of Materials JEP-309: https://github.com/jenkinsci/jep/tree/master/jep/309.

So we already have the standard format defined. So we should all use it. And if by chance it's deemed to miss some things, then JEP-309 should just be amended so that we agree on something. I agree with Nicolas De Loof this subject of handling Jenkins dependencies is becoming wasteful, fluffier and fluffier with all these variants, and should become a tool on its own so that people stop rewriting the (square) wheel everywhere.

I've thought about all this just a bit and I wonder technically this should be:

I find a tool like this would be good to be usable everwhere easily using CLI, so writing it in Go seems a good direction
at the same time, Jenkins ecosystem and consumers would probably need it in Java. Ideally, in that case, even Jenkins core associated logic could be extracted (cannot judge of the associated complexity though) so that really everyone use the same resolver tool.

Like for Maven dependencies, we need a tool able to:

compute a dependency tree without (or with as few as possible) downloads
download a whole dependency tree (including optional dependencies or not, here lies some combinatorial too).

batmat@batmat.net (JIRA)

unread,

Oct 4, 2018, 4:36:05 AM10/4/18

to jenkinsc...@googlegroups.com

Baptiste Mathus edited a comment on

FWIW, CWP and Evergreen are using (no enforcement/checker yet, so there's probably smallish discrepancies) both use the Bill Of Materials JEP-309: [https://github.com/jenkinsci/jep/tree/master/jep/309].

So we already have the standard format defined. So we should all use it. And if by chance it's deemed to miss some things, then JEP-309 should _just_ be amended so that we agree on something. I agree with [~ndeloof] this subject of handling Jenkins dependencies is becoming wasteful, fluffier and fluffier with all these variants, and should become a tool on its own so that people stop rewriting the (square) wheel everywhere.

I've thought about all this just a bit and I wonder technically this should be:

* I find a tool like this would be good to be usable everwhere easily using CLI, so writing it in Go seems a good direction
* at the same time, Jenkins ecosystem and consumers would probably need it in Java. Ideally, in that case, even Jenkins core associated logic could be extracted (cannot judge of the associated complexity though) so that really everyone use the same _resolver_ tool. A CLI usable packaged version of this tooling (fatjar or whatever) could then be used:
** in the Docker image builds
** in [Evergreen generation of resolved|https://github.com/jenkins-infra/evergreen/blob/52fd0456b9630607e981cb3c27dcfeaa9daea6ef/services/Makefile#L136] {{ingest.json}} from the {{essentials.yaml}}
** from Jenkins-X's updatebot (?)
** from CWP
** ...

Like for Maven dependencies, we need a tool able to:

* compute a dependency tree without (or with as few as possible) downloads
* download a whole dependency tree (including optional dependencies or not, here lies some combinatorial too).

jglick@cloudbees.com (JIRA)

unread,

Oct 10, 2018, 10:42:01 AM10/10/18

to jenkinsc...@googlegroups.com

Jesse Glick commented on

There is already a tool which understands how to update versions of components including incremental versions from a specified branch (such as origin master); as per JENKINS-51929, it would “just” need a shim to also read/write the JEP-309 format. That is only going to be useful, however, if the file is actually listing all the plugins you are bundling, as discussed in JENKINS-53506.

o.v.nenashev@gmail.com (JIRA)

unread,

Jan 12, 2019, 1:02:02 PM1/12/19

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on

What if we propose it as a GSoC 2019 project idea? If we do so, would somebody be interested to mentor it?

batmat@batmat.net (JIRA)

unread,

Jan 14, 2019, 6:25:01 PM1/14/19

to jenkinsc...@googlegroups.com

Baptiste Mathus commented on

I'd be ok to co-mentor this subject.

o.v.nenashev@gmail.com (JIRA)

unread,

Jan 16, 2019, 10:11:06 AM1/16/19

to jenkinsc...@googlegroups.com

Oleg Nenashev assigned an issue to Unassigned

Jenkins /

Change By:	Oleg Nenashev
Assignee:	James Strachan

o.v.nenashev@gmail.com (JIRA)

unread,

Jan 16, 2019, 10:11:08 AM1/16/19

to jenkinsc...@googlegroups.com

Oleg Nenashev updated an issue

Jenkins /

Change By:	Oleg Nenashev
Labels:	gsoc-2019-project-idea

josephp90@gmail.com (JIRA)

unread,

Mar 20, 2019, 7:25:02 PM3/20/19

to jenkinsc...@googlegroups.com

Joseph Petersen updated an issue

Jenkins /

Change By:	Joseph Petersen

Jenkins does offer a web UI to manage plugin installation from a live master instance, but in many deployments administrator would like to control installed plugins by version and by tooling, before jenkins master starts.
* docker official image provides an install-plugins.sh script which evolved far beyond its initial "help script" scope.
* configuration-as-code implemented plugin installation hack based on PluginSite.Plugin#deploy. (Removed as of v1.8 of JCasC plugin)
* open-source update center does expose plugins-version.json on casc request so one can manage versions for all hosted plugins, not just latest
* custom war packager does support plugin installation on his own
* jenkins evergreen does manage plugin installation based on updates from metadata server, using his own logic implemented by nodejs evergreen client.
* jenkinsfile-runner does manage plugin installation implemented in Go

* coreOS provides a [groovy script|https://github.com/coreos/jenkins-os/blob/master/init.groovy] to handle required plugins
* cloudbees DEV@Cloud uses a set of Chef Ruby recipes for the same purpose
* etc

Everybody is re-inventing the wheel, partially implementing the "details" of plugin management (signed metadata, artifacts checksums, plugins detached from core, ...). It becomes obvious Jenkins should provide adequate tooling for plugin installation _outside_ jenkins live instance.

o.v.nenashev@gmail.com (JIRA)

unread,

Mar 20, 2019, 7:31:03 PM3/20/19

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on

FTR https://jenkins.io/projects/gsoc/2019/project-ideas/plugin-installation-manager-tool-cli/

josephp90@gmail.com (JIRA)

unread,

Mar 20, 2019, 7:36:02 PM3/20/19

to jenkinsc...@googlegroups.com

Joseph Petersen commented on

In JCasC we ended up removing the plugin management support was causing issues and startup. See https://github.com/jenkinsci/configuration-as-code-plugin/pull/769

Thank you Oleg Nenashev for the GSoC write up

josephp90@gmail.com (JIRA)

unread,

Mar 20, 2019, 7:37:06 PM3/20/19

to jenkinsc...@googlegroups.com

Joseph Petersen edited a comment on

In JCasC we ended up removing the plugin management support was causing issues and startup. See [https://github.com/jenkinsci/configuration-as-code-plugin/pull/769]

Thank you [~oleg_nenashev] for the GSoC write up

josephp90@gmail.com (JIRA)

unread,

Mar 20, 2019, 7:37:07 PM3/20/19

to jenkinsc...@googlegroups.com

Joseph Petersen edited a comment on

In JCasC we ended up removing the plugin management support as it was causing issues and on startup. See [https://github.com/jenkinsci/configuration-as-code-plugin/pull/769]

Thank you [~oleg_nenashev] for the GSoC write up

josephp90@gmail.com (JIRA)

unread,

Mar 20, 2019, 7:37:07 PM3/20/19

to jenkinsc...@googlegroups.com

Joseph Petersen edited a comment on

In JCasC we ended up removing the plugin management support as it was causing issues and startup. See [https://github.com/jenkinsci/configuration-as-code-plugin/pull/769]

Thank you [~oleg_nenashev] for the GSoC write up

josephp90@gmail.com (JIRA)

unread,

Apr 6, 2019, 3:45:03 AM4/6/19

to jenkinsc...@googlegroups.com

Joseph Petersen commented on

From JCasC point of view:

Should be possible to add an optional JCasC configurator to Jenkins core that would take care of Plugin installation but it would still require more plugin installation tooling.

Regarding plugin installation tooling

I still strongly believe that tooling should either stay within Java or a mix of Java and Go.

If it was pure Java I could foresee the tool becoming overly large and burden by dependency hell.

A mix of Go and Java could work.

Go would be the main source library and has a better native HTTP client.

Java jar would just be a wrapper for calling go functions: https://medium.com/learning-the-go-programming-language/calling-go-functions-from-other-languages-4c7d8bcc69bf

josephp90@gmail.com (JIRA)

unread,

Apr 6, 2019, 3:49:03 AM4/6/19

to jenkinsc...@googlegroups.com

Joseph Petersen edited a comment on

From JCasC point of view:

Should be possible to add an optional JCasC configurator to Jenkins core that would at least take care of reading the configurations at startup and invoke the installation of plugins before ever starting to configure plugins.
That would at least take care of plugin installation where JCasC is concerned.
Though it would still require more plugin installation tooling.

Regarding plugin installation tooling

I still strongly believe that tooling should either stay within Java or a mix of Java and Go.

If it was pure Java I could foresee the tool becoming overly large and burden by dependency hell.

A mix of Go and Java could work.

Go would be the main source library and has a better native HTTP client.

Java jar would just be a wrapper for calling go functions: [https://medium.com/learning-the-go-programming-language/calling-go-functions-from-other-languages-4c7d8bcc69bf]

josephp90@gmail.com (JIRA)

unread,

Apr 6, 2019, 3:49:03 AM4/6/19

to jenkinsc...@googlegroups.com

Joseph Petersen edited a comment on

From JCasC point of view:

Should be possible to add an optional JCasC configurator to Jenkins core that would at least take care of Plugin reading the configurations at startup and invoke the installation but of plugins before ever starting to configure plugins. That would at least take care of plugin installation where JCasC is concerned. Though it would still require more plugin installation tooling.

Regarding plugin installation tooling

I still strongly believe that tooling should either stay within Java or a mix of Java and Go.

If it was pure Java I could foresee the tool becoming overly large and burden by dependency hell.

A mix of Go and Java could work.

Go would be the main source library and has a better native HTTP client.

Java jar would just be a wrapper for calling go functions: [https://medium.com/learning-the-go-programming-language/calling-go-functions-from-other-languages-4c7d8bcc69bf]

josephp90@gmail.com (JIRA)

unread,

Apr 6, 2019, 3:50:03 AM4/6/19

to jenkinsc...@googlegroups.com

Joseph Petersen edited a comment on

From JCasC point of view:

Should be possible to add an optional JCasC configurator to Jenkins core that .
That would at least take care of reading the configurations at startup and invoke the installation of plugins before ever starting to configure plugins.

That would at least take care of plugin installation where JCasC is concerned.
Though it would still require more plugin installation tooling.

Regarding plugin installation tooling

I still strongly believe that tooling should either stay within Java or a mix of Java and Go.

If it was pure Java I could foresee the tool becoming overly large and burden by dependency hell.

A mix of Go and Java could work.

Go would be the main source library and has a better native HTTP client.

Java jar would just be a wrapper for calling go functions: [https://medium.com/learning-the-go-programming-language/calling-go-functions-from-other-languages-4c7d8bcc69bf]

josephp90@gmail.com (JIRA)

unread,

Apr 6, 2019, 3:51:03 AM4/6/19

to jenkinsc...@googlegroups.com

Joseph Petersen edited a comment on

From JCasC point of view:

Should be possible to add an optional JCasC configurator to Jenkins core.
That would at least take care of reading Reading the configurations at startup and invoke the installation of plugins before ever starting to configure plugins.

That would at least take care of plugin installation where JCasC is concerned.
Though it would still require more plugin installation tooling.

Regarding plugin installation tooling

I still strongly believe that tooling should either stay within Java or a mix of Java and Go.

If it was pure Java I could foresee the tool becoming overly large and burden by dependency hell.

A mix of Go and Java could work.

Go would be the main source library and has a better native HTTP client.

Java jar would just be a wrapper for calling go functions: [https://medium.com/learning-the-go-programming-language/calling-go-functions-from-other-languages-4c7d8bcc69bf]

nicolas.deloof@gmail.com (JIRA)

unread,

Apr 6, 2019, 4:33:02 AM4/6/19

to jenkinsc...@googlegroups.com

Nicolas De Loof commented on

> Should be possible to add an optional JCasC configurator to Jenkins core.

if added to Jenkins Core then JCasC can't be a plugin anymore.

My initial idea was to refactor PluginManager (and siblings) so they can be used as a standalone library for plugin management/installation, then can be either embedded into another tool, or invoked from command line with the adequate `-cp` option pointing to this lib within jenkins.war/WEB-INF/libs.

I'm also fine with a Go implementation which would make perfect sense for portability in various scenarios. In both cases, a well defined JEP on plugin management in Jenkins would help, as today we have to reverse-engineer the existing codebase to understand plugin management and update-center mysteries.

o.v.nenashev@gmail.com (JIRA)

unread,

May 7, 2019, 3:29:05 AM5/7/19

to jenkinsc...@googlegroups.com

Oleg Nenashev commented on

Just an update here, we will have a GSoC project for it this year: https://summerofcode.withgoogle.com/projects/#5741119520899072

The format is yet to be defined, but it will be handled under the Platform SIG umbrella.

o.v.nenashev@gmail.com (JIRA)

unread,

May 7, 2019, 3:29:06 AM5/7/19

to jenkinsc...@googlegroups.com

Oleg Nenashev updated an issue

Jenkins /

Change By:	Oleg Nenashev
Labels:	gsoc-2019 gsoc - 2019- project-idea

natasha.stopa@gmail.com (JIRA)

unread,

May 23, 2019, 11:41:02 PM5/23/19

to jenkinsc...@googlegroups.com

Natasha Stopa assigned an issue to Natasha Stopa

Jenkins /

Change By:	Natasha Stopa
Assignee:	Natasha Stopa

natasha.stopa@gmail.com (JIRA)

unread,

Jun 26, 2019, 6:56:03 PM6/26/19

to jenkinsc...@googlegroups.com

Natasha Stopa updated an issue

Jenkins /

Change By:	Natasha Stopa

Jenkins does offer a web UI to manage plugin installation from a live master instance, but in many deployments administrator would like to control installed plugins by version and by tooling, before jenkins master starts.
* docker official image provides an install-plugins.sh script which evolved far beyond its initial "help script" scope.
* configuration-as-code implemented plugin installation hack based on PluginSite.Plugin#deploy. (Removed as of v1.8 of JCasC plugin)
* open-source update center does expose plugins-version.json on casc request so one can manage versions for all hosted plugins, not just latest
* custom war packager does support plugin installation on his own
* jenkins evergreen does manage plugin installation based on updates from metadata server, using his own logic implemented by nodejs evergreen client.
* jenkinsfile-runner does manage plugin installation implemented in Go
* coreOS provides a [groovy script|https://github.com/coreos/jenkins-os/blob/master/init.groovy] to handle required plugins
* cloudbees DEV@Cloud uses a set of Chef Ruby recipes for the same purpose
* etc

Everybody is re-inventing the wheel, partially implementing the "details" of plugin management (signed metadata, artifacts checksums, plugins detached from core, ...). It becomes obvious Jenkins should provide adequate tooling for plugin installation _outside_ jenkins a live Jenkins instance.

nickbroon@gmail.com (JIRA)

unread,

Jul 4, 2019, 5:09:03 AM7/4/19

to jenkinsc...@googlegroups.com

Nicholas Brown commented on

This might help address the following ansible jenkin plugin management problem: https://github.com/ansible/ansible/issues/24864

fwilhelm.wgt@gmail.com (JIRA)

unread,

Jul 18, 2019, 8:31:02 AM7/18/19

to jenkinsc...@googlegroups.com

Florian Wilhelm commented on

Hi,

I like the idea of this project quite a lot, cool stuff.

I have one potential use case where I don’t find a ticket matching it.

For context, I work at SAP and we have a Jenkins image based on upstream with a custom plugins.txt file. Currently we use latest of all plugins. What I would like is to feed that plugins list into the tool and get a now up to date plugins.txt with all dependencies resolved and versions fixed, so we can use that to build a reproducible image. So, instead of a directory where the plugins are installed, my desired output is a “resolved” plugins.txt file. Is this in scope for this project? I think most required code is already there, and thus implementation should not be too much work (looking at PluginManager.java from a birds eye view).

Kind regards

Florian

jglick@cloudbees.com (JIRA)

unread,

Jul 18, 2019, 12:51:12 PM7/18/19

to jenkinsc...@googlegroups.com

Jesse Glick commented on

JENKINS-47498 might be a better input source, as the list of plugins would actually be mechanically vetted, not merely the latest on offer.