Jenkins Plugin Install Wizard - check out the progress (video) ...

[Tom Fennelly]

Hi.

Falling out of the "Revisiting bundled plugins" thread, we now have ongoing work in PR #1822 (Keith Zantow, Gus Reiber and myself) to unbundle plugins from Jenkins core. As part of that, we are working on adding a Plugin Install Wizard as a way of making it easy to configure the initial set of plugin in Jenkins (for new or existing users).

Here's a short (rough) video showing the wizard: https://youtu.be/9pq5tHm4nWs

Keith did all the UI work with help from Gus, so kudos to them.

Also, please download and try he build from the CI server: jenkins.war.

Regards,

T.

[Daniel Beck]

Hi everyone,

While it is a much desired feature to have an initial setup experience that includes the ability to choose plugins, I'd rather not overwhelm inexperienced users by exposing them to the full list of plugins from the start. Forcing those with no experience to choose between 1000+ plugins before they can even start using Jenkins is an impossible task. They may not even be fully aware what plugins are for, and that Jenkins doesn't really do much without any of them. Who points them to Folders, Timestamper, or Build Timeout Plugin? Sure, they can always accept the defaults, but what if they want to modify something that should be simple enough, like add support for a less popular SCM like Mercurial?

A selection of known good plugins is a much better choice for this particular setup dialog, and that's how it's currently implemented. The initial selection of 'curated' (= available for selection) plugins was provided to Gus and Tom by me (and tweaked by them after some early reviewer feedback). I used the stats data[1] as a starting point and then tried to define useful categories for them. Those aren't always perfect, but should be close enough to work. The specific selections were made based on the following criteria:

- Is the plugin useful, and does it actually work? (Mostly approximated by install count)
- Does the plugin integrate well with Jenkins? Does it work well in secured and distributed Jenkins?
- Are the features it provides useful to someone fairly new to Jenkins?
- Even if it's not popular, is it a reasonably common SCM? (SCM's are so fundamental to successful Jenkins use that these shouldn't be as heavily restricted as some of the other categories.)

I admit the initial choices are somewhat colored by personal preferences, but note that these are simply that -- initial choices to unblock Tom, Gus and Keith, and get the discussion going. I also requested the data be moved into the update site, so we can review the choices periodically and update to what we consider the best choices of plugins for new users. This may not make it into the first version, but would then be added shortly afterwards.

Likewise, the plugins to be selected for installation by default are a subset of the curated list. For the SCMs, I used the 2014 Eclipse survey[2] to determine what's popular. Otherwise I selected a list of what I considered most universally useful, things that have people wondering why they're not core features. Again, despite the criteria above, there's probably some personal bias in this selection, but I'm not sure there's a much better way to provide a first draft.

So please take a look at the current choices and let us know what you think:

https://github.com/jenkinsci/jenkins/blob/unbundling-plugins/war/src/main/js/api/plugins.js

Do you think the criteria above aren't really useful? Do you have other suggestions for the categories, the curated list of plugins, or the default selection? Please let us know. If you're proposing specific changes, please provide some details -- don't just ask why some plugin isn't part of the list.

Thanks in advance,
Daniel

1: http://stats.jenkins-ci.org/jenkins-stats/svg/201508-top-plugins1000.svg
2: http://www.slideshare.net/IanSkerrett/eclipse-community-survey-2014 slide 24

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/7bdf63f7-621c-4dae-b5fc-19c60685fc06%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[James Nord]

I think we should include description-setter plugin.  It's very useful when building parametrized jobs to see from the last 10 built which is the build you are interested in.

matrix-plugin seems to be in the wrong bucket, fits better with "Build Tools" which I have yet to come up for a better name even though I don;t think it works as is.

[Daniel Beck]

On 29.09.2015, at 21:45, James Nord <jn...@cloudbees.com> wrote:

> matrix-plugin seems to be in the wrong bucket, fits better with "Build Tools" which I have yet to come up for a better name even though I don;t think it works as is.

The idea here was that the label axis feature is pretty cool here, and it also lets you build on different nodes even without that. But you're right, it's a weird one.

[Victor Martinez]

Hi there,

Just went through the list of plugins and I'm really surprise there is no references about the JobDSL plugin at all. I do understand the JobDSL and Workflow might coexist and be somehow related but knowing it's been available longer with more number of installations, it's actively maintained by the community and it was one of the first plugins talking about Configuration as Code paradigm I believe it should be part of the Recommended plugins too. What do you think?

Cheers

[Laurence Bordowitz]

What constitutes admission into this special class of "curated" plugins? There will be so many requests of, "I would prefer if my plugin, or a plugin that I love, be in this special list," so knowing how to make the grade, or having a process/committee, will help manage the chaos somewhat.

 

-- Larry Bordowitz

> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsub...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/7bdf63f7-621c-4dae-b5fc-19c60685fc06%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/9267D9A5-147B-4514-A028-C95550D2FF29%40beckweb.net.

[Daniel Beck]

On 29.09.2015, at 23:23, Victor Martinez <victormar...@gmail.com> wrote:

> Just went through the list of plugins and I'm really surprise there is no references about the JobDSL plugin at all. I do understand the JobDSL and Workflow might coexist and be somehow related but knowing it's been available longer with more number of installations, it's actively maintained by the community and it was one of the first plugins talking about Configuration as Code paradigm I believe it should be part of the Recommended plugins too. What do you think?

Last I checked, Job DSL does not integrate well with secured instances. Grant one user Job/Configure on a single freestyle job and they can do whatever they want to your instance. Therefore it fails the second criterion I mentioned.

[Slide]

I hate to say this, but this effort is really coming off the wrong way, at least to me. I understand the idea of making the user experience better, but to me its coming across as CloudBees employees pushing CloudBees plugins into this new user experience. It may be my own personal perception, but that is how its coming across.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/D6B66186-92C3-4949-8EA8-CAFBFEE5F291%40beckweb.net.

[Daniel Beck]

On 30.09.2015, at 00:03, Slide <slide...@gmail.com> wrote:

> I hate to say this, but this effort is really coming off the wrong way, at least to me. I understand the idea of making the user experience better, but to me its coming across as CloudBees employees pushing CloudBees plugins into this new user experience. It may be my own personal perception, but that is how its coming across.

Could you be a bit more specific?

I mean, yes, I included CloudBees Folders, but that's because it works well and is useful. Nested views aren't nearly as nice, and they don't play well with permissions (while folders integrate e.g. with matrix-auth, allowing for nice per-team folders).

But other than that, I don't remember including a single "CloudBees plugin".

[Slide]

I thought I saw workflow as one of the plugins that was a "recommended" plugin. If it wasn't there, then I misread and you can ignore me.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/089A6367-36F4-4AF6-8989-441311CD9E40%40beckweb.net.

[Victor Martinez]

That's a fair point actually, thanks for your answer. 

On the other hand, if the security is one of the important pieces then any Builder might be affected easily since AFAIK they don't run in any sandbox therefore you might run "rm -rf /" and change the restriction label as "master". Does it mean "Shell plugin" (I know it's part of the core) is unsecured? If so, does it mean any other plugins which are using it are also unsecured? And that's an example of bypassing the "security layout". Don't get me wrong, I do like the idea of adding value to Jenkins with this wizard feature, but what I don't like is the way of filtering plugins which are useful IMO.

Thanks again

[Tom Fennelly]

On 29 September 2015 at 23:24, Slide <slide...@gmail.com> wrote:

I thought I saw workflow as one of the plugins that was a "recommended" plugin. If it wasn't there, then I misread and you can ignore me.

Workflow is there but it's not a cloudbees plugin.

[Robert Sandell]

Tom, my brain is still stuck on the categories ;)

I think the experience could be made a bit better if the categories was ordered in a bit of a smarter way, both to follow some sort of thought pattern and to ease the user into the depths of what plugins can do.

An example:

Source Code Management

Build Tools

Reporters and Notifiers

CD etc

Job Organisation (folders, views et.al.)

"All"

(I didn't remember what categories you chose exactly so I made some up)

By having the user first think about their build environment (the CI part), the scm, tools and report types etc. And then show conceptual things like CD pipelines. And at the end get the user thinking about how to organise it all into folders and such. and maybe some explanatory texts about the category itself.

That could make the experience a bit easier and more thought through?

/B

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BbPaoLqr5ATeM%2BDgF%2BhLME%3DWRj_2d43q1f1t592syijsW1Dvg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Robert Sandell

Software Engineer

CloudBees Inc.

[Tom Fennelly]

On 30 September 2015 at 10:22, Robert Sandell <rsan...@cloudbees.com> wrote:

Tom, my brain is still stuck on the categories ;)

I think the experience could be made a bit better if the categories was ordered in a bit of a smarter way, both to follow some sort of thought pattern and to ease the user into the depths of what plugins can do.

An example:

Source Code Management

Build Tools

Reporters and Notifiers

CD etc

Job Organisation (folders, views et.al.)

"All"

(I didn't remember what categories you chose exactly so I made some up)

By having the user first think about their build environment (the CI part), the scm, tools and report types etc. And then show conceptual things like CD pipelines. And at the end get the user thinking about how to organise it all into folders and such. and maybe some explanatory texts about the category itself.

That could make the experience a bit easier and more thought through?

/B

Yeah, you may be right. I didn't come up with the categories (I think Daniel did) and, to be honest, I'm not too hung up on that right now because I know we'll probably need to go through a few rounds before we find what works "best" in the general sense.

I emphasise "best" there because I think no matter what we do it will not be perfect and there will be situations where some may find it's not logical (or smart) in some way. That's just the nature of this kind of thing I think. So, if we concentrate on the mechanics of how the wizard looks/works and make sure we provide a way to easily change the categorisations etc later (without having to rewrite code etc), then we should be good. That's my hope anyway.

[Robert Sandell]

Yes, so when coding it up you can make sure that the categories can be ordered, not by sorting on the name and that there also could be description text for each category ;)

/B

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BbPaoK%2B8Y-wHp_MCxtkMNuV-ZzLqN2JsiDLbk5hNUsMcKTy9w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Tom Fennelly]

On 30 September 2015 at 10:51, Robert Sandell <rsan...@cloudbees.com> wrote:

Yes, so when coding it up you can make sure that the categories can be ordered, not by sorting on the name and that there also could be description text for each category ;)

/B

That's how it is now ... they're not sorted by name or anything. The categories are displayed in the order in which they are "loaded", as are the plugins within the categories. So ... we can easily change that order.

So it sounds as though you've spotted a problem somewhere but I don't hear you say where specifically. If you have, please let us know.

[Daniel Beck]

On 29.09.2015, at 23:49, 'Laurence Bordowitz' via Jenkins Developers <jenkin...@googlegroups.com> wrote:

> What constitutes admission into this special class of "curated" plugins? There will be so many requests of, "I would prefer if my plugin, or a plugin that I love, be in this special list," so knowing how to make the grade, or having a process/committee, will help manage the chaos somewhat.

True, we'll probably need to establish some (lightweight) process here -- this may be as simple as what we're currently doing wrt pull requests to core, basically waiting to build enough consensus until someone presses the button.

But note that the majority of plugins will already fail the "popularity" test -- if you don't have 1k+ installs, it's unlikely you'll get onto the list. Unless you all agree this criterion is stupid; in that case, we'll need to think of something :-)

[evernat]

I suggest to add the findbugs and warnings plugins next to checkstyle. They are both largely used. And using findbugs is common when using checkstyle and pmd.

By the way, a wizard suggesting 78 plugins instead of 1100 plugins is really great for newcomers. This list even remind me of 2 or 3 plugins I should have installed.

- Emeric

[Victor Martinez]

I wonder whether we could somehow provide certain type of wizards. Let's say if you are using an enterprise version of Jenkins you might require the below recommend and secured plugins, if you use Jenkins for php then you might require another recommend list of plugins, and so on. 

An an example, eclipse already provide those default installations based on those needs:

- https://eclipse.org/ide/

Then that wizard installation might be more flexible and provide different solutions for different organizations.

What do you think? Is it something feasible from the implementation point of view?

Cheers

[Tom Fennelly]

On 30 September 2015 at 17:51, Victor Martinez <victormar...@gmail.com> wrote:

I wonder whether we could somehow provide certain type of wizards. Let's say if you are using an enterprise version of Jenkins you might require the below recommend and secured plugins, if you use Jenkins for php then you might require another recommend list of plugins, and so on. 

An an example, eclipse already provide those default installations based on those needs:

- https://eclipse.org/ide/

Then that wizard installation might be more flexible and provide different solutions for different organizations.

What do you think? Is it something feasible from the implementation point of view?

Cheers

Hi Victor.

We did debate creating such as wizard on the other thread but came to the conclusion that it would be too difficult to get agreement on it. So, we just decided to go for the easier option as a v1, see how that goes and then decide how it might evolve.

T.

[James Nord]

OT: That's why you should never run builds on your master...

[Michael Neale]

However, wasn't the statistic that greater than 60% of installations (may have been 80%) run builds only on master, non distributed across slaves? 

(this does not change your point, McDonalds is still one of the most popular restaurants in the world, popularity doesn't necessarily mean a good idea!). 

[Victor Martinez]

Totally agree, even though it's one of the "anti" patterns, but there are a bunch of different reason why people use the master as a build slave. You could somehow easily manipulate it in certain ways, no just removing files randomly  but IMO if you are thinking about an utopian scenario, then those recommended plugins might solve those cases. On the other hand, since they are a bunch of different organisations with different requirements, some of them are more related to automate with the freedom of no using any security layouts as we probably do in our Jenkins community, trusting people. others are more based on security and blocking any changes if they are not part of any CAB. 

Therefore, IMO those categories and recommended plugin as long as they are not part of any Commercial version shouldn't be blocked by those 3 "filters", don't you think? I mean, if I was Company who provides a Jenkins enterprise version I shouldn't allow any unsecured plugins for bare things even though it's also based on OSS, but as long as this is an open source version it shouldn't restrict those plugins at all. 

Again, I strongly believe this wizard is a great idea but this approach prunes too much plugins.

Thanks

[Daniel Beck]

On 30.09.2015, at 08:30, Victor Martinez <victormar...@gmail.com> wrote:

> On the other hand, if the security is one of the important pieces then any Builder might be affected easily since AFAIK they don't run in any sandbox therefore you might run "rm -rf /" and change the restriction label as "master". Does it mean "Shell plugin" (I know it's part of the core) is unsecured? If so, does it mean any other plugins which are using it are also unsecured? And that's an example of bypassing the "security layout". Don't get me wrong, I do like the idea of adding value to Jenkins with this wizard feature, but what I don't like is the way of filtering plugins which are useful IMO.

(Accidentally sent this to Victor directly)

It's possible to secure an instance well enough from this by e.g. setting the number of executors on master to 0. Another option would be plugins that limit what can be built where using the QueueTaskDispatcher extension point. Sure, users can still wreak havoc on slaves, but that's probably less of an issue than having unrestricted access to JENKINS_HOME.

Regarding the plugin criteria, it looked like a good idea to not surprise users by including plugins that make any security configuration they may configure irrelevant. Maybe I'm wrong about this and nobody cares (but then I wonder why Role Strategy is so popular, complex security setups that are too painful doing with matrix-auth is the one thing it does). So if there are others who think that plugins undermining an admin-defined security setup should be allowed into the wizard, please say so.

[Daniel Beck]

On 01.10.2015, at 08:42, Victor Martinez <victormar...@gmail.com> wrote:

> Therefore, IMO those categories and recommended plugin as long as they are not part of any Commercial version shouldn't be blocked by those 3 "filters", don't you think? I mean, if I was Company who provides a Jenkins enterprise version I shouldn't allow any unsecured plugins for bare things even though it's also based on OSS, but as long as this is an open source version it shouldn't restrict those plugins at all.
>
> Again, I strongly believe this wizard is a great idea but this approach prunes too much plugins.

Please provide an alternative list of criteria, or plugins -- right now it's not clear how you'd like to proceed here. Or are you rejecting the concept of a curated list entirely, preferring to show all 1000+ plugins?

[Victor Martinez]

Sure, Since there was already an agreement of having a single wizard as Tom already noticed, sorry about that suggestion, I joint the thread a bit late, I believe in the below criteria (see my red and blue comments)

- Is the plugin useful, and does it actually work? (Mostly approximated by install count) 

   Agree

- Does the plugin integrate well with Jenkins? Does it work well in secured and distributed Jenkins? 

   As I already mention "secured" shouldn't pruned any plugins as long as those are heavily used by the community and it is an open source version

- Are the features it provides useful to someone fairly new to Jenkins? 

   What do you mean about someone fairly new to Jenkins? Configuration as Code (Workflow, JobDSL or jenkins builder) will be fairly complex to understand by someone new to Jenkins, even using  you use Workflow, don't you think?Then some of those recommended plugins don't follow this criteria.

- Even if it's not popular, is it a reasonably common SCM? (SCM's are so fundamental to successful Jenkins use that these shouldn't be as heavily restricted as some of the other categories.) 

   Agree

As an example of one of my production jenkins instances, there are a bunch of plugins and probably some of them will not part of the recommended list, since the above list criteria.

Plugin	Version	Number of jobs
hipchat	0.1.6	449
throttle-concurrents	1.8.3	292
rebuild	1.22	291
timestamper	1.5.13	261
build-failure-analyzer	1.13.0	254
envinject	1.90	236
mercurial	1.50	163
build-blocker-plugin	1.6	157
heavy-job	1.1	157
mailer	1.15	150
jira	1.39	114
git	2.3.5	89
multiple-scms	0.3	75
ws-cleanup	0.21	56
ownership	0.7	47
parameterized-trigger	2.25	45
github	1.11.3	26
matrix-project	1.6	24
gradle	1.24	22
build-timeout	1.14.1	20
ant	1.2	20
groovy	1.19	20
conditional-buildstep	1.3.3	16
junit	1.9	15
chucknorris	0.5	10
groovy-postbuild	1.10	8
copyartifact	1.31	8
build-user-vars-plugin	1.4	7
email-ext	2.38.2	6
job-dsl	1.38	5
postbuildscript	0.16	5
jenkins-multijob-plugin	1.13	5
ircbot	2.25	5
custom-tools-plugin	0.4.4	4
htmlpublisher	1.4	4
sonar	2.1	3
subversion	2.5.1	3
ghprb	1.28.6	2
scriptler	2.7	2
javadoc	1.1	2
locks-and-latches	0.6	2
xvnc	1.17	2
findbugs	4.56	2
cobertura	1.9.5	2
feature-branch-notifier	1.3	2
unity3d-plugin	0.7	2
nunit	0.16	2
artifactory	2.2.4	1
publish-over-ssh	1.11	1
dynamicparameter	0.2.0	1
extended-choice-parameter	0.34	1

[Baptiste Mathus]

IIUC, though it seems a good idea to try and base the list to the actual usage of plugins, this wouldn't work for many plugins that do not have to be configured.

For example, it wouldn't work for mine which is enabled globally (buildtriggerbadge, obviously required in the list if you ask me).

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/a29c2bab-1061-4567-9629-a4f3f0b838eb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Baptiste <Batmat> MATHUS - http://batmat.net
Sauvez un arbre,
Mangez un castor !

[Daniel Beck]

On 01.10.2015, at 15:36, Victor Martinez <victormar...@gmail.com> wrote:

> - Does the plugin integrate well with Jenkins? Does it work well in secured and distributed Jenkins?
> As I already mention "secured" shouldn't pruned any plugins as long as those are heavily used by the community and it is an open source version

I think we'll have to agree to disagree here. If a plugin, without warning you, undermines whatever security the admin sets up, it should not be available in this dialog.

> - Are the features it provides useful to someone fairly new to Jenkins?
> What do you mean about someone fairly new to Jenkins? Configuration as Code (Workflow, JobDSL or jenkins builder) will be fairly complex to understand by someone new to Jenkins, even using you use Workflow, don't you think?Then some of those recommended plugins don't follow this criteria.

You'll notice I left out plugins that, secured or not, are mostly useful when scripting around in Jenkins internals. The kind of plugin that makes people file issues like JENKINS-26416 or JENKINS-24790. For example, Groovy Postbuild and Scriptler fall into this category.

Looking at the list of plugins you're showing me I can say with confidence that this setup dialog is not meant for you, but that's okay -- it's not meant for me either! It is for getting less (or in-)experienced users started with a useful set of features, rather than dumping them into an empty instance that doesn't even have their SCM.

[Victor Martinez]

I really appreciate your feedback again, it is constructive and probably when I started this thread I was not constructive in the way I wanted, so I apologize about it. 

So coming back to those points, I'm giving that feedback according to my own experience, as you can imagine. But I feel somehow the Workflow plugin is being highlighted as recommended plugin IMO, as I mentioned previously Configuration as Code for someone fairly new is likely a 3 years approach (from the standardization point of view) compared to the CI approach which is likely a 15 years approach. In that case that list of plugins  as long as it's only focused in less experienced users with less experimental concepts should be more tiny, and somehow shouldn't "promote" anything hardcore complex even though it's something I like and I use quite often, but being strict about those 4 criteria points. What do you think?

In any case, I don't want to be blocker on this feature, just highlight my concerns

Thanks again

[Tom Fennelly]

I think we all agree that the wizard should have a trimmed down list of "suggested" plugins, where the "suggested" plugins are a subset of "all" plugins and the "recommended" plugins are a subset of the "suggested" plugins:

• "all" (> 1000 plugins)
• "suggested" (< 100 plugins atm)
• "recommended" (< 20 plugins atm)

I'd hope all would agree that having the wizard display "all" plugins (by default) is just plain nuts :) That's not going to help anyone (new or veteran Jenkins user). I'd hope all would agree that we just list/show the "suggested" plugins (by default).

So what do I mean by "by default" ? To me, the scrollable list of "suggested" plugins is most useful to new Jenkins users (and probably even new to CI/CD). I think it's really only new users that will NOT filter the list in some way - simply coz they're not sure what they're looking for and so wouldn't know what to filter on. To me, this is the only group to which the "suggested" list of plugins makes sense. Most other users will very likely know what they're looking  and so will filter the list Vs scrolling all the suggested plugins (~100 of).

I think once the user starts applying a filter to the selection (by typing in the search box) then we could actually filter on "all" Jenkins plugins (all 1000+ of them). In other words, we can think of the "suggested" plugins as just being the "default" filtered list.