Revisiting bundled plugins

[Kohsuke Kawaguchi]

I'm coming back to age old discussions of JENKINS-9598.

If you look at the out of the box experience of Jenkins today, it is really dated and not ideal. We have CVS and Subversion bundled out of the box, which is much less commonly used than before. We have some other very commonly used plugins, such as git plugin, parameterized-trigger plugin, envinject plugin, and so on not available out of the box.

In addition, the critical new subsystem like workflow plugins are not made available out of the box to users, despite the fact that the industry is shifting from old days of build & test automation to continuous delivery.

For many of us in this list who know Jenkins inside out, this is not a problem --- the first thing I do when I start a new Jenkins is to go to plugin manager and install a whole bunch of plugins. How hard can that be, you might ask.

But stop for a second and think about the fact that in the past 12 months, we've added more than 30,000 installations. More and more new users are coming to Jenkins. Many of the admins of those 30,000 installations each had to learn what plugins are useful, which of the dozen "git" plugin is necessary, and discover that the workflow plugin is the way forward for writing a complex orchestration.

I think we are creating less than steller experience for those users. It creates a wrong perception, and makes the barrier of entry harder.

Back then we talked about this (and you can see some comment in JENKINS-9598 as well as the meeting minutes), we have collectively felt that we don't just keep on adding more bundled plugins, and that the proper solution is to ship a lean jenkins.war that has a better setup wizard like experience, and among other things (such as some initial system config setup of key parameters), it'd help people install the right set of plugins.

JENKINS-9598 was filed 4 years ago, and that meeting discussion happened 2 years ago. I think it's safe to say as a community we have failed to address this problem. It is understandable because this is not a pain that many of the core contributors feel (and so we put more weight on downsides like download size increase.) I'm not trying to argue that the consensus built in JENKINS-9598 is wrong, but I feel that we are letting perfect get in the way of better.

So this is a proposal to make one time reshuffling of the plugins that Jenkins bundles out of the box. We add the following plugins and their dependencies:

• git
• parameterized-trigger
• workflow

Then remove the following plugins:

• cvs (split from core in 1.340, 929KB)
• ant (split from core in 1.430, 90KB)
• maven-plugin (split from core in 1.296, 11MB)

This is a first step of improving the out of the box experience, and this journey will include a better setup wizard down the road, at which point we will get rid of the bundled plugins more or less entirely. This change will also not jeopardize the setup wizard change down the road. Unlike those plugins that were split off from the core, plugins that were born outside core can be unbundled any time at a later point without any backward compatibility consequences.

The proposed change includes some plugin removals. This has a backward compatibility implication --- when we unbundle plugin that was split from core at 1.X, people upgrading directly from version <1.X will see a loss of functionality until s/he installs the plugin from plugin manager. S/he can avoid this problem by first upgrading to the version of Jenkins that bundles them as a plugin (say current LTS 1.609) instead of going straight to the latest. So the damage is limited. Given that and the fact that 1.430 is released 4 years ago, I think this compatibility implication is very minor that only affects a very small set of people.

I've added maven-plugin in the chopping block because it is by far the biggest plugin coming in at 11MB. Some of the concerns people raised in JENKINS-9598 was the increased download size. I'm worried a lot less about it than the out of the box experience, but nonetheless I tried to be considerate for those people. Removing 11MB from jenkins.war would offset the size increase coming from newly bundled plugins.

Any objections, thoughts, feedbacks?

--

Kohsuke Kawaguchi

[Kanstantsin Shautsou]

Many times before some starting guide was discussed. IMHO better create starter-plugin that will provide some groups of plugins as Java/PHP/etc packs so user can easily choose what install.

Or probably provide predefined sections like SCM that will dynamically provide i.e. 3 top installed scm plugins (i guess it will be git, subversion) for installation, 4 flow plugins (and here workflow will appear) and etc. for fast installation.

IMHO it more safe to bundle button plugins that provides buttons for core functionality that wasn't implemented and not accepted into core code like PollSCM, SafeRestart and etc.

I don't think that core needs ship any bundled plugins, especially workflow that is not stable and has no UI for end-users.

Also less plugins less core size that is good for container images and jenkins stability.

[Kanstantsin Shautsou]

+ Folders Plugin is good candidate as grouping entity for jobs hierarchy.

[Richard Bywater]

I'm not really sure what the difference is between a bundled and non-bundled plugin (is it just that the same plugin but stored in the WAR file rather than a standalone file?)

Assuming it was more or less just a matter of adding plugins into the WAR file, then I'd be in favour of having two flavours of WAR file - a bare-bones Jenkins WAR which doesn't have any additional plugins bundled and a "Starter" Jenkins WAR file which is along the lines of what you have in your proposal.

Would that be worthwhile or would the difference in size between those two versions of WARs be minimal enough not to warrant any effort in that space?

Richard.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAN4CQ4zHznf-0bTPOgjBv29EFWRbcYTidB7M7uGWgZ4xO-sh9A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Arnaud Héritier]

I agree about this need to remove as such as stuffs as possible by default and to offer in counter part a better classification and organisation of plugins

I would like to see a wizard like Intellij Idea has nowadays:

* Which SCM will you use ?

* Which langages will you use ?

* Which build tools will you use ?

* .....

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAN4CQ4zHznf-0bTPOgjBv29EFWRbcYTidB7M7uGWgZ4xO-sh9A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

-----

Arnaud Héritier

http://aheritier.net

Mail/GTalk: aheritier AT gmail DOT com

Twitter/Skype : aheritier

[Arnaud Héritier]

Ohhhh and I forgot a point: Let's fix the nightmare created by pinned plugins at the same time.

It is a concept really difficult to understand for users.

[Arnaud Héritier]

And if it is for next Xmas: Don't forget all users who are deploying Jenkins in a environment with no internet (or a limited access). Thus if we could pack together some set of plugins (and required dependencies) and allow jenkins to "eat" them. Deploying plugins one by one is a nightmare and error prone

[John Tatum]

One of my sore points with Jenkins has always been the included plugins. Personally, I would rather see nothing included. The whole point of plugins is to have a modular system, so including things out of the box just ends up being more stuff users have to get rid of on a clean install because they do not use/need/want it.

I am all for stripping plugins out, but I think including plugins is going the wrong direction. This is especially true for things like build tools and SCM clients because in many cases there is additional software that needs to be installed on the system for the plugin to be useful. It makes no sense to me to include a plugin that will not work out of the box.

My next point against including plugins is that basing what plugins are included on the flavor of the month (in terms of the tools available) inherently means that the Jenkins project would be following trends and fads in development. What happens when NewSCM is just as popular as Git? Do we now include two plugins out of the box?

It also means assumptions about how users are handling the FotM. For example, many Git hosts and servers provide a Subversion bridge, so just because somebody is using Git for SCM does not mean they are using the Git client.

I will second Arnaud's assessment of pinned plugins. I think they are useful to somebody who understands them(under the right circumstances), but that is offset by being entirely unintuitive.

In my opinion, Jenkins should be as un-opinionated and tool agnostic as possible. In my experience, this is the source of a great deal of Jenkins' appeal.

As far as a wizard goes, I do not know how useful it would really be. There are some config/settings changes (ie, workspace location, listening port, the option to configure some security settings) that would be convenient to be made through a wizard, but I do not think there are enough to justify it.

IMO, Jenkins is very user friendly. Are there things that could be improved? Absolutely, but at the end of the day I can think of quite a few things that are far more important to me, as a user, than what plugins are bundled with the .war.


John Tatum
john....@live.com
http://scientifichooliganism.net

Although this e-mail and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and/or upon which it is opened, it is the responsibility of the recipient(s) to ensure that it is virus and/or defect free and John Tatum bears no responsibility for any loss and/or damage arising in any way from its use.

---

From: aher...@gmail.com
Date: Wed, 5 Aug 2015 22:53:33 +0200
Subject: Re: Revisiting bundled plugins
To: jenkin...@googlegroups.com

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFNCU--e6HMkRVRO3p1MQCCCdFkkrQKeNLD0vbCAv%3D2TU1qjdw%40mail.gmail.com.

[Jesse Glick]

On Wed, Aug 5, 2015 at 1:56 AM, Kohsuke Kawaguchi <k...@kohsuke.org> wrote:
> We add the following plugins and their dependencies:
>
> git
> parameterized-trigger
> workflow

-1, no new bundled plugins.

> Then remove the following plugins:
>
> cvs (split from core in 1.340, 929KB)
> ant (split from core in 1.430, 90KB)
> maven-plugin (split from core in 1.296, 11MB)

+1. While you are at it, get rid of external-monitor-job which I think
few people use.

[Vincent Latombe]

I agree with Jesse, bundling new plugins is just pushing back the problem. What we really need is a better first time experience and I think Arnaud's wizard proposal makes sense.

You could add windows-slaves to this list.

 

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr30OJnOB%2BAfeTJPpPXk4gCSW-HuCyEdW-fF%3DGwphcgHVA%40mail.gmail.com.

[Rafael Ribeiro Rezende]

I work in an industry which is inherently conservative about adding new stuff into their environments. So, I honestly think that this question might not affect only newcomers.

Since we started using Jenkins we have set up our own plugin hub for internal plugins (with backend-update-center) and allowed direct access the default public repository as well. Soon people started questioning the credibility of the public plugins, and some suggested to create a sort of moderation to filter "unverified" plugins. There are several reasons for concern, but what's my point here? Most people take the bundled plugins as reliable. Some plugins would not be accepted in some enterprises if they weren't "native" features. Of course, the rule works in the other direction as well. If the bundled plugin gives a bad experience, the whole tool feels like a bad experience.
Personally I don't know in which side I am. There are cons and pros. In any case, whatever the butler serves should be very well served! Plugins come with responsibility.

One feature that could leverage the credibility issue is to have some sort of ratings or so right in the plugin manager view, or "Verified" labels maybe. Anyway, this is another topic...

[Stephen Connolly]

I agree that adding bundled plugins just makes life more complex.

In my view Jenkins would come bare bones empty... BUT you could upload a zip of plugins and Jenkins would install all required dependencies from that zip.

That way, if I am in an isolated network I just need the zip and the war to be downloaded. 

People could provide a service to generate such war files on demand based on a plugin selection for example.

If Jenkins already has a newer version of one of the plugins in the war, then that would be left as is.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAH-zGCj6%3DBA9%3DG-m1OHF-nk8g1_yyqZvUVJjuesiQ3vjaVirZw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Sent from my phone

[Richard Bywater]

+1 - sounds a great idea to me

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMxHKwh_uNAPcmj77Zq01HYs7KPHTrN1J78mG%3DG%3DJN2bwQ%40mail.gmail.com.

[oliver gondža]

I would love to see bundled/pinned plugins to go away and removing some
bundled plugins without introducing new ones sounds like a good plan to me.

What I would like to see implemented eventually is a support for custom
plugin collections: Simple list of plugins (with minimal versions,
perhaps) to be proposed to user and installed with one click distributed
online as a single file / base64 encoded string. Everybody, can easily
share what they use: "Hey, check out my .NET development tooling!".

With that in place, Jenkins can be distributed with simple 'default'
collection preconfigured that can change in time following current trends
/ user needs. The same way user is invited to create first job, we will
invite users that have not installed any plugin so far to install
recommended plugins. Admittedly, this is a step towards setup wizard...

--
oliver

[Kohsuke Kawaguchi]

The wizard that you describe is quite in line with what's discussed in JENKINS-9598.

However, note that at this time I'm specifically limiting the scope of the request to make a tactical shuffling of bundled plugins.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFNCU-9i_B44EjTAvvJHPYNZd4OdSF8k2iXzFNB46fUwzRRVog%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Kohsuke Kawaguchi

[Kohsuke Kawaguchi]

2015-08-07 0:12 GMT-07:00 Vincent Latombe <vincent...@gmail.com>:

+1. While you are at it, get rid of external-monitor-job which I think
few people use.

You could add windows-slaves to this list.

external-monitor-job was split off from core in 1.467 and windows-slaves is 1.547. Those are too new to my taste to consider removal IMHO.

As a recap, when a plugin is split from core in version X, people upgrading from a version earlier will experiene functionality loss.

--

Kohsuke Kawaguchi

[Kohsuke Kawaguchi]

I think it's a mistake if we just stop bundling plugins all together without adding something else to offset the lost new user experience, like a setup wizard.

We don't guide users toward plugin manager, and so when the first time user don't see his SCM of choice in the config UI, it adds to the learning curve that he has to go to plugin manager. And when you type "git" in there there are 30+ plugins that show up.

Git still has one thing going; that is, at least it's very clear what keyword you need to search. Parameterized trigger plugin and workflow has this even bigger problem that new users would have no clue that they need to be looking for them. They have to learn about the existence of that feature somewhere else first, adding to even bigger learning curve.

And none of the plugins I'm proposing is a "flavor of the month." Git has been very popular quite some time, and the parameterized trigger plugin has a long history. workflow is strategically important for Jenkins. If I start proposing we change what we bundle every month, that is a problem, but this is the first time ever I propose this in the 10 years of this project.

Finally, I know this is not something seasoned Jenkins users will care about. They know about the plugin manager, they know the "plugin list" Wiki page, and perhaps they might even know about @jenkins_release to learn about new plugins. And for them, the effort we spend on the setup wizard is an effort we are not spending on other things they do care.

But really we are doing this to improve the experience for new users, and we all need to understand that that is an important thing for the project, too.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/BLU179-W79DC5537065F17911647C5F7740%40phx.gbl.

For more options, visit https://groups.google.com/d/optout.

--

Kohsuke Kawaguchi

[Slide]

I really doubt that someone new to Jenkins will want to try and use the workflow plugin. Getting a simple Maven or Free Style job setup is enough for most initial users. I would say that intermediate to expert users are the ones that you should be guiding to workflow. Workflow seems to complex for someone new to the whole thing.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAN4CQ4xqEynupsJ94GeHRDhoQc0cgJUhOS4KTgMA3oHxPPSp%3DQ%40mail.gmail.com.

[Kohsuke Kawaguchi]

Thanks for all the comments.

I started responding to comments one by one, but I think it's just going to end up dilluting the conversation, so instead let me explain where I am coming from for one more time, in the hope that it will frame the discussion in the way I want.

A number of you have said they'd rather see bundles plugins reduced or go away entirely. As an eventual solution, no one is arguing against that. I'm just making an observation that that is not a trivial work and it'll take some time to get there.

Updating the bundled plugin is a short-term temporary fix until then. When we get to the setup wizard, we can stop bundling plugins, and we are all in the happy state. What I'm trying to do is to buy us some additional cheap "new user happiness" (NUH) until that comes in.

IOW, if you think of NUH as a function over time, I'm just trying to lift this function up a little bit over a certain time period until we add the setup wizard. The orange region below. It's a very small effort that buys us some NUH that we can do right now.

The question isn't whether the setup wizard is better than what I'm proposing here. Most of us agree on that one. The question is why we shouldn't buy this orange area of NUH.

Some people have suggested that we should distribute core w/o plugins or come up with some other means to help people create/share/install a group of plugins. No doubt that's a useful feature, but giving more choice to users does not help NUH. Choosing requires an effort, and we already provide a lot of choices. What we lack is the reasonable combination of plugins we endorse that creates a good first experience, until they get more comfortable with Jenkins and start tinkering with their preferred plugin set on their own.

Think of Subway the sandwich shop. You can specify exactly what you want down to every ingredient, but most people don't do that because it'll take too much conscious efforts to choose. Instead, we go for one of their pre-built recipes, like Roast Beef sandwich, and we assume they did a reasonable job putting together a reasonable combination that tastes good.

I'm just trying to do the same thing here.

--

Kohsuke Kawaguchi

[Kanstantsin Shautsou]

From first days when i started using jenkins this bundled plugins were a pain. 

Also all bundled plugins has irrelevant statistics, atm job dsl and build flow plugins has more installations rather than workflow. After you bundle it nobody will see a clear situation. Git plugin also has many sensitive issues for production usages and if it will restore upgraded versions to bundled - it will be disaster for end-users. 

I can also draw the graph:

[Oleg Nenashev]

As an OSS contributor, I'm -1 for any new bundled plugins excepting the decoupling from the Jenkins core.

суббота, 15 августа 2015 г., 4:29:38 UTC+3 пользователь Kanstantsin Shautsou написал:

[Jesse Glick]

On Fri, Aug 14, 2015 at 5:51 PM, Kohsuke Kawaguchi <k...@kohsuke.org> wrote:
> when a plugin is split from core in version X, people upgrading
> from a version earlier will experiene functionality loss.

Well I think this is what we should be spending a little effort
fixing. IMO there be *no* bundled plugins, so a new installation
(virgin `$JENKINS_HOME`) has no enabled plugins. The question is how
to deal with upgrades. We are already tracking split plugins by
version

https://github.com/jenkinsci/jenkins/blob/fd83a285b3b1560ba19122ff33edc522de7027f0/core/src/main/java/hudson/ClassicPluginStrategy.java#L324-L337

but so far we only use this for implicit dependencies. We also track
last-saved version, a close proxy for last-used version

https://github.com/jenkinsci/jenkins/blob/fd83a285b3b1560ba19122ff33edc522de7027f0/core/src/main/java/jenkins/model/Jenkins.java#L318-L330

so we should able to determine which split plugins the user expected
to be installed. So we should install those, and only those, during
the first startup after the upgrade:

1. We could try to load them from the update center. This would be
vulnerable to dropped Internet connections and so on, however.

2. Or, Stephen Connolly wrote a system allowing plugins to be bundled
as “optional”, and only conditionally enabled during startup. So the
WAR download size would remain large, but you would not see plugins
enabled unless you were upgrading from a version before they were
split.

With this fixed, the problem reduces to helping novice users pick
reasonable plugins.

[Jesse Glick]

On Fri, Aug 14, 2015 at 6:20 PM, Kohsuke Kawaguchi <k...@kohsuke.org> wrote:
> when you type "git" in [the plugin manager] there are 30+ plugins that show up.

It has already been proposed that we introduce an API for adding
columns to Plugin Manager. This should be pretty easy. Then we can add
a column showing known installation count, which we already track
somewhere because the wiki shows it. And then either hide plugins with
(say) <1000 installations by default, or bubble up the most-installed
plugins to the top when using the Filter field, etc. That provides a
straightforward, organic solution to the issue you bring up, without
needing to haggle over whether individual plugins are sufficiently
worthy.

[Stephen Connolly]

https://github.com/jenkinsci/optional-plugin-helper-module 

is the module for anyone who is interested.

 

With this fixed, the problem reduces to helping novice users pick
reasonable plugins.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr09ek0W5HhxHf6Vo2a4zcVSq-tbMYLBD9Pyf6HYr_5Xqw%40mail.gmail.com.

[Tom Fennelly]

On Monday, August 17, 2015 at 9:11:25 PM UTC+1, Jesse Glick wrote:

On Fri, Aug 14, 2015 at 5:51 PM, Kohsuke Kawaguchi <k...@kohsuke.org> wrote:
> when a plugin is split from core in version X, people upgrading
> from a version earlier will experiene functionality loss.

Well I think this is what we should be spending a little effort
fixing. IMO there be *no* bundled plugins, so a new installation
(virgin `$JENKINS_HOME`) has no enabled plugins. The question is how
to deal with upgrades. 

I tried to distill my understanding of what Jesse described in a diagram ala ...

 Obviously it's high level and doesn't go into detail on e.g. how a plugin selection wizard might actually work + doesn't detail how split plugins are actually resolved.

[Tom Fennelly]

Something I was wondering in order to help with upgrades ... could we use a simple plugin that needs to be installed that captures and records all split plugin info from the instance. This info can then be used on the subsequent install so as to install the exact versions of split plugins. An upgrade would be aborted if that info was not present during startup. Or is there a better way or existing info that can be used?

[Tom Fennelly]

On Tuesday, August 18, 2015 at 10:01:28 PM UTC+1, Tom Fennelly wrote:

Something I was wondering in order to help with upgrades ... could we use a simple plugin that needs to be installed that captures and records all split plugin info from the instance. This info can then be used on the subsequent install so as to install the exact versions of split plugins. An upgrade would be aborted if that info was not present during startup. Or is there a better way or existing info that can be used?

Prob not practical where there's a lot of Masters? 

[domi]

I don't have a solution for this, but to be honest, I think this is just way to complicated and just screams for issues in the long run. We should go for less magic and more transparency - I think we should just inform the users about this changes and not try to heal the world. We can also inform the users with an info message right within Jenkins after he updated to a version where no plugins are bundled anymore.

...just my 2cents...

/Domi

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/35c09963-21aa-450a-8f47-4e87bb5015d9%40googlegroups.com.

[Tom Fennelly]

Hi Domi.

When you say "I don't have a solution for this", I think you are referring specifically to the issues around upgrades and split plugins, right?

Wrt split plugins, I think the problem for a user doing an upgrade would be that they might not know exactly which split plugins they now need to install (that they are now missing), having done an upgrade. We could give them a message/warning telling them to "install split plugins" (or however it would be worded), but it probably wouldn't be a lot of help to them. I'd imagine it might be a "What the F***" moment for them.

T.

[Jesse Glick]

On Tue, Aug 18, 2015 at 5:01 PM, Tom Fennelly <tom.fe...@gmail.com> wrote:
> could we use a simple plugin that needs to be installed that captures and records all split
> plugin info from the instance. This info can then be used on the subsequent
> install so as to install the exact versions of split plugins. An upgrade
> would be aborted if that info was not present during startup. Or is there a
> better way or existing info that can be used?

I thought I already proposed everything that was needed here. You know
the version of Jenkins that was running before the upgrade (with the
caveat below). From that and the static list in
`ClassicPluginStrategy` you can compute which plugins were part of
core before the upgrade and have since been split off. Those are the
ones that must be installed now to prevent a regression.

Any plugins that were previously bundled will still be there in
`$JENKINS_HOME/plugins/*.jpi` so those will still be loaded, unless
the user disabled them.

The caveat: you actually know the last version in which global
configuration was saved (if 1.301+), which is probably close to the
last version run. If they differ, the last-saved version was very
likely older than the last-run version—since downgrading core does not
generally work anyway—so at worst we wind up listing a plugin which
was split in between those two versions. But such a plugin would have
been bundled in the later version, and thus already in your plugins
directory, so we can simply skip over it.

So if the user runs with a fresh `$JENKINS_HOME`, they get zero
plugins, but we can add a wizard or whatever UI to make
recommendations.

If on the other hand they last ran 1.532.x, they would have had some
bundled plugins, including the splits prior to that point (…, `ldap`,
`pam-auth`, `mailer`). They might have upgraded or disabled some of
them. After the upgrade, the plugins they were running before remain
in the same state, except that previously bundled plugins are just
plain old installed plugins now (any `*.jpi.pinned` markers will just
be ignored). Furthermore, during startup we automatically install
`matrix-auth`, `windows-slaves`, `antisamy-markup-formatter`,
`matrix-project`, and `junit` (getting these from
`WEB-INF/optional-plugins/*.hpi` where the module Stephen mentioned
looks for them): again, just as plain old installed plugins, which the
user could subsequently choose to disable or even uninstall
completely. Thereafter Jenkins will not touch your plugin list, unless
we split another plugin from core and you upgrade past the split
point: to make the upgrade as transparent as possible, we again
silently install that plugin.

`jenkins.war` would get slightly smaller, as the few plugins which
were previously bundled but not split or a dependency of a split (I
think just `ssh-slaves` and `translation`?) are removed. As features
are refactored into split plugins it should stay about the same size.
We could consider removing some very old and bulky splits (like
`maven-plugin` and `subversion`, the two oldest and bulkiest!) to make
the WAR 21% smaller; people upgrading directly from Hudson (!) 1.309
or earlier could still have these plugins automatically installed
during startup so long as they were connected to the Internet.

Anyway, this is all just a technical proposal, contingent on agreement
with the basic goal of not having bundled plugins any more. I did not
get the impression that Kohsuke is really on board with that.

[Tom Fennelly]

On Wednesday, August 19, 2015 at 1:55:58 PM UTC+1, Jesse Glick wrote:

I thought I already proposed everything that was needed here.

You did for sure. Sorry about that. I should not post late at night.

The caveat: you actually know the last version in which global
configuration was saved (if 1.301+), which is probably close to the
last version run. If they differ, the last-saved version was very
likely older than the last-run version—since downgrading core does not
generally work anyway—so at worst we wind up listing a plugin which
was split in between those two versions. But such a plugin would have
been bundled in the later version, and thus already in your plugins
directory, so we can simply skip over it.

Where this might cause frustration is if an earlier split plugin was uninstalled/removed and we end up installing it again, thinking it was split since the last version (as we know it). Not the end of the world though (I guess?).

[Jesse Glick]

On Wed, Aug 19, 2015 at 9:52 AM, Tom Fennelly <tom.fe...@gmail.com> wrote:
> I should not post late at night.

I have been guilty of quite rude late-night posts. :-)

>> The caveat: you actually know the last version in which global
>> configuration was saved (if 1.301+), which is probably close to the
>> last version run. If they differ, the last-saved version was very
>> likely older than the last-run version—since downgrading core does not
>> generally work anyway—so at worst we wind up listing a plugin which
>> was split in between those two versions. But such a plugin would have
>> been bundled in the later version, and thus already in your plugins
>> directory, so we can simply skip over it.
>
> Where this might cause frustration is if an earlier split plugin was
> uninstalled/removed and we end up installing it again, thinking it was split
> since the last version (as we know it)

Assuming the user saves global configuration at least occasionally (or
we could also force a save after changing plugins etc.), this would
not occur since the new core upgrade interval would not include the
split point.

[Kohsuke Kawaguchi]

The procedure of core deciding when and what split plugins to install looks good. My take on Jenkins.version field is bit different that I think there will be a lot of people who haven't touched the system config for quite some time, which means it cannot be used as a reliable indication of what version was running immediately before the upgrade. But it does tell you what version have run in the past, so it's still useful as a conservative indicator, and if we have to err between breaking people's upgrade by not installing the right split plugin vs installing extra plugins, we'd be obviously choosing the latter.

So the end result is the same as what we are all saying.

2015-08-19 5:55 GMT-07:00 Jesse Glick <jgl...@cloudbees.com>:

Anyway, this is all just a technical proposal, contingent on agreement
with the basic goal of not having bundled plugins any more. I did not
get the impression that Kohsuke is really on board with that.

We have to do the "plugin selection wizard" in Tom's picture (aka the proper solution of JENKINS-9598) first before removing bundled plugins. If we do the latter first without former, new user experience gets even worse. I believe that's a mistake.

--

Kohsuke Kawaguchi

[Tom Fennelly]

On 19 August 2015 at 16:52, Kohsuke Kawaguchi <k...@kohsuke.org> wrote:

2015-08-19 5:55 GMT-07:00 Jesse Glick <jgl...@cloudbees.com>:

Anyway, this is all just a technical proposal, contingent on agreement
with the basic goal of not having bundled plugins any more. I did not
get the impression that Kohsuke is really on board with that.

We have to do the "plugin selection wizard" in Tom's picture (aka the proper solution of JENKINS-9598) first before removing bundled plugins. If we do the latter first without former, new user experience gets even worse. I believe that's a mistake.

--

Kohsuke Kawaguchi

Right, you don't want to remove the bundled plugins without providing an easy way for people to, on a first run, select an initial set of plugins. Forcing a new user to go to the plugin manager before having a Jenkins instance that can do anything would be a horrible first experience of Jenkins. So, removing bundled plugins + still having a nice first experience for users should be top priority.

As for the issue with the persistence of the version number, I think we can make that problem go away over time by storing that version somewhere else (as opposed to in the global config) and making sure it gets persisted in a reliable way (or just making sure the global config gets persisted after every startup, if that's not a problem).

​

Also, I updated that flow diagram based on feedback from Daniel Beck. Some timing “issues” are not represented in this diagram. For example, the “Install” and “Upgrade” execution paths would probably need to happen at different times in the Jenkins startup sequence (e.g. before or after jobs are loaded).

[Tom Fennelly]

Gus Reiber is going to draw some screens for what a "Plugin Selection Wizard" might look like, allowing us to have a conversation about that.

[Gus Reiber]

If I am following all this correctly, we have general agreement that Jenkins should stop bundling plugins all together, yeah?

...but some technical ifs and buts about upgrading (Jesse, Stephen, KK, and Tom can solve, but should not use "pinning" to do so)?

Then to the extent we do want people to start using Jenkins with *some* plugins (this might still be a bit contentious, maybe we do want to start with 0 plugins), we want to put the most minimal UI possible in the front of a Jenkins initial setup that lets users pick those plugins, yeah?

...just want to make sure I am on the same page as everyone.

[Sacha Labourey]

While I understand the engineering-idealism of a micro-kernel type architecture with dynamically loaded plugins where we end up packaging "nothing" outside of the kernel, the danger with that view is one of user experience. This view that since everybody is different, we might as well only provide just the core and let each decide for themselves, doesn't lead to a good experience. This is "design by exception". IMO, it is much better to:

• Provide a default user experience with pre-loaded/packaged plugins that satisfy even 50% of the users (guide them, show them a good/typical way)
• Let them customize this base easily (this is where the current proposal of bundle of plugins, etc. is good IMO)
• Let advanced users, through some kind of manifest, get to the exact-specific set of plugins they need as they move from core to core

i.e.make it easy for new users by default, let advanced users deal with their specific setup.

My perception from this discussion is that we are aiming for the exact opposite i.e. slick and virgin for everybody and hope new users will be able to magically decide what a good average getting started experience might be, this is likely to only satisfy the advanced users. Advanced users should have a way, but it doesn't need to be the default way as they'll always been able to deal with non-default behaviour.

iOS as a pretty nice kernel architecture. But as far as I remember, my iPhone came with pre-installed applications and ringtones :)

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/dc5546de-3716-4222-a457-8195bb6bc6e0%40googlegroups.com.

[Daniel Beck]

On 22.08.2015, at 15:40, Sacha Labourey <sa...@cloudbees.com> wrote:

> My perception from this discussion is that we are aiming for the exact opposite i.e. slick and virgin for everybody and hope new users will be able to magically decide what a good average getting started experience might be, this is likely to only satisfy the advanced users.

The 'slick and virgin' part could be achieved by only using bundled plugins for functionality extracted/detached from core, and not installing them on a new system. Many of the bundled plugins aren't even popular, and almost all of them are really only bundled to not break upgrades, so making them the default set of features for Jenkins doesn't really make sense.

On a new installation (that would otherwise lack a lot of fairly essential features, such as at least one SCM implementation), a setup dialog of some kind could enable new users to quickly get started. Advanced users could skip right past that and use the plugin manager to install what they need. The responses by Kostya, Vincent, Oliver, and Tom seem to aim towards something like this. I suggested basically this last year in JENKINS-9598.

> my iPhone came with pre-installed applications

Illustrates one of the arguments against bundling plugins well: These apps cannot be deleted either if you're not using them. A different way of 'bundling' plugins that would only actually install them when needed to not break the instance on upgrade (and allow deleting the plugin), would complement a 'plugin setup' of some kind well.

AFAICT I don't think you're disagreeing with most responses here -- it's just that the specific approach of bundling a plugin (nobody can opt out) is undesirable, and an alternative approach that would benefit both new users and experienced users would be much preferred.

> • Provide a default user experience with pre-loaded/packaged plugins that satisfy even 50% of the users (guide them, show them a good/typical way)

> • Let them customize this base easily (this is where the current proposal of bundle of plugins, etc. is good IMO)

The first could be achieved through the second with an option to install a recommended set of plugins suitable for most (new) users. Don't know know what you want because you're new? 'Install Recommended Plugins' (and maybe select which SCM) and done.

[Sacha Labourey]

Hello Daniel,

On Sun, Aug 23, 2015 at 12:44 AM, Daniel Beck <m...@beckweb.net> wrote:

On 22.08.2015, at 15:40, Sacha Labourey <sa...@cloudbees.com> wrote:

> My perception from this discussion is that we are aiming for the exact opposite i.e. slick and virgin for everybody and hope new users will be able to magically decide what a good average getting started experience might be, this is likely to only satisfy the advanced users.

The 'slick and virgin' part could be achieved by only using bundled plugins for functionality extracted/detached from core, and not installing them on a new system. Many of the bundled plugins aren't even popular, and almost all of them are really only bundled to not break upgrades, so making them the default set of features for Jenkins doesn't really make sense.

On a new installation (that would otherwise lack a lot of fairly essential features, such as at least one SCM implementation), a setup dialog of some kind could enable new users to quickly get started. Advanced users could skip right past that and use the plugin manager to install what they need. The responses by Kostya, Vincent, Oliver, and Tom seem to aim towards something like this. I suggested basically this last year in JENKINS-9598.

I understand this, but it forces new users to make a choice, when they are likely not equipped to make a choice. So we are forcing a first download, a choice, followed by another download (based on the choice she made), and possibly a restart. That is convoluted to me. 

 

> my iPhone came with pre-installed applications

Illustrates one of the arguments against bundling plugins well: These apps cannot be deleted either if you're not using them. A different way of 'bundling' plugins that would only actually install them when needed to not break the instance on upgrade (and allow deleting the plugin), would complement a 'plugin setup' of some kind well.

Well, you are taking an iOS specific choice (not able to uninstall) to simplify the argument but that's not my point :) My point is that my dad and countless people sure enjoy receiving a phone that just works for the average use. He might not use the Passbook app, for sure, he might need other apps, for sure, but the default setup and user experience was great and made it possible for him to discover more, and become opinionated over time about what he wanted vs. not. Here, we are asking people to be opinionated as they do a first install. Live if upon booting iOS, it would simply start the App Store and tell you "well, just build the specific iPhone that matches your need, it does nothing for now."

 

AFAICT I don't think you're disagreeing with most responses here -- it's just that the specific approach of bundling a plugin (nobody can opt out) is undesirable, and an alternative approach that would benefit both new users and experienced users would be much preferred.

Forcing a two-steps download process is very very odd to me, it makes for a horrible user experience. If you want to offer a jenkins-core download for hardcore user, I think it is great, but it doesn't feel like it should be the default. 

 

> • Provide a default user experience with pre-loaded/packaged plugins that satisfy even 50% of the users (guide them, show them a good/typical way)
> • Let them customize this base easily (this is where the current proposal of bundle of plugins, etc. is good IMO)

The first could be achieved through the second with an option to install a recommended set of plugins suitable for most (new) users. Don't know know what you want because you're new? 'Install Recommended Plugins' (and maybe select which SCM) and done.

Right, two steps :) 

[Kanstantsin Shautsou]

When you are installing any linux distro, you has ability to choose software and it also requires restart. Gradle also has no default plugins enabled. If you want suggest somebody the number of plugins that is good by default for you, then make docker image and share (how Stephen proposed). You can, also, rebundle in your enterprise jenkins anything you want. But, please, don't pollute FOSS jenkins. This thread contains enough feedback from people that really using Jenkins (and not managers) to close this topic, and even more - how increase first usage experience in other ways.

This topic was also rejected by governance meeting.

Sent from my iPad

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/kRobm-cxFw8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFgqX_PB-v4kHOw474D9nSiD3jyPnrLpY-2scusEKNy48enTjA%40mail.gmail.com.

[Gus Reiber]

As I look over this thread, there does seem to be consensus that if we were to change the Jenkins plugin bundle at all, we would want to effectively remove them all from being installed.

We would want to empty that set of bundled plugins not just because the distribution download size might get smaller, but more importantly, because the bundled set is immediately installed, difficult to fully remove, and given special status through version upgrades. Also, pinning is weird.

....that seems like a consensus view, IMO.

That the primary Jenkins distribution should be completely barebones and neither come with any suggestion of what plugins to install nor a quick and flexible way to choose and install them does not seem like a consensus. Rafael's somewhat circumspect position of wanting a recommended list or lists, but not wanting that list forced on him seems closer to the consensus. 

Stephen Connolly then offers a sketch of how these recommended plugin sets could come to life, and that also seems to be close to consensus.

For argument's sake, let's say we were to do the Rafael/Connolly solution and allow 3rd parties to easily create plugin sets.

How would we advertise those plugin sets? How would a new user know they exist? 

Would there be something in the core Jenkins that would let new users know that these sets were available?

(...I mean these as real questions, not gotcha statements...)

...these questions are particularly important if the point of one or more of these sets is meant to facilitate new users getting started with Jenkins.

[Daniel Beck]

On 23.08.2015, at 11:44, Sacha Labourey <sa...@cloudbees.com> wrote:

> I understand this, but it forces new users to make a choice, when they are likely not equipped to make a choice. So we are forcing a first download, a choice, followed by another download (based on the choice she made), and possibly a restart. That is convoluted to me.

A UI like OS X Installer's optional 'Customize' may work here to really reinforce that there needn't be a choice. Just click 'Next' and you get the defaults. And plugin installation doesn't require a restart (except possibly in very rare cases).

That second 'step' could be pretty much hidden behind a progress indicator. It'll just take a few seconds (unless you're behind a proxy and then you're used to configuring all the things). It's not like we're asking the user to download and upload the plugins manually. Alternatively we package whatever is in the default set with the WAR, so it just needs to be extracted for a speedup. But this would imply that the default set definition is also bundled, when I'd prefer it to be downloaded from the primary update site on startup, so it can be updated (as installation of slightly outdated Jenkins releases is common due to LTS).

However, many of the choices you'd get with a reasonable set of default plugins are in fact rather necessary. Are you using Git, Mercurial, Subversion, Perforce, …? Are you using Maven, Ant, Gradle, …? Want integration with GitHub, GitLab, JIRA, HipChat, BitBucket, Gerrit, Docker, Jabber, Redmine, Slack, …? Just installing all of them would be possible, but I remember the instances I've run and how hesitant my coworkers were to change anything because they didn't understand everything they saw on the config page. It can become overwhelming pretty fast.

A large part of Jenkins' appeal is its plugin ecosystem and customizability. I don't think there's a need to hide that from new users; just that the initial selection should be curated to greatly reduce the complexity of the choices they make. Dumping them into today's plugin manager would be bad. But I think everyone is capable of selecting their SCM and build tool from a list of ~8 or so each. This should be a reasonable start for a majority of installations.

[Sacha Labourey]

Thanks for the feedback Daniel. This setup makes the (high) availability (and bandwidth capacity) of the "update" center (the "setup" center in that scenario) a key requirement (center down => no install of Jenkins is possible anymore). 

Cheers,

sacha

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/1C08D80D-4B8A-42DF-AC30-27A2434519EE%40beckweb.net.

[Michael Neale]

+1 on removing those and +1 on at least adding git (I can't imagine a world without git - may just be me?). 

FWIW, for many people the size isn't as much as an issue as it once was. Look at the success of docker (even the official jenkins docker image gets a good lot of downloads), for the convenience of "batteries included" people do tolerate what were once absurdly large downloads (yes, they joke about it). It may be limited to docker though - I don't think a 350MB war fie would go down well ;) 

[Jesse Glick]

On Sun, Aug 23, 2015 at 2:47 PM, Sacha Labourey <sa...@cloudbees.com> wrote:
> This setup makes the (high) availability
> (and bandwidth capacity) of the "update" center (the "setup" center in that
> scenario) a key requirement (center down => no install of Jenkins is
> possible anymore).

http://updates.jenkins-ci.org/ is already mirrored and the servers are
expecting heavy traffic. And with the _current_ WAR you already need
to hit the UC just to get, say, Git support. So I do not think this is
a reason to bloat the standard download.

It is always possible to produce an ISO image of Jenkins with the top
200 plugins preinstalled for overnight mailing of CDs to benighted
regions of the world, such as central Massachusetts. :-)

Also I agree with Daniel that we should not “check by default”
traditionally bundled plugins without a good reason, i.e., other than
compatibility for upgrades of split plugins. How many people really
use `external-monitor-job`, or even know what it does? This is just a
feature that Kohsuke at some point (years ago) decided would be neat
to implement, and happened to put in core. Certainly does not have as
broad appeal as, say, `matrix-project` (currently bundled), or
`cloudbees-folder` (currently not). Yet it clutters everyone’s _New
Item_ screen with a mysterious option.

[Arnaud Héritier]

That's why at the very beginning of the thread I added:

> And if it is for next Xmas: Don't forget all users who are deploying Jenkins in a environment with no internet (or a limited access). Thus if we could pack together some set of plugins (and required dependencies) and allow jenkins to "eat" them. Deploying plugins one by one is a nightmare and error prone

In many environment the security rules are not allowing to connect to the internet and the update-center. If something must be done in what we package or not by default we need to not forget the "no-internet" use-case because even nowadays with few default bundled plugins you cannot do a lot of things if you start your instance from scratch.

Arnaud

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAFgqX_M-2F_7DuupK__vmC%3D4j7To%2BP4t9Q_XcpgZxeBksveJbg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

-----

Arnaud Héritier

http://aheritier.net

Mail/GTalk: aheritier AT gmail DOT com

Twitter/Skype : aheritier

[Daniel Beck]

On 27.08.2015, at 15:19, Arnaud Héritier <aher...@gmail.com> wrote:

> Deploying plugins one by one is a nightmare and error prone

It takes about ten minutes to make your folder of blessed plugin HPIs into an update site.

https://jenkins-ci.org/content/juseppe-custom-update-site-jenkins

Skip the wizard, point the plugin manager to your own site, 'select all' and install. Done.

[Arnaud Héritier]

I didn't know this one. I will give it a try.

Maybe a thing to better promote.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/B0176087-5218-4469-B5F6-C17F98EFAB5A%40beckweb.net.

For more options, visit https://groups.google.com/d/optout.

[Kohsuke Kawaguchi]

2015-08-23 9:42 GMT-04:00 Kanstantsin Shautsou <kanstan...@gmail.com>:

This topic was also rejected by governance meeting.

Nit picking, but factual clarification. The proposal wasn't rejected. I retracted it based on the feedback.

--

Kohsuke Kawaguchi

[Gus Reiber]

Hey all,

   So with the dust settled here a bit, I am hoping to solidify some points of agreement:

1) Jenkins plugin bundling has some issues. Not only is the list of what gets installed no longer optimal, it contains a mix of feature plugins that sometimes are nice, but aren't essential to Jenkins and "detached" plugins that are essential to Jenkins but have been implemented as plugins for quicker updates and ease of implementation. Also, the "Bundled" plugins, as they are today, cannot be uninstalled and are carried over from upgrade to upgrade, making all those no longer optimal feature plugins a pile of bloat and potential instability that present a real annoyance to a lot of Jenkins users today.

2) This can be fixed. A collection of CloudBees employees, particularly Tom Fennelly, wants to go fix these issues such that after the next LTS, no one will be saddled with plugins they don't want, but cannot remove.

3) Of the optional, feature style plugins, it would be nice to be able to start with some set of plugins. To do this right, the user needs some means of selecting this initial starting set of plugins, so we will add a light-weight UI widget to the front of a newly installed Jenkins that will allow the user to make that initial selection.

Tom and I, working through and with Daniel and KK and everyone else who is interested in the community, intend to start working on a PR for the December LTS that will make Jenkins better in the above ways.

Tom has assembled a doc describing the effort here: https://docs.google.com/document/d/1cG7BTk6BWAr-rPTSVa8HrfqKEROACy_SecIfqT3Udk0/edit#

Is there anyone who objects to these basic steps?

...the details of which we are eager to smooth out with your feedback.  

[Jesse Glick]

On Thu, Aug 27, 2015 at 11:37 AM, Gus Reiber <gre...@cloudbees.com> wrote:
> "detached" plugins
> that are essential to Jenkins but have been implemented as plugins for
> quicker updates and ease of implementation.

Split plugins are just plugins, which by definition are not
“essential” to Jenkins. They may be very useful and commonly desired,
or they may be obscure, but you can always disable them. (Even bundled
plugins, though these cannot be _uninstalled_.)

> Of the optional, feature style plugins

I.e., just “plugins”.

I am not trying to split hairs here, but to emphasize again that split
plugins deserve no special status when deciding which plugins ought to
recommended to new users. Each plugin should be evaluated on its
intrinsic merits, not historical inclusion in core sources. Some
examples to illustrate my point:

* `ssh-slaves` (split, currently bundled): used in most larger
installations, probably good to recommend.
* `windows-slaves` (split, currently bundled): useful in some
Windows-based installations but often hard to troubleshoot compared to
JNLP slaves; debatable.
* `external-monitor-job` (split, currently bundled): somewhat exotic
use case, probably best omitted.
* `credentials` (dependency of splits, currently bundled): should be
recommended to all users.
* `git` (not currently bundled): good to recommend to new users, at
least until we have an SCM selection wizard.
* `chromedriver` (not currently bundled): useful for a small number of
users, but too obscure to be recommended.

[Gus Reiber]

Fair enough. 

Daniel is offering the same concerns and it seems the doc isn't clear enough on exactly how we intend to do away with the notion of essential plugins and completely unwind the current notion of bundled plugins. Fixing exactly what we don't like about bundled plugins will be the primary goal of our PR. Daniel is going to help add some clarity to the doc. 

My hope is still that, in general, the consensus here is that we are doing the right thing removing or at least diminishing bundled plugins, and adding a GUI selector so that users can choose which plugins they start using.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/kRobm-cxFw8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1gp9VGUkpw0P%2BKVd3iq7JNaY1%2BrWcZzesaGefTXJUCkA%40mail.gmail.com.

[Daniel Beck]

After discussion with Tom and Gus I changed parts of the chapter "The Jenkins Distribution (jenkins.war)" that were referring to essential plugins, as none of the plugins are required for Jenkins to properly work -- they just extend its functionality.

Jesse (and everyone who read the first revision), could you please take another look?

> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAOcHHXx7%2Bti3ojyqvcTZW4K5pwtHDgR%2BBjTGSZtUsnP2dDeiow%40mail.gmail.com.

[Tom Fennelly]

Hey Daniel, thanks for clarifying. I'm on vacation this week but keen to see us come to some consensus on this. I agree with Gus too ... I think the main thing we need to get agreement on here is on the question of if we unbundle plugins and provide a wizard to make sure new users are not confronted with an unusable Jenkins. I think we can figure out the details of how that should happen (the implementation) in a PR and in conversations, so wouldn't get too hung up on that.

Sent from my iPhone

> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/F397B8F7-7195-4F4A-B3FA-E9E933E0CAAF%40beckweb.net.

[Robert Sandell]

+1

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/785c0090-8f16-437d-9bb7-1aa2496a72a9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Robert Sandell

Software Engineer

CloudBees Inc.

[James Nord]

Yeah of you forget about tracking down the dependencies and downloading those one by one and making sure you get the version compatible with the version of Jenkins you want to run rather than the latest. And you expect new users to do that?!

[Slide]

Wouldn't it be possible to have a war customizer that runs on the website to download a custom war with the plugins you want/need with dependency resolution? I remember some site doing this for eclipse some time ago and it worked very well.

On Fri, Aug 28, 2015 at 4:42 PM James Nord <james...@gmail.com> wrote:

Yeah of you forget about tracking down the dependencies and downloading those one by one and making sure you get the version compatible with the version of Jenkins you want to run rather than the latest.  And you expect new users to do that?!

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/8920bb43-b5be-4b3d-ac3d-c2ebfd24d465%40googlegroups.com.

[James Nord]

That should work, but I don't see how that would work with mirroring. I would rather see something generate a zip containing the plugins and deps being generated that you can upload in one rather than generating new wars, otherwise the poor people that use the native packages will be out in the cold.
Also need to consider security in this as you know what you get from the update center is the correct binary...

[Daniel Beck]

On 29.08.2015, at 01:42, James Nord <james...@gmail.com> wrote:

> Yeah of you forget about tracking down the dependencies and downloading those one by one and making sure you get the version compatible with the version of Jenkins you want to run rather than the latest. And you expect new users to do that?!

I'm not sure what you mean. The setup dialog would of course also resolve dependencies, like plugin manager already does today. And plugin manager also knows whether a plugin present on the update site and its dependencies can be installed on the current Jenkins, so this can be handled similarly (maybe through hiding/removal of the plugin recommendation rather than a red warning note). So what specifically is your concern here?

[Daniel Beck]

On 29.08.2015, at 10:46, James Nord <james...@gmail.com> wrote:

> That should work, but I don't see how that would work with mirroring. I would rather see something generate a zip containing the plugins and deps being generated that you can upload in one rather than generating new wars, otherwise the poor people that use the native packages will be out in the cold.
> Also need to consider security in this as you know what you get from the update center is the correct binary...

The idea here is to make it easier for new users to get started with useful functionality. Right now we bundle a bunch of irrelevant features (e.g. Monitor External Job) and are missing a lot of popular and -- by today's standard -- essential features. (A nice side effect is to get rid of the annoying parts of 'bundling', like no uninstall.)

So while these are all relevant points, it's a completely different issue, only affecting very experienced users in an unusually restrictive environment only. Everyone else just uses the update center.

[James Nord]

Daniel,

So while these are all relevant points, it's a completely different issue, only affecting very experienced users in an unusually restrictive environment only. Everyone else just uses the update center. 

You responded to this when it was explicitly about restrictive environments.

And if it is for next Xmas: Don't forget all users who are deploying Jenkins in a environment with no internet (or a limited access). Thus if we could pack together some set of plugins (and required dependencies) and allow jenkins to "eat" them. Deploying plugins one by one is a nightmare and error prone 

So to say deploying plugins one by one has a workaround - when the workaround will not work in the environment described doesn't a workaround make.

So while these are all relevant points, it's a completely different issue

Fair enough - but you [cs]hould have said that earlier.

 only affecting very experienced users

I disagree with that.  There is no correlation between the security paranoid companies/governments and the experience of the Jenkins user.  Infact you may well find that in restrictive sites you are more likely to get noobies as the install may have to be provided by the IT department who are not developers and have not even seen the UI before.

So this bare-bones jenkins won't do anything for new users - which is a bit of a concern - not even build a job when freestyle gets pulled out of core, and that to me really kind of sucks and leaves a user feeling like the tool is broken - rather than they need to do something.  So that I feel would be a mistake of epic proportion.

A long long time ago I was setting up cruise control, I wasted an awful amount of time over the period of a week and I just couldn't get it to work with my subversion repo and my build tooling.  I then downloaded Jenkins and was running in less than one hour.   Why am I telling you this - because that first impression is golden.  If you remove a %age of users that can't access the update center to get even the simple FreeStyleJob then we are the new cruise control.  Do I have a solution - Yes/No/Maybe...   Eclipse.org  provides "distributions"  you can get the basic IDE and add the plugins you want - or you can get the Java version, or the J2EE version, or the PHP version...   If we have something that has no functionality out of the box - I think we need somthing that also comes with some functionlity in the box - you can still show the wizzard to customize (if you have a internet connection) but if you don;t you have something you can at least experiment with.

[Gus Reiber]

So James, are you then arguing that we should leave the existing plugin bundle alone?

...or are you just saying a remote connection for initial plugin collecting is problematic?

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/kRobm-cxFw8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/92cc1041-695e-4d3c-ae6d-cd53c782dedf%40googlegroups.com.

[Jesse Glick]

On Tue, Sep 1, 2015 at 12:30 PM, James Nord <jn...@cloudbees.com> wrote:
> Eclipse.org
> provides "distributions" you can get the basic IDE and add the plugins you
> want - or you can get the Java version, or the J2EE version, or the PHP
> version...

We can consider that for sure. I think the main objection was the
added effort of creating a matrix of native installers with each of
these distributions.

Also proposed was a downloadable plugin package with simple
instructions for dumping into your `plugins` dir, which would avoid
the combinatorial problem.

[Stephen Connolly]

What I suggested is that you could upload to Jenkins a ZIP of plugins and then that zip could be used as a source of plugins to install. The plugin dependencies could be resolved correctly using the optional "bundled" plugins mechanism I wrote or Jenkins could index the bundle and build local update centre metadata to feed the update centre ui. You could even let people dump a bunch of bundle zips into a directory and scan those to build up an update centre source. 

Then we (community or 3rd parties) just have a service that creates the bundles for you and attaches a signature so that integrity can be verified.

That means that somebody in a restricted environment has two and only two downloads: Jenkins core and the plugin bundle. 

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3GYQY-hxu6t_4fxumZ9MF2HgNA-C%3DGECW3Y%3D054NS_pQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Sent from my phone

[Michael Neale]

He also brought up the eclipse.org style "distributions" or "flavors" which I am sure has been brought up before (I can't recall objections though)

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAOcHHXzFTqgWx-d%2B7u%3Dq2rDgcGOdL6D6kRY5ASn%2BbYyU9B1E%3Dg%40mail.gmail.com.

[Baptiste Mathus]

2015-09-01 20:08 GMT+02:00 Jesse Glick <jgl...@cloudbees.com>:

On Tue, Sep 1, 2015 at 12:30 PM, James Nord <jn...@cloudbees.com> wrote:
> Eclipse.org
> provides "distributions"  you can get the basic IDE and add the plugins you
> want - or you can get the Java version, or the J2EE version, or the PHP
> version...

Well, actually Eclipse is somehow backing away a bit from that model. I think that indeed prepackaged versions leads to big complexity and user frustration since not a lot of people will actually find the exact things you need. 

And technically: downloading + installing additional things may not be as-well tested and multiply combinatorials. Hence writing a /wizard/ (called oomph in Eclipse [1]) that works is currently a ongoing work (and is already working but still needs battle testing I guess).

[1] https://www.eclipse.org/downloads/installer.php

-- Baptiste 

[Daniel Beck]

> On 01.09.2015, at 13:23, Michael Neale <michae...@gmail.com> wrote:
>
> He also brought up the eclipse.org style "distributions" or "flavors" which I am sure has been brought up before (I can't recall objections though)

Variants make life extremely difficult. Remember each release (war) gets packaged for half a dozen platforms. There’s also Docker.

Then there’s the storage space issue — we currently already take down older releases from the mirrors because of storage space limitations.

> What I suggested is that you could upload to Jenkins a ZIP of plugins and then that zip could be used as a source of plugins to install. The plugin dependencies could be resolved correctly using the optional "bundled" plugins mechanism I wrote or Jenkins could index the bundle and build local update centre metadata to feed the update centre ui. You could even let people dump a bunch of bundle zips into a directory and scan those to build up an update centre source.
>
> Then we (community or 3rd parties) just have a service that creates the bundles for you and attaches a signature so that integrity can be verified.
>
> That means that somebody in a restricted environment has two and only two downloads: Jenkins core and the plugin bundle.

Would a script (Powershell or Bash) work as well, that you’d have the user execute?

E.g. have the user select which plugins they'd like, or even select from certain presets. This will result in a command like

wget -q -O - http://foo.jenkins-ci.org/bar/SOMEID | bash

This will then go on to download all plugins and their dependencies. Dump them in /plugins and you're done. Or upload them to Jenkins all at once using something like http://html5demos.com/dnd-upload

[Michael Neale]

I agree, I always dreaded going to eclipse.org and worrying I have just downloaded 300Meg of the wrong thing. 

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/kRobm-cxFw8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANWgJS5UFH3C00BpmbR6fYaZr1f_TSwVX2FE8WhEUhB9PY%2BXbA%40mail.gmail.com.

[Tom Fennelly]

It's been a while since I installed eclipse simply because I hated it and moved away from it. My experience of the litany of download options was the same as Mic's i.e. pissed off after wasting 30 mins waiting on the wrong download to complete.

I think we could engineer multiple options for this and they'd all be wrong for a %age of cases, so arguing one set of %ages over another is just moving the problem somewhere else and seems like a waste of time to me.

What's the easiest option to get started on that works for a high %age of people? Lets get something working and then see how we can evolve it to fill gaps.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAKVMTi4u1Z8POdikNK%3DFrOTQbL8aveiZ02YEZ-2sFwrjdmCiKw%40mail.gmail.com.

[James Nord]

So James, are you then arguing that we should leave the existing plugin bundle alone?

Hell no :)

...or are you just saying a remote connection for initial plugin collecting is problematic?

This could be problematic for a certain type of users who are not experienced jenkins admins and that the current proposal will mean that if they installed fresh today they could at least play with Jenkins, whereas in a years time they would not be able to play with jenkins (no Job types!) and then they could just walk away after the initial download.

There are multiple ways to solve this issue - but if we are no longer bundling anything then I think when that is switched on we need something that can get a simple system up for a newbie in a restrictive environment.

I've mentioned the eclipse bundles, theres downloading a zips by selection of multiple plugins, or you could embed the version of the metadata that the new wizard uses in the war at build time, and this could be used to generate a shell/bash script that sucks down those exact files on a different system...  There is plenty more possibilities for solutions, I just think we need to have one in place such that if there is no internet access we can provide at least some instructions to a new user in order to help them get up and running and evaluate Jenkins (after all - its that first impression that counts!)

[James Nord]

Lets get something working and then see how we can evolve it to fill gaps.

Well I can see the headlines now if you release with that in mind 

    "Jenkins, marketed as the leading CI/CD software, no longer supports JUnit the tool used in 99.9% of all development" <snip>out of the box </snip> <- cos you know journalists like to make good headlines

So the solution doesn't need to be big flashy or even polished, but it needs to provide these users a way to get a system into a state that is no worse than what they would have today (and ideally it should be in a better state) - and it IMO needs to be there at the same time that the new plugin startup wizard thing lands.

[Michael Neale]

Yes the unbundling if things sounds doors sound like a painful idea, but that brings us back to the problem of a consensus of what is bundled.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/6a434c93-65ff-491c-bf2d-0c968dfee10b%40googlegroups.com.

[Stephen Connolly]

For me, the issue is *forced* bundling. If we switch to optional bundling with a wizard to allow selection that seems less evil.

Though the risk them becomes everyone and their mother wanting another plugin in the optionally bundled set and before long Jenkins.war comes with 1100+ plugins optionally bundled.

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAKVMTi5zfcmy0UAhw%3DK1u5kuLkRycxe5xwoqyJz6DNTunDmTwA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Sent from my phone

[Stephen Connolly]

I think one thing that people don't quite grasp is how annoying the current bundling mechanism is:

* if a plugin is bundled, it will be installed automatically.

* you can disable it manually, so it can be "uninstalled" but it will still be in the JENKINS_HOME/plugins directory and show in the installed plugins screen (not a major annoyance)

* you can upgrade/downgrade it by either using the update centre or manually uploading the specific version... When that happens a .pinned file gets created.

* once there is a .pinned file then the plugin will never be overwritten by the bundled one.

So what can happen to your instance?

* you upgrade a bundled plugin last year... Now you upgrade Jenkins and it crashes because the pinned file is causing the older version to remain... And other plugins depend on the newer bundled version.

That is one of the worse ones. Ideally the pinned file should have been including some Jenkins version info so that when upgrading Jenkins the pinned plugins can be unpinned if necessary.

However it seems hard to grandfather in that concept now... In any case there is the question of what plugins should be bundled... If any...

I think we could create two tiers of plugins:

* plugins that make a commitment to be compatible with "current" Jenkins releases, cutting releases as necessary and more importantly, testing compatiblity

* everything else.

Bundled plugins could only come fr the first tier... Perhaps with some other criteria to select applicability 

[Tom Fennelly]

On 2 September 2015 at 15:04, James Nord <jn...@cloudbees.com> wrote:

Lets get something working and then see how we can evolve it to fill gaps.

Well I can see the headlines now if you release with that in mind 

    "Jenkins, marketed as the leading CI/CD software, no longer supports JUnit the tool used in 99.9% of all development" <snip>out of the box </snip> <- cos you know journalists like to make good headlines

Oh come on James, this is so overly dramatic :) There'd be no shortage of Jenkins headlines if a Journalist was so motivated and I think this would be way down the list.

So how big an issue is this do you think:

1. What percentage of Jenkins install locations do you expect have no internet connection?
2. Of the percentage from #1, what percentage do you percentage do you think would involve a total newbie to Jenkins, such that they would not be capable of working the kinds of tactics normally employed in these kinds of situations (whatever they are - pre-defined archives, Juseppe etc) ?

Seems to me like this conversation has focused in on something that's only ever going to be a problem for a fairly small group of people. It also seems to me that, for the situation we are talking about here (no internet etc), we should be talking about people that know what they're at (not Jenkins newbies). If we can't make that assumption then I don't see how we can make the assumption that these same people with limited skills are going to be able to work any of the solutions floated above (scripts to run on other systems etc).

For version 1 of this, can't we just document some possible solutions to this problem and point the user to these solutions from inside the wizard on detecting the fact that there's no internet connection? This seems easy enough and we can test it over a period of time. If it's a big issue we'll at least have some real data to work with.

[Antonio Muñiz]

+1 Tom.

That's common sense, It's always useful :)

That percentage of people who try to install Jenkins offline, use to know what they are doing, so a link to "Tips to install Jenkins offline" should be enough.

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BbPaoKO%2BqYJ6eYZqbLcz3jX2gH0CVX%2BwZeMnyob3%3DPn%2BiqmDw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Antonio Muñiz

Software Engineer
CloudBees, Inc.

[James Nord]

On Wednesday, September 2, 2015 at 8:17:44 PM UTC+1, Stephen Connolly wrote:

I think one thing that people don't quite grasp is how annoying the current bundling mechanism is:

* once there is a .pinned file then the plugin will never be overwritten by the bundled one.

That was fixed [JENKINS-24046]- if the bundled plugin is newer than the pinned one then it will be updated. 

[James Nord]

On Wednesday, September 2, 2015 at 9:57:28 PM UTC+1, Tom Fennelly wrote:

On 2 September 2015 at 15:04, James Nord <jn...@cloudbees.com> wrote:

Lets get something working and then see how we can evolve it to fill gaps.

Well I can see the headlines now if you release with that in mind 

    "Jenkins, marketed as the leading CI/CD software, no longer supports JUnit the tool used in 99.9% of all development" <snip>out of the box </snip> <- cos you know journalists like to make good headlines

Oh come on James, this is so overly dramatic :)

Yes it was intended to be:)

 

1. What percentage of Jenkins install locations do you expect have no internet connection?

I don't know - we don't collect statistics from those types of machines :-/

1. Of the percentage from #1, what percentage do you percentage do you think would involve a total newbie to Jenkins, such that they would not be capable of working the kinds of tactics normally employed in these kinds of situations (whatever they are - pre-defined archives, Juseppe etc) ?

I would think the percentage would be high - but thats a gut feeling based on nothing...

 

For version 1 of this, can't we just document some possible solutions to this problem and point the user to these solutions from inside the wizard on detecting the fact that there's no internet connection? This seems easy enough and we can test it over a period of time. If it's a big issue we'll at least have some real data to work with.

For v1 all I am saying is we need to have a solution to the problem (not what that solution needs to be).  If the chosen solution is pointing users to a document that explains some workarounds (possibly provides links to scripts or juseppe etc) then a solution exists and I am happy.

If the solution is telling users they need an internet connection then I am not happy (as that's not a solution).

 

[Michael Neale]

I have worked in the past on software for facilities that were truly offline (military, literal ships in the navy etc) and they have a lot of their own techniques for bundling software as even off the shelf things that claim to work offline dont. For a class of advanced user I don't think this would be a problem at least. 

--

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/kRobm-cxFw8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/9a802d09-be04-4b07-9f21-6404870c9b21%40googlegroups.com.

[Daniel Beck]

> On 04.09.2015, at 07:31, James Nord <jn...@cloudbees.com> wrote:
>
>> • What percentage of Jenkins install locations do you expect have no internet connection?

>
> I don't know - we don't collect statistics from those types of machines :-/

We might, as the usage stats are sent by users' browsers. If they go through the proxy, we could add some connection test to Jenkins to collect stats about that -- for future releases.

I still expect that completely offline environments (including the users of Jenkins) are really rare compared to those where Jenkins doesn't get to connect through the proxy, but desktop users do, so that should be a start.

>> For version 1 of this, can't we just document some possible solutions to this problem and point the user to these solutions from inside the wizard on detecting the fact that there's no internet connection? This seems easy enough and we can test it over a period of time. If it's a big issue we'll at least have some real data to work with.
>
> For v1 all I am saying is we need to have a solution to the problem (not what that solution needs to be). If the chosen solution is pointing users to a document that explains some workarounds (possibly provides links to scripts or juseppe etc) then a solution exists and I am happy.

I think we should be able to do this pretty easily. There are several possible approached to deal with this, and we can point them to it. Even if we just package the previous bundled plugins into a zip file and make that available as a separate download, it won't be worse (except for having to download two things instead of one) than what we have today. But we should be able to do something much better than that (scripted bundle creation or script to download selection + depenencies) with a bit of effort.

[Tom Fennelly]

On Friday, September 4, 2015 at 3:31:54 PM UTC+1, James Nord wrote:

For v1 all I am saying is we need to have a solution to the problem (not what that solution needs to be).  If the chosen solution is pointing users to a document that explains some workarounds (possibly provides links to scripts or juseppe etc) then a solution exists and I am happy.

If the solution is telling users they need an internet connection then I am not happy (as that's not a solution).

+1 Agreed  

[Tom Fennelly]

On Friday, September 4, 2015 at 5:37:26 PM UTC+1, Daniel Beck wrote:

>> For version 1 of this, can't we just document some possible solutions to this problem and point the user to these solutions from inside the wizard on detecting the fact that there's no internet connection? This seems easy enough and we can test it over a period of time. If it's a big issue we'll at least have some real data to work with.
>
> For v1 all I am saying is we need to have a solution to the problem (not what that solution needs to be).  If the chosen solution is pointing users to a document that explains some workarounds (possibly provides links to scripts or juseppe etc) then a solution exists and I am happy.

I think we should be able to do this pretty easily. There are several possible approached to deal with this, and we can point them to it. Even if we just package the previous bundled plugins into a zip file and make that available as a separate download, it won't be worse (except for having to download two things instead of one) than what we have today. But we should be able to do something much better than that (scripted bundle creation or script to download selection + depenencies) with a bit of effort.

This sounds like a practical approach to me +1

[Tom Fennelly]

A WiP PR relating to this: https://github.com/jenkinsci/jenkins/pull/1822

[Tom Fennelly]

It would be good if some people could try out what we've done so far and let is know. Here's a link to a jenkins.war from the PR Builder (or get the latest build via the PR itself).

One of the important things people might could check is the upgrade path and how the detached/split plugins are handled. All the plugins that were bundled in the jenkins.war are still bundled (in WEB-INF/detached-plugins), but are only conditionally loaded on upgrade (not loaded at all on a new install - the wizard handles that).

The upgrade ("last" running jenkins instance version was less than "this") conditions under which detached plugins are loaded from WEB-INF/detached-plugins are:

1. The plugin in WEB-INF/detached-plugins was detached since the "last" running version.
   
2. The plugin in WEB-INF/detached-plugins is already already installed in "this" jenkins instance.
• The only effect here will be a possible upgrade of the plugin if the installed version is less than the bundled version AND the installed version is not pinned.
  
3. The plugin in WEB-INF/detached-plugins is a dependency of one of the plugins selected via #1 or #2 above.
   
• i.e. we also install the dependencies of plugins selected via criteria #1 or #2 above. For this, we use the "Plugin-Dependencies" attribute in the manifest. The dependency plugin must also be in WEB-INF/detached-plugins.

The wizard you'll see (if you have an empty JENKINS_HOME i.e. not an upgrade) is just a temp wizard I created to allow us put in some of the backend bits. It only allows you to install some "recommended" plugins as a quick starter. You can try this of course but don't waste time critiquing it because it will be getting binned. Keith (Zantow) and Gus are building a nicer wizard and we should have that soon (will allow easy plugin selection etc).

[Jesse Glick]

On Wed, Sep 16, 2015 at 3:42 AM, Tom Fennelly <tom.fe...@gmail.com> wrote:
> The plugin in WEB-INF/detached-plugins is already already installed in
> "this" jenkins instance.
> The only effect here will be a possible upgrade of the plugin if the
> installed version is less than the bundled version AND the installed version
> is not pinned.

-1, `*.jpi.pinned` should be ignored henceforth, and we should leave
the installed version alone, period.

[Tom Fennelly]

On Wednesday, September 16, 2015 at 2:50:31 PM UTC+1, Jesse Glick wrote:

On Wed, Sep 16, 2015 at 3:42 AM, Tom Fennelly <tom.fe...@gmail.com> wrote:
> The plugin in WEB-INF/detached-plugins is already already installed in
> "this" jenkins instance.
> The only effect here will be a possible upgrade of the plugin if the
> installed version is less than the bundled version AND the installed version
> is not pinned.

-1, `*.jpi.pinned` should be ignored henceforth,

Daniel Beck is of the opinion that .pinned should be honoured.

It would help if you said why you think this.

 

and we should leave
the installed version alone, period.

Again, it would be helpful to know why you say this.

What I have seen here is that, if left with just the installed version (from the previous release - pre upgrade), you can end up getting exceptions on upgrade.

For example, Jenkins 1.547 has external-monitor-job version 1.2. Version 1.4 of this plugin received updates that were moved out from Jenkins core. Jenkins core now has a hard dependency on external-monitor-job version 1.4+ (probably indirectly via one of the plugins that were detached between those releases?). Therefore, without updating this plugin to at least version 1.4 as part of the upgrade, we get the error you saw 2 days ago (not able to resolve hudson.model.Messages.ExternalJob_DisplayName) when upgrading from 1.547 to a new Jenkins that doesn't run version 1.4 of this plugin.

Maybe this could be solved by only updating already installed detached plugins if the new version is a dependency of one of the detached plugins since the last release?

The only other way I can see of solving this is by telling people that they need to update all plugins before shutting down the old version of Jenkins and installing the new version. That sucks imo (and probably wouldn't work anyway in many cases). Alternatively, we could just say that they need to manually update plugins after upgrade, which also seems to suck.

[Gus Reiber]

I also hate "pinned". WtF does it mean? I now know, but it is sorta crazy.

If we have to keep it, we have to keep it, but it does suck.

--

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/kRobm-cxFw8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/61c9a184-ef21-447d-88db-8886bfa33ab5%40googlegroups.com.

[Daniel Beck]

The issue that caused me to think we may need to keep pinning around is your change to External Monitor Job 1.4 where a resource was moved from core into the plugin. Tom upgraded from 1.547 to 1.62x (with his bundling changes) and wondered where the error that a Messages entry for that job type wasn't found came from.

What are we going to do when we notice there's something broken with the first release of a detached (and therefore compat-bundled) plugin? Fix and release the plugin, and increase the compat-bundled version? This won't help users who have the older release. Should we just tell them to please update the plugin version? I guess one approach would be to repurpose the current PinningIsBlockingBundledPluginMonitor to just say "You should really update these plugins, we think the old releases are broken" or something like that?

[Jesse Glick]

On Wed, Sep 16, 2015 at 1:33 PM, Daniel Beck <m...@beckweb.net> wrote:
> The issue that caused me to think we may need to keep pinning around is your change to External Monitor Job 1.4 where a resource was moved from core into the plugin.

I see. Basically this is what `PinningIsBlockingBundledPluginMonitor`
was designed to help with, though it sounds like you are getting some
error earlier and more intrusively than that was designed to handle.
Can we treat this particular error as a robustness bug?

Alternately, be more aggressive and force updates of “split-bundled”
plugins newer than an installed version, disregarding historical
pinned status (and deleting the existing monitor). I think that is a
much more reasonable proposal when we are no longer bundling plugins
just for fun, and we are only talking about updating to a version very
soon after the split—from 1.0 to 1.0.1, say.

> What are we going to do when we notice there's something broken with the first release of a detached (and therefore compat-bundled) plugin? Fix and release the plugin, and increase the compat-bundled version?

That is what we have done in the past.

[Daniel Beck]

> I see. Basically this is what `PinningIsBlockingBundledPluginMonitor`
> was designed to help with, though it sounds like you are getting some
> error earlier and more intrusively than that was designed to handle.
> Can we treat this particular error as a robustness bug?

It's not a notable problem with Monitor External Job because the bundled version was increased in real Jenkins. In no-bundling Jenkins OTOH, the old version would stick around and exhibit this issue.

> Alternately, be more aggressive and force updates of “split-bundled”
> plugins newer than an installed version, disregarding historical
> pinned status (and deleting the existing monitor). I think that is a
> much more reasonable proposal when we are no longer bundling plugins
> just for fun, and we are only talking about updating to a version very
> soon after the split—from 1.0 to 1.0.1, say.

We also discussed this as "We're not doing compat-bundling upgrades just for fun" (so no version increases just to get some cool functionality in there) and it seems like a viable approach. Worst case is we'll need to backport the critical fix to an older x.y.1 so we don't include a bunch of unrelated stuff.

>> What are we going to do when we notice there's something broken with the first release of a detached (and therefore compat-bundled) plugin? Fix and release the plugin, and increase the compat-bundled version?
>
> That is what we have done in the past.

Right, the question starts in the sentence after that. Bundling and pinning allowed for upgrading of these (or at least showing a stern warning), while new compat-bundling would not, at least not as currently proposed.

[Jesse Glick]

On Wed, Sep 16, 2015 at 3:49 PM, Daniel Beck <m...@beckweb.net> wrote:
> Worst case is we'll need to backport the critical fix to an older x.y.1 so we don't include a bunch of unrelated stuff.

Exactly.

[Tom Fennelly]

So where are we now guys?

1. Are we still saying we are not going to update existing plugins on an upgrade? Jesse seemed definitive on this one but I still don't know the reasoning.
2. Are we still saying pinned files should be ignored ala what Jesse said earlier? Again, I still haven't heard the reasons why they should be ignored. This only seems relevant to me if the answer to #1 is yes (we are updating existing plugins on an upgrade).

Basically, I want to agreement on what changes are needed on the PR.

[Tom Fennelly]

Sorry ... typo in #2 on the last email. I should have said "This only seems relevant to me if the answer to #1 is no" i.e. we will upgrade if the plugin does not meet the minimum version requirements i.e. is less than the bundled version (in /WEB-INF/detached-plugins).

--

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/kRobm-cxFw8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/d7209aea-08d3-46ec-92f4-ff5070cc1410%40googlegroups.com.

[Tom Fennelly]

Maybe we should clarify what it means for a plugin to be bundled in "/WEB-INF/plugins" Vs "/WEB-INF/detached-plugins".

My understanding/interpretation is:

• /WEB-INF/plugins: These are plugins that must always be installed/updated. Same as before. The versions are kept as up-to-date as possible (I think, right?).
  
• /WEB-INF/detached-plugins: Selectively installed based on the criteria I outlined above/yesterday (Sept 16). The versions are set to the minimum required for Jenkins to work without throwing a wobbler of errors i.e. not kept up-to-date with latest version of that plugin.
  

So as I see it, upgrades from /WEB-INF/detached-plugins only happen if the installed version is below the minimum version required for the new Jenkins to work. We don't just upgrade "for fun".

[Daniel Beck]

On 17.09.2015, at 11:26, Tom Fennelly <tom.fe...@gmail.com> wrote:

> Sorry ... typo in #2 on the last email. I should have said "This only seems relevant to me if the answer to #1 is no" i.e. we will upgrade if the plugin does not meet the minimum version requirements i.e. is less than the bundled version (in /WEB-INF/detached-plugins).

Yep, it looks like that's what Jesse is arguing for. I'd have kept pinning around for now, but it looks _really_ unpopular. It's basically what we discussed a few days back, without the pinning.

Regarding your proposal, note that there are detached plugins that have since picked up dependencies. Those dependencies would also be in detached-plugins/ right?

Not sure we should keep plugins/ -- probably. Ideally official Jenkins would come without any. (Which means we'd lose Translation Assistance... not sure what a good solution is here...)

[Tom Fennelly]

On Thursday, September 17, 2015 at 11:16:41 AM UTC+1, Daniel Beck wrote:

On 17.09.2015, at 11:26, Tom Fennelly <tom.fe...@gmail.com> wrote:

> Sorry ... typo in #2 on the last email. I should have said "This only seems relevant to me if the answer to #1 is no" i.e. we will upgrade if the plugin does not meet the minimum version requirements i.e. is less than the bundled version (in /WEB-INF/detached-plugins).

Yep, it looks like that's what Jesse is arguing for. I'd have kept pinning around for now, but it looks _really_ unpopular. It's basically what we discussed a few days back, without the pinning.

So this is what's in the PR then, minus the pinning. I'm fine with removing pinning, but would be interested to know why it's so unpopular? There seems to be a strong logic to it's existence and not liking doesn't seem like a good reason to ditch it.

 

Regarding your proposal, note that there are detached plugins that have since picked up dependencies. Those dependencies would also be in detached-plugins/ right?

Right, I outlined that in the post I made here yesterday ... detached plugin dependencies (and dependencies of dependencies etc) are deployed if that detached plugin is deployed (it gets that info from the manifest). Dependencies must also be in the /WEB-INF/detached-plugin folder in the war.

 

Not sure we should keep plugins/ -- probably. Ideally official Jenkins would come without any. (Which means we'd lose Translation Assistance... not sure what a good solution is here...)

I think this should stay for now at least (and so does KK).

For one thing, the new approach to JavaScript modularization (which these changes are using - and will be used in other ui enhancements) uses plugins to bundle JavaScript "bundles" (see here). Yes, jenkins modules etc etc, but they do not allow easy upgrade and require what I consider to be ugly hacks in order to make the web resources available (UnprotectedRootActions etc sorry ... yeuck).

It seems to me like there's a problem here in that all plugins are treated equally when they really are not all equal. Some plugins provide actual "features", while other plugins are just "dependency" plugins (call them what you like) and are dependencies of "feature" plugins OR Jenkins core itself. I'd think it makes sense to allow these "dependency" type plugins to be bundled in Jenkins core, but not the "feature" plugins. Jenkins modules could work instead of some of these "dependency" plugins, but not them all because of the updating issue with modules (you need a new Jenkins).

[Daniel Beck]

On 17.09.2015, at 13:02, Tom Fennelly <tom.fe...@gmail.com> wrote:

> while other plugins are just "dependency" plugins … and are dependencies … Jenkins core itself.

These do not exist today. If you need them for other changes, this will need to be considered separately. Remember, every plugin, even the bundled ones, can be disabled, which would in this case break Jenkins. Maybe the solution to that issue even blocks finalizing this one?

[Tom Fennelly]

On Thursday, September 17, 2015 at 1:04:15 PM UTC+1, Daniel Beck wrote:

On 17.09.2015, at 13:02, Tom Fennelly <tom.fe...@gmail.com> wrote:

> while other plugins are just "dependency" plugins … and are dependencies … Jenkins core itself.

These do not exist today.

Sure, I'm just suggesting that there should be some sort of distinction of different types of plugins. They all seem to be treated the same atm, which seems wrong to me.

 

If you need them for other changes, this will need to be considered separately. Remember, every plugin, even the bundled ones, can be disabled, which would in this case break Jenkins. Maybe the solution to that issue even blocks finalizing this one?

Well I do think these things need to be considered at some stage, but not as blockers for this (imo). Jenkins has always been at varying degrees of mercy of people disabling plugins at a bad time, right? Sure, this would be more unmerciful though :)

[Jesse Glick]

On Thu, Sep 17, 2015 at 6:16 AM, Daniel Beck <m...@beckweb.net> wrote:
>> we will upgrade if the plugin does not meet the minimum version requirements i.e. is less than the bundled version (in /WEB-INF/detached-plugins).
>
> Yep, it looks like that's what Jesse is arguing for.

Yes. So my running proposal (beware that I have not had time to work
through every implication, the onus is on you to test the corner
cases!):

· nothing, nada, in `WEB-INF/plugins/`
· `WEB-INF/detached-plugins/` only for plugins historically split out
of core, and only the 1.0 release created at the time of the split
· but amended as needed to 1.0.x to pick up important fixes either
for accidental regressions in the original split, or core
incompatibilities introduced subsequently
· also for the transitive dependencies of split plugins (e.g.,
`script-security`), in the minimal possible versions satisfying stated
dependencies
· `*.jpi.pinned` to be ignored/deleted henceforth
· plugin list in an existing installation left unmodified except
· split plugins added as needed, according to last-run version
· in the special case of a *previous* automatic installation of a
1.0 split plugin, if 1.0.x is now in `detached-plugins`, upgrade
· likewise, if a dependency of a split plugin is newer than the
installed version, upgrade
· delete `PinningIsBlockingBundledPluginMonitor`

> we'd lose Translation Assistance

Just check it by default in the new plugin wizard. I do not see how
this is any different from any other plugin we broadly recommend users
install.

> dependencies of "feature" plugins OR Jenkins core itself

Jenkins core cannot depend on a plugin, this makes no sense. (There
are `jenkins-module`s shared among core and plugins, but the user has
no control over their installation, they are just in `WEB-INF/lib/`.)

[Tom Fennelly]

On Thursday, September 17, 2015 at 3:59:46 PM UTC+1, Jesse Glick wrote:

So my running proposal
· nothing, nada, in `WEB-INF/plugins/`

> dependencies of "feature" plugins OR Jenkins core itself

Jenkins core cannot depend on a plugin, this makes no sense. 

(There
are `jenkins-module`s shared among core and plugins, but the user has
no control over their installation, they are just in `WEB-INF/lib/`.)

I think maybe KK needs to chip in here. I thought I outlined a scenario where I thought it does make sense from the pov of how we plan to do some UI enhancements going forward (and KK seemed to think so too). I still don't get why it's such a no-no. Well, I do get the issues re it being possible for someone to disable them, but that to me is a different issue and goes back to my point about the lack of distinction between different kinds of plugins and what people are allowed to do to them, how the appear in the PM etc.

Anyway, if this is the approach and this restriction is imposed then we need to rethink how we do the UI stuff in Jenkins core (wizard, config page changes etc). That's another discussion I guess, but does seem to indicate to me that there's a hole (in this logic, or my understanding :) ) that needs filling. jenkins-modules are available but don't seem to meet the requirements (easy exposing of web resource without Java hackery) and can't be updated (require a new Jenkins).