Plugin site prototype

[Daniel Beck]

Hi everyone,

As previously discussed[1], Jenkins users would benefit from a better plugins index. Right now, all they have is the 'Plugins' wiki page[2] and there's so much wrong with it, one wonders whether it gets anything right. Which leaves users to basically use Google to find things -- clearly not a great situation for a tool like Jenkins, whose major appeal is in the large and growing number of plugins and what they enable users to do.

So Gus and Michael McCaskill have recently worked on building a better plugins index -- the plugins site, to be integrated with jenkins.io -- which itself was only recently overhauled to be actually useful for more than just the changelog. Initial designs were presented a few months ago[3].

A lot of progress has been made over the last few weeks, and we finally managed to deploy the work in progress to a public site:
http://cloudbees-plugin-site-staging.s3-website-us-east-1.amazonaws.com/

This pretty much looks like the finished site could. A few things worth mentioning:

* Index page shows most installed, recently updated, and trending (i.e. rapidly growing adoption) plugins.
* Shiny URL for plugins (which will be something like plugins.jenkins.io/git once productive)
* Three different list visualizations: Grid, List, and Table.
* Filter by maintainer (which as metadata is a bit of a mess due to inconsistencies in POMs, but we try)
* It includes the Jenkins wiki page contents of the plugin (for now, could do something like allow opt-in to GitHub based docs in the future)

There's a few glitches left, but we wanted to get this work in progress out to get your feedback. Things like which plugin categories show up, and how they're labeled, is really easy to change at any time, so don't let that distract you. Love it? Hate it? Please let me know!

Daniel


1: Roughly starting at https://groups.google.com/d/msg/jenkinsci-dev/EMbE3a4u8nA/WiNpTSM4CgAJ
2: https://wiki.jenkins-ci.org/display/JENKINS/Plugins
3: https://groups.google.com/d/msg/jenkinsci-dev/2nqWoE1FfOo/2EX2-l9dBQAJ

[Daniel Spilker]

Hi,

I like it a lot! Thanks to everyone for the great work!

my 2 cents:

* optional dependencies are not labeled as such, but should be. It's an important info that shows which additional plugins will be installed or not

* the search for maintainers should be improved. I only maintain the Job DSL plugin, but the page shows 7 plugins and the most relevant (Job DSL) is shown last. http://cloudbees-plugin-site-staging.s3-website-us-east-1.amazonaws.com/?maintainers=Daniel%20Spilker

* what is the workflow for editing the page? is the data pulled from Confluence frequently?

Daniel

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/679434B9-A494-4775-9153-3A816D46FB5E%40beckweb.net.
For more options, visit https://groups.google.com/d/optout.

[Robert Sandell]

Looks good!

How do I get the logo/icon to display instead of the default text?

Some "issues" I've found:

The Dependencies section shows all dependencies as "required" (as I interpret the missing info) even though some of them are optional.

The index links goes to the wiki instead of the anchor point on the same page.

/B

On Wed, Aug 31, 2016 at 1:07 PM, Daniel Beck <m...@beckweb.net> wrote:

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/679434B9-A494-4775-9153-3A816D46FB5E%40beckweb.net.
For more options, visit https://groups.google.com/d/optout.

--

Robert Sandell

Software Engineer

CloudBees Inc.

[Robert Sandell]

And it doesn't seem to behave well on firefox; the browse/filter bar disappears and resets for every click I make.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAKqW32ADOH6Tup69UT-bqb2XE7Eope6q59mCZ55djZma1Na%2BtA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Daniel Beck]

> On 31.08.2016, at 13:22, Daniel Spilker <ma...@daniel-spilker.com> wrote:
>
> * optional dependencies are not labeled as such, but should be. It's an important info that shows which additional plugins will be installed or not

Good point. Also, we currently don't cover implied dependencies to detached plugins, and it still shows the raw ID rather than name for dependencies, so there's still a bit of potential for improvement here ;-)

> * the search for maintainers should be improved. I only maintain the Job DSL plugin, but the page shows 7 plugins and the most relevant (Job DSL) is shown last. http://cloudbees-plugin-site-staging.s3-website-us-east-1.amazonaws.com/?maintainers=Daniel%20Spilker

The plight of the common first name -- 'Spilker' works. should probably not be clever and split search tokens. Or is searching for plugins with multiple specified maintainers a relevant use case?

> * what is the workflow for editing the page? is the data pulled from Confluence frequently?

I think the wiki text cache expires after six hours.

[Daniel Beck]

> On 31.08.2016, at 13:24, Robert Sandell <rsan...@cloudbees.com> wrote:
>
> The index links goes to the wiki instead of the anchor point on the same page.

True, that's weird. I'll make sure to put it on the to-do list if it isn't yet.

[Christopher Orr]

Looks pretty cool!

On 31 Aug 2016, at 13:31, Daniel Beck <m...@beckweb.net> wrote:

* the search for maintainers should be improved. I only maintain the Job DSL plugin, but the page shows 7 plugins and the most relevant (Job DSL) is shown last. http://cloudbees-plugin-site-staging.s3-website-us-east-1.amazonaws.com/?maintainers=Daniel%20Spilker

The plight of the common first name -- 'Spilker' works. should probably not be clever and split search tokens. Or is searching for plugins with multiple specified maintainers a relevant use case?

Yeah, it’s weird that the <developers> POM <name> value rather than <id> value is used — searching ?maintainers=<id> works fine.  e.g. "orrc" gives the expected results, but "Christopher Orr" returns multiple Christophers.

I also noticed a bunch of wiki HTML problems, some of which were already mentioned.  But mainly I was impressed how well the HTML display works already :)

For example on the android-emulator page: http://cloudbees-plugin-site-staging.s3-website-us-east-1.amazonaws.com/android-emulator

- The {expand} macro doesn’t work, so stuff like "View older versions" (at the very bottom) doesn’t work

- All Table of Contents links lead to the wiki, rather than an anchor within the page

- All plugin page links lead to the wiki, rather than the plugin page on this new site

Also, all install graphs seem to have a hardcoded minimum Y value of 10000 installs — way above that of most plugins.

I’m not sure why there’s a prominent "Download" button, given than nobody should ever be downloading plugins from this site.  There should preferably be a link to some instructions, telling people how to install plugins.

Overall, the design is nice, but I don’t like that the plugin details appear in a "popup" over the page — I have to close this popup with the small "X" or back arrow, before I can use the search field again.

If the plugin details appeared inline below the search bar, like search results do, it would be a lot nicer (IMO).

What does the "Recently installed" list on the home page mean?  The home page is also a bit bare — for what should be a big landing page, there’s a lack of title or information somehow.

Anyway, very nice — the searching, sorting and categorisation works really well.  Overall a huge improvement!

Regards,

Chris

[Daniel Beck]

> On 31.08.2016, at 15:50, Christopher Orr <ch...@orr.me.uk> wrote:
>
> Yeah, it’s weird that the <developers> POM <name> value rather than <id> value is used — searching ?maintainers=<id> works fine. e.g. "orrc" gives the expected results, but "Christopher Orr" returns multiple Christophers.

Both work. Sometimes we only have one or the other, so it's tolerant in what it can search for. It tries to always show the name if available though.

> I also noticed a bunch of wiki HTML problems, some of which were already mentioned. But mainly I was impressed how well the HTML display works already :)

The power of scraping rendered pages! (And all of the issues you're seeing directly follow from that…)

> Also, all install graphs seem to have a hardcoded minimum Y value of 10000 installs — way above that of most plugins.

That's an interesting observation. Anyone else has comments on this? This was a contested issue between Gus and me as well, so feedback on this would be appreciated.

> I’m not sure why there’s a prominent "Download" button, given than nobody should ever be downloading plugins from this site. There should preferably be a link to some instructions, telling people how to install plugins.

Good point. Manually downloading is miserable due to dependencies etc., so maybe a popup explaining how to install in Jenkins (and bury a download link somewhere if the user insists -- e.g. for offline instances) would be useful.

I wonder whether we can do something like let users specify their Jenkins URL on the plugin site in HTML 5 local storage, and add a "install this" (like "tweet this") URL to Jenkins that the site redirects to. That way, they can discover plugins and almost seamlessly install them.

> What does the "Recently installed" list on the home page mean?

It's what I was referring to as "trending" plugins in my initial post: The math is still work in progress, but the idea is to show these three lists:

1. The most popular plugins (most installations total) -- right now it's mostly what was bundled before, but that should flatten out after a while. It'll still be driven by recommended plugins in the wizard, but not as much as bundling did before.
2. Recent releases. Because new things (both plugins and updates) are interesting.
3. Trending plugins (which are rapidly growing their relative installation counts) -- we need to be careful not to place the plugin that grew 10x from 10 installs to 100 installs first, but going rapidly from 2000 to 2500 (+25%) from one month to the next is impressive and makes the plugin very interesting.

[Jesse Glick]

Very nice.

On Wed, Aug 31, 2016 at 7:07 AM, Daniel Beck <m...@beckweb.net> wrote:
> Filter by maintainer (which as metadata is a bit of a mess due to inconsistencies in POMs, but we try)

Is the POM `<maintainer>` even at all reliable? Could we
additionally/instead check for the GitHub user(s) to perform recent
releases and/or merge recent pull requests?

> could do something like allow opt-in to GitHub based docs in the future

That would be splendid, but I guess is part of a larger discussion
about jenkins.io content aggregation from plugins.

Would be wise to offer a direct link to the wiki, in case the inlining
is broken, and to pick up additional stuff not yet here like the link
to changes since the last release.

[Daniel Beck]

> On 31.08.2016, at 16:34, Jesse Glick <jgl...@cloudbees.com> wrote:
>
> Is the POM `<maintainer>` even at all reliable? Could we
> additionally/instead check for the GitHub user(s) to perform recent
> releases and/or merge recent pull requests?

This site uses (latest) update-center.json + the per-plugin stats at http://stats.jenkins.io/plugin-installation-trend/ + scraping wiki pages for its data.

And since update-center.json is populated using the POM metadata, that's all we have. A fix should probably address update-center.json generation, rather than the site.

Which may ultimately be weird -- e.g. duplication -- due to different user names between GitHub and LDAP, but probably still better than what we have now.

Or just distinguish "Maintainers" and "This version was released by"?

> Would be wise to offer a direct link to the wiki, in case the inlining
> is broken, and to pick up additional stuff not yet here like the link
> to changes since the last release.

Good point.

Also, we probably should have at least some of the links from the wiki's plugin info box on the new plugins site as well.

[R. Tyler Croy]

(replies inline)

On Wed, 31 Aug 2016, Daniel Beck wrote:

>
> > What does the "Recently installed" list on the home page mean?
>
> It's what I was referring to as "trending" plugins in my initial post: The math is still work in progress, but the idea is to show these three lists:
>
> 1. The most popular plugins (most installations total) -- right now it's mostly what was bundled before, but that should flatten out after a while. It'll still be driven by recommended plugins in the wizard, but not as much as bundling did before.
> 2. Recent releases. Because new things (both plugins and updates) are interesting.
> 3. Trending plugins (which are rapidly growing their relative installation counts) -- we need to be careful not to place the plugin that grew 10x from 10 installs to 100 installs first, but going rapidly from 2000 to 2500 (+25%) from one month to the next is impressive and makes the plugin very interesting.

This doesn't quite make sense to me. The home page already lists 'Recently
Updated' and also 'Most Installed' as a separate column, so I'm confused as to
why the math would include things from there.

Asked differently, what is the goal of presenting these different blocks of
plugin lists below the search field? There are three lists of plugins, and one
list of categories. The latter doesn't feel like it belongs, but it's unclear
what the design motivation for listing the plugins was.


- R. Tyler Croy

------------------------------------------------------
Code: <https://github.com/rtyler>
Chatter: <https://twitter.com/agentdero>

% gpg --keyserver keys.gnupg.net --recv-key 1426C7DC3F51E16F
------------------------------------------------------

[R. Tyler Croy]

(replies inline)

On Wed, 31 Aug 2016, Daniel Beck wrote:

> > Also, all install graphs seem to have a hardcoded minimum Y value of 10000 installs ??? way above that of most plugins.

>
> That's an interesting observation. Anyone else has comments on this? This was a contested issue between Gus and me as well, so feedback on this would be appreciated.

The problem, IMHO, with non-relative graphs here is that the Y-axis scales
upwards but not downwards, which makes less popular plugins look a *LOT* less
popular.

For example:
<http://cloudbees-plugin-site-staging.s3-website-us-east-1.amazonaws.com/structs>

Which scales up to 50k, versus:
<http://cloudbees-plugin-site-staging.s3-website-us-east-1.amazonaws.com/openstack-cloud>


If the latter link had a relatively scaled graph, you would notice that it's
doubled in installs over the past 5 months or so; that's cool information
that's obscured here.

[Daniel Beck]

> On 31.08.2016, at 19:33, R. Tyler Croy <ty...@monkeypox.org> wrote:
>
> This doesn't quite make sense to me. The home page already lists 'Recently
> Updated' and also 'Most Installed' as a separate column, so I'm confused as to
> why the math would include things from there.
>
> Asked differently, what is the goal of presenting these different blocks of
> plugin lists below the search field? There are three lists of plugins, and one
> list of categories. The latter doesn't feel like it belongs, but it's unclear
> what the design motivation for listing the plugins was.

We are unable to manually curate "interesting" plugins for the home page like app stores would. But we still want them to use the plugins site to explore what plugins exist, rather than use it just as a different search field from Google's.

So the idea was to generate "top lists" of interesting plugins based on their properties. What kinds of plugins are interesting in some way, and can be highlighted for people who want to explore?

* The ones that most people use.
* The ones that have recently been updated.
* The ones that are trending right now ("used a lot more than in the past").

As to the difference between the first and third, it's basically the difference between Nokia and iPhone market shares back in 2010. Absolute numbers: Nokia wins. Growth: iPhone wins.

Also, I totally stole the idea from packagecontrol.io, because it's great :-)

[Gus Reiber]

Tyler, I am confused by your post.

256 is A LOT less than 50k. A LOT, A LOT less. 20x less.

10K is a lot less than 50k, so if anything, Open-Stack Cloud's 256 installs is being exaggerated. It is being graded on an easier scale than Structs, which is also being graded on an easier scale than Mailer, which has over 100k installs and is graded on a 200k scale.

To someone wondering if a plugin is worth installing, moving from 20 installs to 40 installs is 2x, but mostly meaningless. Moving from 1 install to 10 is totally meaningless despite being 10x. Whereas, moving from 75k to 125k over the same time would be a big deal.

Using your own examples, the Structs jump really is remarkable. The Open-stack jump, isn't particularly, unless you are that plugin's developer.

For the recently installed....

As Daniel described, the point of the recently installed is to give newer plugins a way to shine.

If a plugin has been around for a long time, it has had more time to accumulate installations. This levels that playing field, tried-and-true vs. new-and-exciting.

[Jesse Glick]

Your example highlights an issue: there is no reason to show `structs` to a user at all, since it offers no user-level features. It is just an API. No one chose to install it, they just got it as a dependency from some update of another plugin.

When I worked on NetBeans we could mark modules as being libraries (~ APIs) or bridges (integrations), which would get installed only as needed, and would normally be hidden from display. Perhaps Jenkins needs an equivalent. W.r.t. “bridges” I have an open RFE to deprecate optional dependencies in favor of these.

[R. Tyler Croy]

(replies inline)

On Thu, 01 Sep 2016, Gus Reiber wrote:

> Tyler, I am confused by your post.
> 256 is A LOT less than 50k. A LOT, A LOT less. 20x less.
> 10K is a lot less than 50k, so if anything, Open-Stack Cloud's 256 installs
> is being exaggerated. It is being graded on an easier scale than Structs,
> which is also being graded on an easier scale than Mailer, which has over
> 100k installs and is graded on a 200k scale.
>
> To someone wondering if a plugin is worth installing, moving from 20
> installs to 40 installs is 2x, but mostly meaningless. Moving from 1
> install to 10 is totally meaningless despite being 10x. Whereas, moving
> from 75k to 125k over the same time would be a big deal.
>
> Using your own examples, the Structs jump really is remarkable. The
> Open-stack jump, isn't particularly, unless you are that plugin's developer.
>

There are a number of niche plugins which will *never* see tens of thousands
installations. Regardless of how the overall installation number is viewed, a
relatively scaled Y-axis, like that on the current plugin page
(https://wiki.jenkins-ci.org/display/JENKINS/Openstack+Cloud+Plugin), conveys
useful information (IMHO) about how that specific plugin is trending/growing.

Said differently, I think the minimum Y-axis at 10,000 makes that graph only
useful for plugins in a larger install-range (5000+), which is only the top
115 (ish) of 1200 plugins available in the update center. For the long-tail it
doesn't strike me as useful.

> For the recently installed....
>
> As Daniel described, the point of the recently installed is to give newer
> plugins a way to shine.
> If a plugin has been around for a long time, it has had more time to
> accumulate installations. This levels that playing field, tried-and-true
> vs. new-and-exciting.

Aside from "Browse Categories"
which doesn't make sense to me on the same level with three lists of plugins, I
think the "Recently Installed" list is much less problematic if it's simply
titled "Trending", then we can hand-wave all the behind-the-scenes math just
like Twitter and Facebook do :)

[Gus Reiber]

+1 to that.

I would love to filter out plugins that aren't directly feature enabling.

At the moment, we lack any sort of metadata to make that distinction, but I think that would help the site greatly.

I think it would also be good to mark those OSS Jenkins is recommends during the install process to filter in or out.

...but that doesn't change the fact of the math, though, as far as the charts go.

[Daniel Beck]

> On 01.09.2016, at 21:43, Gus Reiber <gre...@cloudbees.com> wrote:
>
> At the moment, we lack any sort of metadata to make that distinction, but I think that would help the site greatly.

We could introduce the 'library' category of plugins and add a special meaning to that.

However I wouldn't make that hide the plugin -- there may be legitimate uses to want to learn about it -- just deemphasize it (e.g. lower ranking)

[Robert Sandell]

On Tue, Sep 6, 2016 at 2:21 AM, Daniel Beck <m...@beckweb.net> wrote:

> On 01.09.2016, at 21:43, Gus Reiber <gre...@cloudbees.com> wrote:
>
> At the moment, we lack any sort of metadata to make that distinction, but I think that would help the site greatly.

 

We could introduce the 'library' category of plugins and add a special meaning to that.

It already exists https://wiki.jenkins-ci.org/label/JENKINS/plugin-library

 

However I wouldn't make that hide the plugin -- there may be legitimate uses to want to learn about it -- just deemphasize it (e.g. lower ranking)

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/515693DD-6F7E-47FB-B753-C44D2E5A6EDF%40beckweb.net.

For more options, visit https://groups.google.com/d/optout.

[Baptiste Mathus]

Wondering if that should be somehow expressed in the hpi itself, or if it's fine/a feature to have it outside. I guess having it in the project itself would have an advantage: if a plugin goes from user-facing to only library-plugin (or the contrary), that could be expressed per version. 

Great work BTW!

-- Baptiste

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CALzHZS2GhXLz31m2XDjiVfgdcv2AmTRPCJ4qLGGVs-UhKXXH2g%40mail.gmail.com.

[Oleg Nenashev]

Just to bump this thread to the top.

I have not voted for this site yet, but I definitely support the change. There are much features which could be added, but it would be great to kick-off the site and the underlying infrastructure to the beta-* version on jenkins.io. It would allow making some contributions there using the common PR flow.

@Daniel What is the current plan for this feature?

Thanks in advance,
Oleg

среда, 7 сентября 2016 г., 10:10:54 UTC+2 пользователь Baptiste Mathus написал:

Wondering if that should be somehow expressed in the hpi itself, or if it's fine/a feature to have it outside. I guess having it in the project itself would have an advantage: if a plugin goes from user-facing to only library-plugin (or the contrary), that could be expressed per version. 

Great work BTW!

-- Baptiste

2016-09-06 10:51 GMT+02:00 Robert Sandell <rsan...@cloudbees.com>:

On Tue, Sep 6, 2016 at 2:21 AM, Daniel Beck <m...@beckweb.net> wrote:

> On 01.09.2016, at 21:43, Gus Reiber <gre...@cloudbees.com> wrote:
>
> At the moment, we lack any sort of metadata to make that distinction, but I think that would help the site greatly.

 

We could introduce the 'library' category of plugins and add a special meaning to that.

It already exists https://wiki.jenkins-ci.org/label/JENKINS/plugin-library

However I wouldn't make that hide the plugin -- there may be legitimate uses to want to learn about it -- just deemphasize it (e.g. lower ranking)

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/515693DD-6F7E-47FB-B753-C44D2E5A6EDF%40beckweb.net.

For more options, visit https://groups.google.com/d/optout.

--

Robert Sandell

Software Engineer

CloudBees Inc.

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.