Yesterday's security advisory

[Daniel Beck]

Hi everyone,

We now had the situation where the number of vulnerabilities far exceeded what the security team could handle.
https://jenkins.io/security/advisory/2017-04-10/

As previously discussed on this list, I've suspended distribution of plugins that are currently vulnerable.
https://jenkins.io/blog/2017/04/10/security-advisory/#distributing-vulnerable-plugins

List of affected plugins:
https://github.com/jenkins-infra/backend-update-center2/blob/1be044d25a312ca90336044f501e0b9e38ca3b2e/src/main/resources/artifact-ignores.properties#L187...L209

Any thoughts about this, now that it has happened?

---

As I wrote in the blog post, I was unable to contact all maintainers. Most maintainers of affected plugins with fewer than 500 installations didn't learn about this in advance. This is really not how we usually work. I consider this to be an exceptional situation.

So, again, to affected plugin maintainers, I really am sorry. I just didn't see a feasible alternative to the chosen approach. Perhaps this thread can result in some ideas -- What should we do differently in the future, if a situation similar to this one ever comes up again?

---

And then there's plugins that needed to be delisted since they have mandatory dependencies on delisted plugins:
https://github.com/jenkins-infra/backend-update-center2/blob/1be044d25a312ca90336044f501e0b9e38ca3b2e/src/main/resources/artifact-ignores.properties#L212...L218

From a security POV, there's nothing wrong (that I'm aware of) with any of these, other than that they bring along with them an unsafe plugin. Some of these are clearly tied to build-flow, so while that is gone, so are they. Then, there are the others (maintainers in CC):

- uno-choice: This depends on Scriptler. I discussed that plugin with Domi (Scriptler's maintainer) when we couldn't get the fixes finished, and plan to work with him to fix the various issues over the next several weeks or so. Once that gets restored, uno-choice would also be published again.
- externalresource-dispatcher: This depends on Build Flow, whose maintainers added a deprecation notice to the plugin wiki last year. I would be surprised if that got revived again. So there's probably no good solution, other than cutting this dependency, if we keep unsafe plugins delisted.

[Bruno P. Kinoshita]

Hi Daniel,

Surprised to see scriptler there, wasn't expecting it given the number of people using it. If there is anything I can do to help, just let me know. I am a bit busy this week, but can definitely stop a couple of hours two or three days this week and maybe during the upcoming holidays to help fixing some issues.

I can try to help with scriptler (have been studying it to submit some pull requests / RFE's) and possibly with dynamicparameter. uno-choice was created based on dynamicparameter, so likely we can compare the code, look at some git commits, and try to pull the fix (likely security script plug-in integration?) into dynamicparameter.

Happy to test, reproduce bugs, or try to submit one or two pull requests if necessary.

>-- What should we do differently in the future, if a situation similar to this one ever comes up again?

Excellent approach. Let's tackle the problem, not the blame :-)

It is not clear from your e-mail if that was a case of a problem in the process used to communicate issues to plug-in maintainers, or if it was caused due to the number of issues vs. number of people working on the issues. Do we need to increase the number of people in the security team, or define some timely process, like having a dashboard or notification system, that sends an e-mail every week with the number of issues pending notification?
Cheers
Bruno

________________________________
From: Daniel Beck <m...@beckweb.net>
To: Jenkins Developers <jenkin...@googlegroups.com>
Cc: Bruno P. Kinoshita <brunod...@yahoo.com.br>; neurals...@gmail.com
Sent: Tuesday, 11 April 2017 9:51 PM
Subject: Yesterday's security advisory

[Daniel Beck]

> On 11.04.2017, at 12:03, 'Bruno P. Kinoshita' via Jenkins Developers <jenkin...@googlegroups.com> wrote:
>
> Surprised to see scriptler there, wasn't expecting it given the number of people using it. If there is anything I can do to help, just let me know. I am a bit busy this week, but can definitely stop a couple of hours two or three days this week and maybe during the upcoming holidays to help fixing some issues.

Scriptler is also the one I hated most to delist. The advisory lists all the things currently wrong with Scriptler that we know of. Some of these I expect to be rather straightforward to fix. What needs some thinking is how to implement script execution in jobs in a way that's safe to use. Giving non-admins Run Scripts is not a good idea (in fact, this quirk of Scriptler is a major reason we threw in the changes to the auth strategies in this advisory), and build step configuration protection as implemented is rather easily circumvented, as demonstrated.

> It is not clear from your e-mail if that was a case of a problem in the process used to communicate issues to plug-in maintainers, or if it was caused due to the number of issues vs. number of people working on the issues.

Both.

It's not enough to just assign an issue in SECURITY to plugin maintainers -- many haven't had to deal with a privately reported security issue before, so the process required to prepare a fix is unusual, and we need to explain how to go about doing that. Then there's the problem that they may not even react, so I need to follow up via email, etc.

Once we have maintainers' attention, and they understand the process we'd like them to follow, there's the issue of implementation. Especially wrt scripting issues, some more advanced features may not map cleanly to sandboxing/approval. So these fixes took a lot more supporting effort than usual. And in cases where there's no maintainer, or they're not able to spend the time to fix them, implementation falls back to the security team.

And even once the issue is fixed, there's overhead in coordinating release of the fix, writing and publishing the advisory, blog, and warnings, sending notifications, etc.

Even just getting these ~8 plugin releases out at the same time has required some pretty major coordination (well, for me -- maybe I just suck at this). Luckily for me, ikedam, Ulli Hafner, and Daniel Spilker were absolutely amazing to work with, and also handled fixing their plugins very well. (And the rest of the releases were performed by members of the security team, and they of course also know what they're doing!)

> Do we need to increase the number of people in the security team

We're always looking for more volunteers! Membership is open to most active contributors. Prerequisites and process at https://jenkins.io/security/#team

[Bruno P. Kinoshita]

Thanks for the detailed response Daniel.

>And even once the issue is fixed, there's overhead in coordinating release of the fix, writing and publishing the advisory, blog, and warnings, sending notifications, etc.

Perhaps we could create a type of Certified or Supported tag to plugins. Plug-ins under this tag/category would

* Use an ~recent LTS version
* Have security issues patches applied by the security team in 15 running days if there is no response from the maintainer
* Be released by the security team if a patch has been approved/reviewed but there was no response or availability of the plug-in maintainer

As a plugin developer, I would feel bad for others pushing changes and releasing the plug-in without notifying me; but if we had this category, then I think users/build engineers would favour to install plug-ins listed there, and as a plugin maintainer I would accept the items listed above as a way to show users updates would be released in a faster way.

Maybe this, combined with number of installations, could be a better indicator of how well maintained and production-ready the plug-in is (e.g. I would probably submit a plug-in to this category after I had confidence its API was ready and I had enough tests, fixed enough bugs, etc). Just food for thought.

>It's not enough to just assign an issue in SECURITY to plugin maintainers

Using what was suggested above, we would still assign the issue to them, but the security team could also work on a fix and propose the fix. Then the author of the plug-in could simply approve or provide further feedback.


> (...) (well, for me -- maybe I just suck at this)

I believe we worked on the active-choices plug-in issue, where we had to include the security script plug-in (SECURITY-255?). So having worked directly with you in the Jenkins security team, and having already dealt with CVE's in other libraries/projects in some companies, I must say that from the moment the issue was assigned to me, until we had the review, release schedule, and finally the review, everything was very professional and simple - from a plugin maintainer POV.

I felt quite bad for having a security issue in my code, but at no moment I felt like I was being blamed or pressured to release it as fast as possible (just to not disclose or leak anything about the issue at hand and carefully test the fix :). So kudos and thank you and all those involved in the security team! You definitely do not suck at this.

>We're always looking for more volunteers! Membership is open to most active contributors. Prerequisites and process at https://jenkins.io/security/#team

Not the most active contributor, but will take a look to see if I can enlist, and maybe try to help in some way :)

Cheers
Bruno
________________________________
From: Daniel Beck <m...@beckweb.net>

To: jenkin...@googlegroups.com
Sent: Tuesday, 11 April 2017 10:55 PM
Subject: Re: Yesterday's security advisory

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/6EA3819C-6BB0-4FB6-898F-DB09549F61B2%40beckweb.net.
For more options, visit https://groups.google.com/d/optout.