We must handle security advisories better

109 views
Skip to first unread message

R. Tyler Croy

unread,
Oct 2, 2014, 11:59:17 AM10/2/14
to jenkin...@googlegroups.com
I already gave Kohsuke these opinions in person yesterday, but I don't think
this is solely his problem so I wanted to bring it up on the dev mailing list.

It's important to me that this is a constructive discussion and we talk about
how we can do things better moving forward.

IMO there's a few problems with the way we have handled security advisories in the
past:

* Low feedback times to researchers who discover vulnerabilities. One example
I found in core, the submitter of the SECURITY issue did not get feedback
for one calendar month on the issue. This is a problem as some security
researchers are of the opinion that if they don't hear back from a vendor
in a timely manner, they should disclose to the public in order to get the
hole closed.

* Lack of transparency to the Jenkins community into the SECURITY project in
JIRA. I'm not of the opinion that SECURITY should be a publicly visible
project in JIRA but we *must* come up with some criteria to give people
access that prevents Kohsuke from being the only one paying attention.

* Vendor notification, by virtue of Kohsuke being a Cloudbees employee they
were aware and able to update in a timely manner, but I do not believe we
communicated with vendors like Shining Panda CI, who rely on publicly
facing Jenkins instances, that there was a big release coming before it was
publicly announced. I don't think we should tell all companies using
Jenkins before a public announcement, but I do think that maintaining a
list of companies and organizations that run Jenkins as a public service
should get a few hours of lead time.


I have some ideas on what we can do to improve this. As some of you may know I
work for Lookout, a mobile security company, and I can probably rope in members
of our research team if need be to provide insight from the security research
perspective, the ones disclosing vulnerabilities, if you all have questions
there.


- R. Tyler Croy

------------------------------------------------------
Code: <https://github.com/rtyler>
Chatter: <https://twitter.com/agentdero>

% gpg --keyserver keys.gnupg.net --recv-key 3F51E16F
------------------------------------------------------

Stephen Connolly

unread,
Oct 2, 2014, 12:22:23 PM10/2/14
to jenkin...@googlegroups.com
On 2 October 2014 16:59, R. Tyler Croy <ty...@monkeypox.org> wrote:
I already gave Kohsuke these opinions in person yesterday, but I don't think
this is solely his problem so I wanted to bring it up on the dev mailing list.

It's important to me that this is a constructive discussion and we talk about
how we can do things better moving forward.

IMO there's a few problems with the way we have handled security advisories in the
past:

  * Low feedback times to researchers who discover vulnerabilities. One example
    I found in core, the submitter of the SECURITY issue did not get feedback
    for one calendar month on the issue. This is a problem as some security
    researchers are of the opinion that if they don't hear back from a vendor
    in a timely manner, they should disclose to the public in order to get the
    hole closed.

Yep. No disagreement that this is an issue.
 

  * Lack of transparency to the Jenkins community into the SECURITY project in
    JIRA.  I'm not of the opinion that SECURITY should be a publicly visible
    project in JIRA but we *must* come up with some criteria to give people
    access that prevents Kohsuke from being the only one paying attention.

There is a mailing list of people... they all seem rather quiet a lot of the time... granted I am one of those people too... so tarring myself with the same brush
 

  * Vendor notification, by virtue of Kohsuke being a Cloudbees employee they
    were aware and able to update in a timely manner, but I do not believe we
    communicated with vendors like Shining Panda CI,

If it is critical to them, then they should ask to be part of the security-cert list. I do not see anyone stopping responsible parties from being advised. OTOH if they are relying on us to do all the donkey work and then complaining that we are a little bit more prepared because of that donkey work... I have slightly less sympathy.

In general I find the community involvement on the jenkins-cert list less vocal than I would like to see.

There are some things I would expect to see more vocal responses to...
 
who rely on publicly
    facing Jenkins instances, that there was a big release coming before it was
    publicly announced. I don't think we should tell all companies using
    Jenkins before a public announcement, but I do think that maintaining a
    list of companies and organizations that run Jenkins as a public service
    should get a few hours of lead time.

If you are on the cert list you would have had all of the LTS RC testing as foreknowledge... at least for the CVE-2014-3666 (if you have not upgraded your Jenkins to a version with the fix for that by the time you read this... stop what you are doing and upgrade it already)

R. Tyler Croy

unread,
Oct 2, 2014, 10:23:55 PM10/2/14
to jenkin...@googlegroups.com
(replies inline)

On Thu, 02 Oct 2014, Stephen Connolly wrote:

> On 2 October 2014 16:59, R. Tyler Croy <ty...@monkeypox.org> wrote:
>
> > I already gave Kohsuke these opinions in person yesterday, but I don't
> > think this is solely his problem so I wanted to bring it up on the dev
> > mailing list.
> >
> > It's important to me that this is a constructive discussion and we talk
> > about how we can do things better moving forward.
> >
> > IMO there's a few problems with the way we have handled security advisories
> > in the past:
> >
> > * Low feedback times to researchers who discover vulnerabilities. One
> > example I found in core, the submitter of the SECURITY issue did not get
> > feedback for one calendar month on the issue. This is a problem as some
> > security researchers are of the opinion that if they don't hear back from
> > a vendor in a timely manner, they should disclose to the public in order
> > to get the hole closed.
> >
>
> Yep. No disagreement that this is an issue.



Do you have any ideas on how we can improve the feedback times? Is it just a
matter of getting more people involved in the SECURITY project?



> > * Lack of transparency to the Jenkins community into the SECURITY project
> > in JIRA. I'm not of the opinion that SECURITY should be a publicly
> > visible project in JIRA but we *must* come up with some criteria to give
> > people access that prevents Kohsuke from being the only one paying
> > attention.
>
> There is a mailing list of people... they all seem rather quiet a lot of
> the time... granted I am one of those people too... so tarring myself with
> the same brush


Here I was thinking I was on, or knew of, every Jenkins mailing list. Is there
another list I'm not aware where security issues are discussed?

On a related note, how are plugin developers who maintain plugins which have
vulnerabilities disclosed against them looped into these conversations?



> > * Vendor notification, by virtue of Kohsuke being a Cloudbees employee
> > they were aware and able to update in a timely manner, but I do not
> > believe we communicated with vendors like Shining Panda CI,
>
>
> If it is critical to them, then they should ask to be part of the
> security-cert list. I do not see anyone stopping responsible parties from
> being advised. OTOH if they are relying on us to do all the donkey work and
> then complaining that we are a little bit more prepared because of that
> donkey work... I have slightly less sympathy.
>
> In general I find the community involvement on the jenkins-cert list less
> vocal than I would like to see.


See previous comments about non-public lists.


I'm not sure we need to do any donkey work to feed updates to vendors relying
on Jenkins, I think having a process where somebody can sign up for pre-release
notifications for public Jenkins instances would be valuable. I would imagine
that the Shining Panda folks, OpenStack, FreeBSD, ASF and a number of other
organizations would gladly fill out a form to subscribe to early access to
security notifications or fixes.


> > who rely on publicly facing Jenkins instances, that there was a big release
> > coming before it was publicly announced. I don't think we should tell all
> > companies using Jenkins before a public announcement, but I do think that
> > maintaining a list of companies and organizations that run Jenkins as a
> > public service should get a few hours of lead time.
> >
>
> If you are on the cert list you would have had all of the LTS RC testing as
> foreknowledge... at least for the CVE-2014-3666 (if you have not upgraded
> your Jenkins to a version with the fix for that by the time you read
> this... stop what you are doing and upgrade it already)



Jesse Glick

unread,
Oct 3, 2014, 12:13:50 AM10/3/14
to Jenkins Dev
On Thu, Oct 2, 2014 at 7:23 PM, R. Tyler Croy <ty...@monkeypox.org> wrote:
> Do you have any ideas on how we can improve the feedback times? Is it just a
> matter of getting more people involved in the SECURITY project?

By all means, if they are motivated to do serious investigations when
issues come in.

> Is there another list I'm not aware where security issues are discussed?

jenkinsci-cert, limited to those people who can also see the SECURITY component.

> On a related note, how are plugin developers who maintain plugins which have
> vulnerabilities disclosed against them looped into these conversations?

Not very well. Currently we cannot even add them on CC to the JIRA
issue, which makes it difficult to handle these.

Daniel Beck

unread,
Oct 5, 2014, 3:49:26 PM10/5/14
to jenkin...@googlegroups.com
Is there something that could be done so that in the process of confirming an issue report, someone takes a look at neighboring code that may also be affected?

I discovered SECURITY-110 (October) because I tried to understand what SECURITY-79 (February) was about (of course, low and medium-but-really-should-be-low impact issues, but still).

SECURITY-93 (low severity, February) looks remarkably similar to SECURITY-138 (critical, October). The former was the job config form (bf53919), the latter the Build With Parameters form (a6a7bec), therefore I assume the difference in severity. But this also means that the low severity issue description explained where a critical severity issue could be found.

Kohsuke Kawaguchi

unread,
Oct 7, 2014, 8:23:15 PM10/7/14
to jenkin...@googlegroups.com
On 10/02/2014 08:59 AM, R. Tyler Croy wrote:
> I already gave Kohsuke these opinions in person yesterday, but I don't think
> this is solely his problem so I wanted to bring it up on the dev mailing list.
>
> It's important to me that this is a constructive discussion and we talk about
> how we can do things better moving forward.
>
> IMO there's a few problems with the way we have handled security advisories in the
> past:
>
> * Low feedback times to researchers who discover vulnerabilities. One example
> I found in core, the submitter of the SECURITY issue did not get feedback
> for one calendar month on the issue. This is a problem as some security
> researchers are of the opinion that if they don't hear back from a vendor
> in a timely manner, they should disclose to the public in order to get the
> hole closed.

Yes, this is true, and the time it takes us to address those issues are
longer than we like.

What we started doing (maybe about a half year ago?) was to form the
"Jenkins CERT" team, who has access to all the issues in the otherwise
private SECURITY project, has access to all the pending fixes as they
are developed, and has a dedicated private email list for discussion.

As Stephen was saying, we didn't see the uptake in the participation to
fixing issues from people outside the usual suspects (Jesse, Stephen,
myself, etc), but the invitation is open to anyone who wants to help.
The only criteria currently is that you have the core commit access
(which means filing CLA), which comes from the fact that you'll be
making changes in the core.

But we actually came a long way in handling them, ranging from a
branching strategy, coordinating with LTS release testing, and so on.
We also recently approved in the project meeting to get GitHub private
repos for the Jenkins CERT team. I think it'll improve our ability to
cooperate.

I think we clearly need to document this better, though. I don't think
even you were aware of this, and that speaks something.



> * Lack of transparency to the Jenkins community into the SECURITY project in
> JIRA. I'm not of the opinion that SECURITY should be a publicly visible
> project in JIRA but we *must* come up with some criteria to give people
> access that prevents Kohsuke from being the only one paying attention.

I covered this above, it's certainly more than just myself already.
Jesse does the heavy lifting of the fixes in many cases.


> * Vendor notification, by virtue of Kohsuke being a Cloudbees employee they
> were aware and able to update in a timely manner, but I do not believe we
> communicated with vendors like Shining Panda CI, who rely on publicly
> facing Jenkins instances, that there was a big release coming before it was
> publicly announced. I don't think we should tell all companies using
> Jenkins before a public announcement, but I do think that maintaining a
> list of companies and organizations that run Jenkins as a public service
> should get a few hours of lead time.

Hopefully our documenting the Jenkins CERT team bit more helps attract
vendors that are affected. Shining Panda seems to have stopped the CI
service [1], though. That's too bad.


The other struggle we haven't touched on is how to coordinate and
deliver security fixes to plugins. A part of this is fairly mechanical,
such as adding plugin developers to individual JIRA issues selectively.



> I have some ideas on what we can do to improve this. As some of you may know I
> work for Lookout, a mobile security company, and I can probably rope in members
> of our research team if need be to provide insight from the security research
> perspective, the ones disclosing vulnerabilities, if you all have questions
> there.
>
>
> - R. Tyler Croy
>
> ------------------------------------------------------
> Code: <https://github.com/rtyler>
> Chatter: <https://twitter.com/agentdero>
>
> % gpg --keyserver keys.gnupg.net --recv-key 3F51E16F
> ------------------------------------------------------
>

[1] http://shiningpanda.com/shiningpanda-ci-clap-de-fin.html
--
Kohsuke Kawaguchi http://kohsuke.org/

Slide

unread,
Oct 7, 2014, 9:57:14 PM10/7/14
to Jenkins Developer List

The big issue I've seen is when there are security issues with plugins, the plugin developer is not brought into the mix quickly enough, or at all and the person who files the issue gets tired of waiting for the issue to be fixed and just files a normal issue. Something needs to be done to involve plugin developers quickly, even if they don't have access to the security area.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Kohsuke Kawaguchi

unread,
Oct 8, 2014, 4:55:49 PM10/8/14
to jenkin...@googlegroups.com

Agreed.

This hinges on ability to add individuals to specific issues (sorta like
watchers). This does seem doable,

- https://answers.atlassian.com/questions/16088/a
- https://answers.atlassian.com/questions/283448/a

Is there anyone familiar with JIRA who knows how to do this?

On 10/07/2014 06:57 PM, Slide wrote:
> The big issue I've seen is when there are security issues with plugins,
> the plugin developer is not brought into the mix quickly enough, or at
> all and the person who files the issue gets tired of waiting for the
> issue to be fixed and just files a normal issue. Something needs to be
> done to involve plugin developers quickly, even if they don't have
> access to the security area.
>
> On Oct 7, 2014 5:23 PM, "Kohsuke Kawaguchi" <k...@kohsuke.org
> ------------------------------__------------------------
> Code: <https://github.com/rtyler>
> Chatter: <https://twitter.com/agentdero__>
>
> % gpg --keyserver keys.gnupg.net <http://keys.gnupg.net>
> --recv-key 3F51E16F
> ------------------------------__------------------------
>
>
> [1] http://shiningpanda.com/__shiningpanda-ci-clap-de-fin.__html
> <http://shiningpanda.com/shiningpanda-ci-clap-de-fin.html>
> --
> Kohsuke Kawaguchi http://kohsuke.org/
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to jenkinsci-dev+unsubscribe@__googlegroups.com
> <mailto:jenkinsci-dev%2Bunsu...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jenkinsci-de...@googlegroups.com
> <mailto:jenkinsci-de...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.


--
Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
Try Jenkins Enterprise, our professional version of Jenkins

Christopher Orr

unread,
Mar 27, 2015, 5:10:07 AM3/27/15
to jenkin...@googlegroups.com
Have there been any changes here with regards to plugin owner access?


I was also a bit surprised in general to see that SECURITY issues do not
become publicly-visible after the security advisory is made public.

I've come across core/plugin changes recently which referred to
SECURITY-*, but when I searched for the issues in JIRA, I didn't have
permission to view it, which makes it (slightly) harder to look out for
that type of issue elsewhere.

The security advisories themselves tend to be very short, making it hard
to make a judgement on how urgent it really is to update (e.g. if I
don't allow anonymous access, or my instance isn't publicly-visible,
perhaps certain issues aren't critical). So having access to the
SECURITY-* issue itself would be nice; not everyone can read the git
diff to figure out what the problem is.

Regards,
Chris

Jesse Glick

unread,
Mar 27, 2015, 7:46:40 AM3/27/15
to Jenkins Dev
On Fri, Mar 27, 2015 at 5:09 AM, Christopher Orr <ch...@orr.me.uk> wrote:
> The security advisories themselves tend to be very short, making it hard to
> make a judgement on how urgent it really is to update (e.g. if I don't allow
> anonymous access, or my instance isn't publicly-visible, perhaps certain
> issues aren't critical).

If this kind of judgment is not possible given the current advisories,
then that is a problem in the advisories. The intent is that
everything you need to know to judge whether or not *you* need to
accept this update is contained within the text of the advisory.

Kohsuke Kawaguchi

unread,
Mar 27, 2015, 9:42:32 PM3/27/15
to jenkin...@googlegroups.com
The main obstacle in making issues visible is our lack of knowledge of how to configure JIRA to do that. It is rumored to be doable, but it is non trivial, and someone needs to spend some time to figure that out.

The inability to control who can see SECURITY-* issues on individual issue basis is causing a lot of other problems, such as merging two duplicate issues together, or tracking vulnerabilities in plugins. Disclosure of vulnerabilities is a big one, too.

Incidentally I've been lately working on containerizing JIRA to simplify upgrades, and as a side effect of this, it's a lot easier now to create a clone of issues.jenkins-ci.org and experiment with the potentially dangerous setting changes like this.


I agree with Jesse that advisories not informative is a separate problem. If you or anyone knows some better written advisories that we should mimic, please let me know. When I originally started doing it, I looked around and just tried to follow the model, I do remember looking at advisories from Atlassian, like this one.

I do try to cover who can mount an attack (an attack that can be done by anonymous user vs an attack that requires a certain existing permission in Jenkins), Whether it's safe for people who are running Jenkins inside firewall is a bit strange question to me --- if you trust people who have access to your network, then you don't care about any vulnerabilities, no?

But perhaps there are several important questions like that, for which we can provide Yes/No answers.


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr39BSSjXSoF3mbYi2LF1PpKyVmxmY1571APuJxNt_BvjA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Kohsuke Kawaguchi

Arnaud Héritier

unread,
Aug 17, 2015, 6:36:48 AM8/17/15
to jenkin...@googlegroups.com
Hi,

  I missed this discussion and found it by looking some details about the process to manage security issues.
  I'm not sure if a need was really defined but to restrict the visibility of an issue (or only a comment) we can just define in Jira a "security level" to apply on issues. 
  A security level is something you define on top of the project/issues permissions to restrict the visibility of an issue or comment

Arnaud 


For more options, visit https://groups.google.com/d/optout.



--
-----
Arnaud Héritier
Mail/GTalk: aheritier AT gmail DOT com
Twitter/Skype : aheritier

Kanstantsin Shautsou

unread,
Aug 17, 2015, 1:45:32 PM8/17/15
to Jenkins Developers


On Monday, August 17, 2015 at 1:36:48 PM UTC+3, Arnaud Héritier wrote:
Hi,

  I missed this discussion and found it by looking some details about the process to manage security issues.
  I'm not sure if a need was really defined but to restrict the visibility of an issue (or only a comment) we can just define in Jira a "security level" to apply on issues. 
  A security level is something you define on top of the project/issues permissions to restrict the visibility of an issue or comment

Arnaud 
Does it require right configured groups in jira? 

Arnaud Héritier

unread,
Aug 17, 2015, 5:01:51 PM8/17/15
to jenkin...@googlegroups.com
Jira is using a RBAC like model (roles - groups users)

For each project you have various roles in which you add user(s) and/or group(s)

In permissions schemes you define for each permission (browse project, open issue, ...) which project role(s), group(s) and/or user(s) are allowed

In security schemes you define which project role(s), group(s) and/or user(s) are allowed to browse an issue or a comment. The security scheme has a default configuration and is supposed to restrict the visibility provided by the permission "browse project" (perhaps "view comment", I don't remember if it has specific permission). The ability to change the security level is a distinct permission this you can give this ability to a subset of people. 
Reply all
Reply to author
Forward
0 new messages