Changes to Security Severity Ratings

[Curtis Koenig]

We've made some changes to the Security Severity Ratings^[1] , for the
most part this is cosmetic with one exception. As of today we ask that
the use of '_sg:<rating>*?*_' no longer be used. If there is question
about the rating level of an bug please make your best estimate, if the
bug needs to be altered alter due to new information then we will do so
at the appropriate time. However, the use of ratings with '?' has become
confusing and conveys an ambiguity to our development partners that is
not helpful.

We will be working to evaluate all '?' ratings and put them in to one of
the standard sg<rating> categories. As always if anyone disagrees with
the rating of a bug then we are open to discussion and modification of
that rating.

Notable changes to the page are:
* Removal of the table for the main ratings
** This was confusing to some and placed an emphasis on parts of the
page we felt should be moved, for the most part this is just a change in
formatting
* Addition of bugzilla security keywords and whiteboard items for tracking
* Addition of Feature Page security codes we are using for tracking

--curtis

[1] https://wiki.mozilla.org/Security_Severity_Ratings

[Brian Smith]

Curtis Koenig wrote:
> We've made some changes to the Security Severity Ratings [1] , for the

> most part this is cosmetic with one exception. As of today we ask that

> the use of ' sg:<rating> ? ' no longer be used. If there is question

> about the rating level of an bug please make your best estimate, if
> the bug needs to be altered alter due to new information then we will
> do so at the appropriate time. However, the use of ratings with '?'
> has become confusing and conveys an ambiguity to our development
> partners that is not helpful.

Which partners?

I assume people on secteam hate the sg:<rating>? marker and will either remove the question mark (thus verifying that they agree with me), change the rating (indicating that they disagree with me), or leave it alone (presumably, ignoring the bug, or indicating that they need to communicate with me to determine what the severity should be). I find this (lack of) feedback from secteam to be very useful information to me as a developer.

I think it is unrealistic for all our contributors (including us paid developers) to be able to accurately rate the severity of bugs on their own, with (nearly) zero training. Further, people make mistakes and there should be more than one person involved in the rating. (That is the point of the sg:<rating>? scheme I mentioned above.) And, finally, it is tempting to downplay the severity of a bug if having a high severity means I have to stop doing the fun work I am doing (and/or miss a deadline) to deal with some security drudgery; some process for ensuring secteam has seen the rating

So, if anything, I would prefer to formalize the sg:<rating>? scheme like we have done for tracking-firefoxX and status-firefoxX flags, and hopefully make it easier for secteam to help negotiate the severity with the developers & contributors of the affected component.

Cheers,
Brian

[Jesse Ruderman]

A sec-triage-needed keyword/flag might be better. [sg:critical?] has
historically meant that the security team isn't sure it's exploitable,
and not everyone will realize the meaning has changed.

> _______________________________________________
> Security-group mailing list
> https://mail.mozilla.org/listinfo/security-group
>

[chris hofmann]

Part of the reasoning around sg:crit? was to help us get out of
unproductive debate and deep research into the criticality, and keep the
focus on just getting the bugs of that class fixed. e.g: "There is
enough evidence here that this could be turned into critical explotable
bug, but no one is going to do that extra level of work, lets just get
the bug fixed quick and move on..."

My fear is that we will now be reverting to a system that needs to
remove all ambiguity about the exploitability, which could leave bugs in
triage longer, and lengthen the cycle between discovery and getting the
fix in the hands of our users.

By "our development partners" I'm assuming you mean the other other
engineers working on the project who also must make trade offs between
time spend evaluating the criticality of the bugs, v. just fixing the
bug. I'm not sure how this serves them either.

-chofmann

>> So, if anything, I would prefer to formalize the sg:<rating>? scheme like we have done for tracking-firefoxX and status-firefoxX flags, and hopefully make it easier for secteam to help negotiate the severity with the developers& contributors of the affected component.

>>
>> Cheers,
>> Brian
>> _______________________________________________
>> Security-group mailing list
>> https://mail.mozilla.org/listinfo/security-group
>>

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

[Daniel Veditz]

On 11/1/11 5:47 PM, Brian Smith wrote:
> I think it is unrealistic for all our contributors (including us
> paid developers) to be able to accurately rate the severity of
> bugs on their own, with (nearly) zero training.

Agreed. If the guideline page doesn't clear it up just mark it a
security bug with no rating and the triage team will get to it.

People can also always mail secu...@mozilla.org if they have a
question about whether a bug is or isn't a security bug, or join us
on the #security IRC channel.

> Further, people make mistakes and there should be more than one
> person involved in the rating.

Highly rated bugs get looked at by lots of people. The mistake to
worry about is rating a bug too low.