*DRAFT* Security Bug Approval Process

[Al Billings]

Hi,

This is a proposal for approving checkins for security bugs that was previously discussed in the security-group within Mozilla. This has been discussed a bit with release management and those of us who do security bug triage (and approval of checkins) a bit. I wanted to circulate this winder to dev-planning and dev-security now.

The overall goal of this process is to avoid accidental disclosure of security issues through checkins and to make it very easy for developers to know when they are clear to go ahead and check in a security fix.

The proposal below is likely to become the plan of record unless significant objections are made to it. We've had enough issues and questions raised in the last few months that the need for a change has become pretty clear to those of us working with this day in and day out.

Please reply with concerns and feedback, either to me or to the group(s).

Al

-----

This is a revisit of the Security Bug Approval proposal from early June. Having spoken to Alex about this a bit, he has been wanting to get this in place and I think the security assurance team is in agreement on this
as well. We've had a number of close calls and potential exposures during the past few release cycles.

We have had a problem in people knowing when they should land high impact or serious security issues on Mozilla Central. There are concerns that landing at the wrong time exposes us to a risk of a Zero Day or similar problems. Developers have said that they both (a) don't want a lot more process but also (b) need to know when it is fine to land a security fix without serious risk.

We have a draft proposal and some action items to try to address the lack of clarity for security bug landing. We explicitly wanted to set up a process that was lightweight for developers. This should be low overhead, just as is landing fixes on ESR.

This process only applies to core-security bugs and not web related bugs or public (non-hidden) bugs.

*Proposal*

The proposal is that we create a new flag, 'sec-approval', which will have ?, +, and - states.

This flag will apply to higher severity security bugs ONLY.

Core-security bug fixes should just be landed by a developer without any explicit approval if:

1. The bug has a sec-low, sec-moderate, sec-other, or sec-want rating.

*or*

2. The bug is a recent regression on mozilla-central. This means that the developer can mark the status flags for ESR, Beta, and Aurora as "unaffected." It also means that we haven't shipped anywhere public in an official release yet.

Otherwise, if the bug has a patch *and* is sec-high or sec-critical, the developer should set the sec-approval flag to '?'.

If the bug is a hidden core-security bug with no rating then either:

a) request sec-approval and wait for a rating, or

b) rate it and then proceed according to whether the bug is low/moderate or high/critical

If developers are unsure about a bug and it has a patch ready, just mark the sec-approval flag to '?' and move on. Don't overthink it!

An automatic nomination comment will be added to bugzilla when sec-approval is set to '?' and this will ask:

a. How obvious is this problem (from reading through the patch)? Can it be deduced by a third party from it?

b. How risky is the fix?

c. What versions of Firefox are affected and to which does the patch apply cleanly?

d. A link to wiki page with details of this process for clarity for developers.

This is similar to ESR approval nomination form.

When the bug is approved for landing, the sec-approval flag will be set to '+' with a comment from the approver to land the patch. At that point, developers can land the patch. This is done on the honor system. There will be no programmatic enforcement of needing the sec-approval flag set to '+' to land.

This will allow us to control when we can land security bugs without exposing them too early and to make sure they get landed on the various branches.

The security assurance team and release management will have their own process for approving bugs:

1. Security assurance team goes through sec-approval ? bugs daily and approves low risk fixes for central (if early in cycle). Developers can also ping us in #security on IRC when important.

2. Security team marks tracking flags to ? for all affected versions when approved for central. (This allows release management to decide whether to uplift to branches.)

3. Weekly security/release management triage meeting goes through sec-approval + and ? bugs where beta and ESR is affected, ? bugs with higher risk (sec-high and sec-critical), or ? bugs near end of cycle.

Please send further thoughts, comments, or concerns about this for discussion.

Al

--
Program Manager
Mozilla Security Team

[Justin Lebar]

It seems like I can't push my security bug to tryserver until after I
get approval, since that would have the same problem as landing on
m-c.

That would make life pretty difficult, since I'd need to write the
patch, get a review, wait a week for approval, land on try, and only
then discover that my patch breaks something.

Three similar sg-high's I've dealt with which have needed tryserver
coverage are bug 775009, bug 757376, and bug 687745. (For those not
in the security group, these are all docshell bugs, one of which is
currently un-resolved.) I don't deal regularly with security bugs, so
I don't know whether I'm an edge case here.

-Justin

> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning

[Al Billings]

On 8/22/12 3:23 PM, Justin Lebar wrote:
> It seems like I can't push my security bug to tryserver until after I
> get approval, since that would have the same problem as landing on
> m-c.

This is currently a problem since the proposal codifies a way of dealing
with our current issue, which is accidental exposure. In other words, we
already run the risk of exposure by pushing to tryserver, whether this
new policy is adopted or not (just as we do when people check into
mozilla-central with a security bug at a bad time in the cycle).

[Gavin Sharp]

I don't think this policy should directly affect your pushes to try
server. The goal of the policy is to prevent us from landing security
patches on mozilla-central at inconvenient times.

But, it does raise a separate point that we might need to discuss
further: pushing security patches (and tests) to try has many of the
same security implications that pushing them to to the main
repositories does (with slightly different visibility
characteristics). Developers working on security bugs need to be wary
of that, and need to know that they should take appropriate
precautions (e.g. don't list the bug #, don't include comments, use an
innocuous summary, etc.).

Gavin

On Wed, Aug 22, 2012 at 3:23 PM, Justin Lebar <justin...@gmail.com> wrote:
> It seems like I can't push my security bug to tryserver until after I
> get approval, since that would have the same problem as landing on
> m-c.
>

>> Al
>>
>> --
>> Program Manager
>> Mozilla Security Team
>>

[Al Billings]

On 8/22/12 7:23 PM, Ehsan Akhgari wrote:
> Is there going to be a triage of the approved but unfixed bugs? I'm
> worried that we might miss bugs which are waiting for approval for a
> few weeks in some cases (such as if the developer goes on vacation,
> for example.)

That's a good idea and we can definitely do that.

> Also, there is a chance that a patch will bitrot if it waits for
> approval for a few weeks. Are we planning to include enough time for
> people to potentially fix up their patches against the recent changes,
> get try server results, etc.?

This is a human driven process. So, if someone says, "Oh, you gave me
approval but my patch is out of date now, can I take a week to update
it?", I don't think any rational person involved (like me) is going to
say that you cannot do so.

This isn't a stick with which to hit people. The overall goal is simply
to avoid accidental exposure of security issues before their time, so we
can shepherd when things go in a bit better. I think it will wind up
being relatively flexible and straightforward for folks.

[Ehsan Akhgari]

On 12-08-23 12:31 PM, Al Billings wrote:
>> Also, there is a chance that a patch will bitrot if it waits for
>> approval for a few weeks. Are we planning to include enough time for
>> people to potentially fix up their patches against the recent changes,
>> get try server results, etc.?
>
> This is a human driven process. So, if someone says, "Oh, you gave me
> approval but my patch is out of date now, can I take a week to update
> it?", I don't think any rational person involved (like me) is going to
> say that you cannot do so.
>
> This isn't a stick with which to hit people. The overall goal is simply
> to avoid accidental exposure of security issues before their time, so we
> can shepherd when things go in a bit better. I think it will wind up
> being relatively flexible and straightforward for folks.

Oh, it seems like I failed to articulate properly here. I was asking
about whether the security team is going to include a buffer before a
given release code freeze or uplift when approving patches. The
scenario that I am worrying about is the security team approving the
patch 3 days before a code freeze and the developer being on vacation at
that time.

Hope that's clearer. :-)

Cheers,
Ehsan

[Al Billings]

On 8/23/12 9:56 AM, Ehsan Akhgari wrote:
>
> Oh, it seems like I failed to articulate properly here. I was asking
> about whether the security team is going to include a buffer before a
> given release code freeze or uplift when approving patches. The
> scenario that I am worrying about is the security team approving the
> patch 3 days before a code freeze and the developer being on vacation
> at that time.
>
> Hope that's clearer. :-)

Oh, well, yes. We'd better. :-)

I don't think we're going to quite rush them all in. There is a
balancing act between "OMG, we checked it in and it is semi-public!" and
"OMG, we checked it in two days before building and now the build is
toast" for stability.

[Mats Palmgren]

On 08/23/2012 01:14 AM, Gavin Sharp wrote:
> I don't think this policy should directly affect your pushes to try
> server. The goal of the policy is to prevent us from landing security
> patches on mozilla-central at inconvenient times.

I think the proposed policy is pointless without addressing the same
exposure of pushes to Try.

> But, it does raise a separate point that we might need to discuss

> further: pushing security patches (and tests) to try ...


You should *not* push tests for security sensitive bugs to Try (or
anywhere else) before the bug is made public (and not even then in
some cases).

> Developers working on security bugs need to be wary
> of that, and need to know that they should take appropriate
> precautions (e.g. don't list the bug #, don't include comments, use an
> innocuous summary, etc.).

That's good advice, but quite often just the code changes leaks
enough information to give a good hint at what the problem is...
and a push to Try without a bug number is as telling as having
a number to cross-reference with Bugzilla to confirm it's hidden.


/Mats

[Gavin Sharp]

On Thu, Aug 23, 2012 at 10:36 AM, Mats Palmgren <mat...@gmail.com> wrote:
> I think the proposed policy is pointless without addressing the same
> exposure of pushes to Try.

That's a good example of the "perfect solution fallacy":
http://en.wikipedia.org/wiki/Nirvana_fallacy#Perfect_solution_fallacy

As I mentioned in my original post, posting patches to Try has
different visibility characteristics than pushing to mozilla-central
(people push all sorts of experimental junk to try, so mining it for
security bugs is harder, particularly if people are cautious with what
they push). So fixing the problem for mozilla-central has value even
if we don't fix the problem for Try.

> You should *not* push tests for security sensitive bugs to Try (or
> anywhere else) before the bug is made public (and not even then in
> some cases).

Generally I think this is sound advice, but there are cases where it
may be the right tradeoff to push such tests to try (e.g. if it helps
you debug the problem).

> and a push to Try without a bug number is as telling as having
> a number to cross-reference with Bugzilla to confirm it's hidden.

No, I don't think it's "as telling", in general (people push lots of
non-security patches to try without bug numbers). Omitting security
bug numbers is a thing we can do to make mining public repositories
for security issues harder, so we should do it, even if it's not going
to be a perfect solution.

Gavin

[Lukas Blakk]

On Aug 23, 2012, at 10:56 AM, Gavin Sharp <ga...@gavinsharp.com> wrote:

> On Thu, Aug 23, 2012 at 10:36 AM, Mats Palmgren <mat...@gmail.com> wrote:

>> I think the proposed policy is pointless without addressing the same
>> exposure of pushes to Try.
>

> That's a good example of the "perfect solution fallacy":
> http://en.wikipedia.org/wiki/Nirvana_fallacy#Perfect_solution_fallacy
>
> As I mentioned in my original post, posting patches to Try has
> different visibility characteristics than pushing to mozilla-central
> (people push all sorts of experimental junk to try, so mining it for
> security bugs is harder, particularly if people are cautious with what
> they push). So fixing the problem for mozilla-central has value even
> if we don't fix the problem for Try.

Also the try repo gets clobbered/reset on a completely random basis so the builds are erased after 14 days, and the code is also not around for long (perhaps a few months at most?).

-Lukas

[Ehsan Akhgari]

On 12-08-23 2:13 PM, Lukas Blakk wrote:
>
> On Aug 23, 2012, at 10:56 AM, Gavin Sharp <ga...@gavinsharp.com> wrote:
>
>> On Thu, Aug 23, 2012 at 10:36 AM, Mats Palmgren <mat...@gmail.com> wrote:

>>> I think the proposed policy is pointless without addressing the same
>>> exposure of pushes to Try.
>>

>> That's a good example of the "perfect solution fallacy":
>> http://en.wikipedia.org/wiki/Nirvana_fallacy#Perfect_solution_fallacy
>>
>> As I mentioned in my original post, posting patches to Try has
>> different visibility characteristics than pushing to mozilla-central
>> (people push all sorts of experimental junk to try, so mining it for
>> security bugs is harder, particularly if people are cautious with what
>> they push). So fixing the problem for mozilla-central has value even
>> if we don't fix the problem for Try.
>
> Also the try repo gets clobbered/reset on a completely random basis so the builds are erased after 14 days, and the code is also not around for long (perhaps a few months at most?).

Setting up a non-clobbering clone of try doesn't take more than 10
minutes, so this is not really relevant, but on the other points I
completely agree with Gavin.

Cheers,
Ehsan

[Ryan VanderMeulen]

On 8/22/2012 6:23 PM, Justin Lebar wrote:
> It seems like I can't push my security bug to tryserver until after I
> get approval, since that would have the same problem as landing on
> m-c.
>
> That would make life pretty difficult, since I'd need to write the
> patch, get a review, wait a week for approval, land on try, and only
> then discover that my patch breaks something.
>
> Three similar sg-high's I've dealt with which have needed tryserver
> coverage are bug 775009, bug 757376, and bug 687745. (For those not
> in the security group, these are all docshell bugs, one of which is
> currently un-resolved.) I don't deal regularly with security bugs, so
> I don't know whether I'm an edge case here.
>
> -Justin
>

(Since my original reply to dev.security appears to be caught in limbo...)

Why not just create a Try-sg that requires ldap access to view/clone?

[Justin Lebar]

> (Since my original reply to dev.security appears to be caught in limbo...)
>
> Why not just create a Try-sg that requires ldap access to view/clone?

FWIW, I think Gavin is probably right that we can mitigate a lot of
risk by obfuscating our try pushes' commit messages. So it's not
clear to me whether this would be worth the effort. (It might be!)

[Mats Palmgren]

On 08/23/2012 07:56 PM, Gavin Sharp wrote:
> As I mentioned in my original post, posting patches to Try has
> different visibility characteristics than pushing to mozilla-central
> (people push all sorts of experimental junk to try, so mining it for
> security bugs is harder, particularly if people are cautious with what
> they push).

I don't think this obscurity results in any security whatsoever.
Looking at our current use of Try, I believe I could automate
sifting through pushes for security fixes to a point where manually
analyzing the result would be no burden at all.

> there are cases where it
> may be the right tradeoff to push such tests to try (e.g. if it helps
> you debug the problem).

For this rare case, surely you must have access to all platforms
internally for testing without having to expose the test on Try?


/Mats

[Gavin Sharp]

On Thu, Aug 23, 2012 at 4:51 PM, Mats Palmgren <mat...@gmail.com> wrote:
> I don't think this obscurity results in any security whatsoever.
> Looking at our current use of Try, I believe I could automate
> sifting through pushes for security fixes to a point where manually
> analyzing the result would be no burden at all.

Maybe so. I'm a little skeptical. But either way, I don't think that
very black/white view of security is the most effective way to think
about this problem. Rather than asking "is it possible to defeat the
security?", I think it's more useful to ask "will the security be
defeated in practice, and how often?". There are things that we can
do, short of eliminating all use of Try for security bugs, that will
make the discovery of security bugs via Try pushes harder, and thus
less likely to occur in practice. I think those things are worth
exploring because the alternative (forbid all use of try for security
bugs) is impossible to enforce and could well end up causing more
trouble than it solves.

> For this rare case, surely you must have access to all platforms
> internally for testing without having to expose the test on Try?

Yes, obviously there are alternatives to using the Try Server that
could be used in almost all cases. But it remains true that people use
Try because it's useful and convenient, and we need to balance that
utility and convenience against security - neither should be treated
as an absolute requirement.

Gavin

[Justin Dolske]

On 8/23/12 5:39 PM, Gavin Sharp wrote:

>> I don't think this obscurity results in any security whatsoever.
>> Looking at our current use of Try, I believe I could automate
>> sifting through pushes for security fixes to a point where manually
>> analyzing the result would be no burden at all.
>
> Maybe so. I'm a little skeptical. But either way, I don't think that
> very black/white view of security is the most effective way to think
> about this problem. Rather than asking "is it possible to defeat the
> security?", I think it's more useful to ask "will the security be
> defeated in practice, and how often?".

Agreed. Security is about risk reduction.

> There are things that we can
> do, short of eliminating all use of Try for security bugs, that will
> make the discovery of security bugs via Try pushes harder, and thus
> less likely to occur in practice.

I suspect the volume on Try is would present a pretty high barrier for
most people snooping around. Sure, it's possible to filter it, but
that's yet more work for someone to do.

OTOH, it's another risk area we should definitely look at... I wonder
how hard it would be to hide pushes with a "[security]" tag, or even
create a complete (but smaller) try-security clone of the Try
infrastructure. But I that can be done independently of the current
Approval proposal.

Justin