Policy 2.7.1: MRSP Issue #187: Require disclosure of incidents in Audit Reports

[Ben Wilson]

The purpose of this email is to begin public discussion on the addition of
a subsection 11 to section 3.1.4 of the Mozilla Root Store Policy. Issue
#187 <https://github.com/mozilla/pkipolicy/issues/187> in GitHub proposes
to require audit reports to list all incidents occurring (or open) during
the audit period of which the auditor has been made aware or to state that
the auditor is unaware of any incidents. This is related to Issue #154
<https://github.com/mozilla/pkipolicy/issues/154> (management assertion
disclosures). That proposal is to have section 2.4 read as follows: "If
being audited to the WebTrust criteria, the Management Assertion letter
MUST include all known incidents that occurred or were still
open/unresolved at any time during the audit period."

Proposed language may be found in the following commits:

-
https://github.com/BenWilson-Mozilla/pkipolicy/commit/f6639f503b743aae402dc0f4841dc3dd5ba88753
-
https://github.com/BenWilson-Mozilla/pkipolicy/commit/6c07c44e4db473dc4d34009f1bc955a0e18cb4c1
-
https://github.com/BenWilson-Mozilla/pkipolicy/commit/5dec00e53b4c6361d85af7644660fe185fcf463d

Proposed language for section 3.1.4 is:

"11. all incidents (as defined in section 2.4) that occurred or were still
open/unresolved at any time during the audit period, or a statement that
the auditor is unaware of any;"

I look forward to your comments, suggestions and discussions.

Ben

[Matthias van de Meent]

On Thu, 22 Oct 2020, 20:53 Ben Wilson via dev-security-policy,
<dev-secur...@lists.mozilla.org> wrote:
> That proposal is to have section 2.4 read as follows: "If
> being audited to the WebTrust criteria, the Management Assertion letter
> MUST include all known incidents that occurred or were still
> open/unresolved at any time during the audit period."
>

> [...]

>
> Proposed language for section 3.1.4 is:
>
> "11. all incidents (as defined in section 2.4) that occurred or were still
> open/unresolved at any time during the audit period, or a statement that
> the auditor is unaware of any;"
>
> I look forward to your comments, suggestions and discussions.

The current MRSP do not bind the requirements on the reporting of
incidents to the CA that the incident was filed on, but generally to
CAs.

Section 2.4 has the general requirement for a CA to report any
incident (which is a failure to comply with the MRSP by any CA). So,
if a CA is aware of an incident with another CA which is included in
the Mozilla root store, that must be reported, and I agree with that.

But, the requirements also extend to having to regularly update these
incidents, and report these incidents in their audit letter, even if
they are not related to that CA.

I suggest to update this wording, and clarify these requirements, to
only include incidents that occurred within the CA's certificate
hierarchy, e.g. "11. all incidents (as defined in section 2.4) that
occurred or were still ..." -> "11. all incidents (as defined in
section 2.4) _within the CA's trust hierarchy_ that occurred or were
still ...".

I believe the same comment applies to issue #154, and to the
requirements in section 2.4, excluding the requirement to file a
report for incidents when discovered.

-Matthias

[Ryan Sleevi]

On Fri, Oct 23, 2020 at 8:55 AM Matthias van de Meent via
dev-security-policy <dev-secur...@lists.mozilla.org> wrote:

> The current MRSP do not bind the requirements on the reporting of
> incidents to the CA that the incident was filed on, but generally to
> CAs.
>
> Section 2.4 has the general requirement for a CA to report any
> incident (which is a failure to comply with the MRSP by any CA). So,
> if a CA is aware of an incident with another CA which is included in
> the Mozilla root store, that must be reported, and I agree with that.
>

This sounds like an overly broad reading of Mozilla Policy, and it's not
clear to me how you reached it. Could you walk me through the language and
help me understand how you reached that conclusion?

It would seem like you might be reaching that conclusion from "When a CA"
and "CAs", is that right?

> But, the requirements also extend to having to regularly update these
> incidents, and report these incidents in their audit letter, even if
> they are not related to that CA.
>

As mentioned above, this seems like an overly broad reading, and I'm
wondering if that's the source of confusion here. Understandably, it would
make no logical sense to expect a third-party reporter to provide updates
for a CA incident, whether that third-party is an individual or another CA.

By the logic being applied here, the ultimate sentence in that same
paragraph would imply that, from the moment a CA incident is filed, all CAs
in Mozilla's Root Program must stop issuance until the affected CA has
resolved the issue, which certainly makes no logical or syntactical sense,
or, similarly, that Section 4.2 of the policy obligates CAs to respond on
behalf of other CAs.

[Matthias van de Meent]

On Fri, 23 Oct 2020 at 17:33, Ryan Sleevi <ry...@sleevi.com> wrote:
>
> On Fri, Oct 23, 2020 at 8:55 AM Matthias van de Meent via dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
>>
>> The current MRSP do not bind the requirements on the reporting of
>> incidents to the CA that the incident was filed on, but generally to
>> CAs.
>>
>> Section 2.4 has the general requirement for a CA to report any
>> incident (which is a failure to comply with the MRSP by any CA). So,
>> if a CA is aware of an incident with another CA which is included in
>> the Mozilla root store, that must be reported, and I agree with that.
>
>
> This sounds like an overly broad reading of Mozilla Policy, and it's not clear to me how you reached it. Could you walk me through the language and help me understand how you reached that conclusion?

Section 2.4 specifies an incident as 'when _a_ CA fails to comply with
any requirement of this policy'. So, an incident is any CA having a
problem.
The next sentence reads that "At a minimum, CAs MUST promptly report
all incidents to Mozilla ...". This can (quite reasonably) be
interpreted as that whenever a CA finds out that any CA fails to
comply with any requirement of the policy, this failure to comply must
be promptly reported to Mozilla.

This is of course not applied this way, but this is quite the
reasonable requirement. Only for the requirements after that the
requirements are becoming more unreasonable for CAs that are not part
of the incident.

> It would seem like you might be reaching that conclusion from "When a CA" and "CAs", is that right?

Correct.

>
>>
>> But, the requirements also extend to having to regularly update these
>> incidents, and report these incidents in their audit letter, even if
>> they are not related to that CA.
>
>
> As mentioned above, this seems like an overly broad reading, and I'm wondering if that's the source of confusion here. Understandably, it would make no logical sense to expect a third-party reporter to provide updates for a CA incident, whether that third-party is an individual or another CA.
>
> By the logic being applied here, the ultimate sentence in that same paragraph would imply that, from the moment a CA incident is filed, all CAs in Mozilla's Root Program must stop issuance until the affected CA has resolved the issue, which certainly makes no logical or syntactical sense, or, similarly, that Section 4.2 of the policy obligates CAs to respond on behalf of other CAs.

Yes, this is indeed the overly broad (and impractical) reading that I
suggest to be updated to be more specific and clear about the intent.
Note that I would like to keep the specific requirement for CAs to
report newly found non-compliances by any CA in the Mozilla root store
to Mozilla, even if this non-compliance originates outside the CAs own
trust hierarchy.

For section 2.4, it might also be as easy as replacing 'CAs' with 'the
CA', as that would point to the CA of 'When a CA' instead of all CAs
in the program.


-Matthias

[Jeff Ward]

Thanks for bringing this up Ben. It is important to consider this requirement in conjunction with #154 and address them together. It seems reasonable to require a CA to disclose all known incidents that are applicable during a given period. It would be important, however, to define “known incident” as a “verified bug” and exclude items such as bugs closed as a duplicate, invalid, etc. It would also make sense to clarify that an incident should only be disclosed once and eliminate duplication when an incident spans two audit periods.

Also keep in mind an auditor typically issues an opinion on management’s assertion of its controls. Audit opinions do not make negative assurance statements, such as not being aware of any incidents during the period. If the CA is required to make this assertion, the auditor’s opinion will consider that statement.

Thanks,

Jeff

[Ben Wilson]

All,

Based on the comments received, I am inclined to clarify the proposed
language under Issues #154 and #187 with reference to a CA's Bugzilla
compliance bugs rather than "incidents". The existing language in section
2.4 of the MRSP already requires the CA to promptly file an Incident Report
in Bugzilla for all incidents.

My proposal for Issue #154 is to add a final sentence to MRSP section 2.4
which would say, "If being audited according to the WebTrust criteria, the
CA’s Management Assertion letter MUST include a complete list of the CA's
Bugzilla compliance bugs that were unresolved at any time during the audit
period."

Under Issue #187, I propose that new item 11 in MRSP section 3.1.4 (required
publicly-available audit documentation) would read: "11. a complete list
of the CA’s Bugzilla compliance bugs that were unresolved at any time
during the audit period."

Regarding guidance on excluding bugs that are flagged as "Invalid" or
"Duplicate" - I propose that we add a section to
https://wiki.mozilla.org/CA/BR_Audit_Guidance and hyperlink to it from the
words "CA's Bugzilla compliance bugs". The guidance would say that invalid
or duplicate bugs do not need to be included in the list.

Also, in response to Jeff's comment, if a bug is in an unresolved status
spanning two audit periods, I think it should still appear in both
management assertions and audit reports because one of the primary
rationales for these requirements is to ensure that auditors are aware of
the CA's compliance status.

Thoughts?

Thanks,

Ben

On Fri, Nov 6, 2020 at 10:36 AM Jeff Ward via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On Thursday, October 22, 2020 at 1:53:40 PM UTC-5, Ben Wilson wrote:

> Thanks for bringing this up Ben. It is important to consider this
> requirement in conjunction with #154 and address them together. It seems
> reasonable to require a CA to disclose all known incidents that are
> applicable during a given period. It would be important, however, to define
> “known incident” as a “verified bug” and exclude items such as bugs closed
> as a duplicate, invalid, etc. It would also make sense to clarify that an
> incident should only be disclosed once and eliminate duplication when an
> incident spans two audit periods.
>
> Also keep in mind an auditor typically issues an opinion on management’s
> assertion of its controls. Audit opinions do not make negative assurance
> statements, such as not being aware of any incidents during the period. If
> the CA is required to make this assertion, the auditor’s opinion will
> consider that statement.
>
> Thanks,
>
> Jeff

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

[Clemens Wanko]

Hi Ben,
that works fine for me from the ETSI auditors perspective.
REM: The ETSI Audit Attestation template requires the auditor to include a full list of Bugzilla compliance bugs – resolved or unresolved – which are relevant for the past audit period.

Best regards
Clemens

[Ryan Sleevi]

On Sun, Jan 24, 2021 at 11:33 PM Ben Wilson via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> All,
>
> Based on the comments received, I am inclined to clarify the proposed
> language under Issues #154 and #187 with reference to a CA's Bugzilla
> compliance bugs rather than "incidents". The existing language in section
> 2.4 of the MRSP already requires the CA to promptly file an Incident Report
> in Bugzilla for all incidents.
>
> My proposal for Issue #154 is to add a final sentence to MRSP section 2.4
> which would say, "If being audited according to the WebTrust criteria, the
> CA’s Management Assertion letter MUST include a complete list of the CA's

> Bugzilla compliance bugs that were unresolved at any time during the audit
> period."
>

> Under Issue #187, I propose that new item 11 in MRSP section 3.1.4
> (required
> publicly-available audit documentation) would read: "11. a complete list

> of the CA’s Bugzilla compliance bugs that were unresolved at any time
> during the audit period."
>

I don't think this is a good change, and doesn't meet the intent of the
problem.

This implies that if Mozilla believed an incident resolved (or, as we've
seen certain CAs do, the CA themselves mark their issue as resolved), that
there's no requirement to disclose this to the auditor other than "Hope the
CA is nice" (which, sadly, is not reasonable).

I explicitly think incident is the right approach, and disagree that
flagging it as compliance bugs is good or useful for the ecosystem. I
further think that even matters flagged as "Duplicate" or "Invalid" _are_
useful to ensure that the auditor is aware of the relevant discussion. For
example, if evidence contrary to the facts stated on the bug (i.e. it was
*not* a duplicate), this is absolutely relevant.

So I guess I'm disagreeing with Jeff and Clemens here, by suggesting that
incident should be any known or reported violation of Mozilla policy, which
may be manifested as bugs, in order to ensure transparency and confirmation
that the auditor had the necessary information and facts available and that
it was considered as part of the statement. This still permits auditors to,
for example, consider the issue as a duplicate/remediated, but given that
the whole goal is to receive confirmation from the auditors that they were
aware of all the same things the community is, I don't think the proposed
language gets to that.

[Ben Wilson]

Here is an edit to proposed subparagraph 11 of MRSP section 3.1.4:

The publicly-available documentation relating to each audit MUST contain at
least the following clearly-labelled information:

11. all incidents (as defined in section 2.4), including those reported in
Bugzilla, that were:
* disclosed by the CA or discovered by the auditor, and
* unresolved at any time during the audit period;

See
https://github.com/BenWilson-Mozilla/pkipolicy/commit/d832883749280a1ee434c299e8d6bb0597dc8095

The idea is that all "incidents" must be reported if they were "unresolved"
- which would include those that occurred or were open - at any time during
the audit period.

Additional guidance and interpretation of the above would be available on
the wiki.

On Thu, Jan 28, 2021 at 2:05 PM Ryan Sleevi <ry...@sleevi.com> wrote:

>
>
> On Sun, Jan 24, 2021 at 11:33 PM Ben Wilson via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
>
>> All,
>>
>> Based on the comments received, I am inclined to clarify the proposed
>> language under Issues #154 and #187 with reference to a CA's Bugzilla
>> compliance bugs rather than "incidents". The existing language in section
>> 2.4 of the MRSP already requires the CA to promptly file an Incident
>> Report
>> in Bugzilla for all incidents.
>>
>> My proposal for Issue #154 is to add a final sentence to MRSP section 2.4
>> which would say, "If being audited according to the WebTrust criteria, the
>> CA’s Management Assertion letter MUST include a complete list of the CA's

>> Bugzilla compliance bugs that were unresolved at any time during the audit
>> period."
>>

>> Under Issue #187, I propose that new item 11 in MRSP section 3.1.4
>> (required
>> publicly-available audit documentation) would read: "11. a complete list

>> of the CA’s Bugzilla compliance bugs that were unresolved at any time
>> during the audit period."
>>
>

[malcol...@gmail.com]

On Thursday, 11 February 2021 at 21:14:13 UTC, Ben Wilson wrote:
> 11. all incidents (as defined in section 2.4), including those reported in
> Bugzilla, that were:
> * disclosed by the CA or discovered by the auditor, and
> * unresolved at any time during the audit period;
>

> The idea is that all "incidents" must be reported if they were "unresolved"
> - which would include those that occurred or were open - at any time during
> the audit period.
>

Wouldn't it be clearer to non-native English speakers to avoid the nuance associated with "unresolved at any time" needing to imply both those that occurred or those that were still open?

Why not amend the language to just say:

11. all incidents (as defined in section 2.4), including those reported in Bugzilla, that:
* were disclosed by the CA or discovered by the auditor, and
* occurred or were open at any time during the audit period;

[Ben Wilson]

I'm fine with that suggestion.

On Fri, Feb 12, 2021 at 5:06 AM malcol...--- via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On Thursday, 11 February 2021 at 21:14:13 UTC, Ben Wilson wrote:

> > 11. all incidents (as defined in section 2.4), including those reported
> in
> > Bugzilla, that were:
> > * disclosed by the CA or discovered by the auditor, and
> > * unresolved at any time during the audit period;
> >

> > The idea is that all "incidents" must be reported if they were
> "unresolved"
> > - which would include those that occurred or were open - at any time
> during
> > the audit period.
> >
>

> Wouldn't it be clearer to non-native English speakers to avoid the nuance
> associated with "unresolved at any time" needing to imply both those that
> occurred or those that were still open?
>
> Why not amend the language to just say:
>

> 11. all incidents (as defined in section 2.4), including those reported in

> Bugzilla, that:
> * were disclosed by the CA or discovered by the auditor, and
> * occurred or were open at any time during the audit period;

[Jeff Ward]

This wording works from a WebTrust perspective as well. Thanks!

[Ben Wilson]

Kathleen and I edited the proposed language (
https://github.com/BenWilson-Mozilla/pkipolicy/commit/a69aa03fb92d1b0c3f74fd560dffefdeed934b45)
to now read:

"The publicly-available documentation relating to each audit MUST contain
at least the following clearly-labelled information:

...
11. all incidents (as defined in section 2.4) disclosed by the CA,
discovered by the auditor, or reported by a third party, that, at any time
during the audit period, occurred or were open in Bugzilla;"

Additional guidance will be provided here:
https://wiki.mozilla.org/CA/Audit_Statements and/or here:
https://wiki.mozilla.org/CA/Responding_To_An_Incident

<https://github.com/BenWilson-Mozilla/pkipolicy/commit/a69aa03fb92d1b0c3f74fd560dffefdeed934b45>

On Mon, Feb 15, 2021 at 11:47 AM Jeff Ward via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On Friday, February 12, 2021 at 10:27:11 AM UTC-6, Ben Wilson wrote:

> This wording works from a WebTrust perspective as well. Thanks!