Auditor letters and incident reports

[Jeremy Rowley]

Hey all,

An interesting issue came up recently with audits. Because the Mozilla policy includes some requirements that diverge from the BRs, the audit criteria don't necessarily cover everything Mozilla cares about. Thus, it's possible to have an incident that doesn't show up on an audit. It's also possible that the auditor determines the incident is not sufficiently important/risky(?) to include it in an audit. For example: https://bugzilla.mozilla.org/show_bug.cgi?id=1458024. Auditors aren't controlled by the CA and operate independently which means the CA can't dictate what goes into the opinion. One solution is to require CAs to list all of the incidents that occur during their audit in the management assertion letter. I posted an addendum to the management assertion on that thread. Going forward, we'll just include it as part of the main body. I need to look into whether I can get our existing audit reissued to the appendix is part of the seal as well.

What do you think about just requiring that as part of the Mozilla policy? Ie - the management assertion letter must include a list of the incidents active/opened during the audit period. Something like that could ensure transparency and make sure all incidents are disclosed to the auditor, distinguishing the CA's disclosures from the auditors.

Jeremy

[Jeremy Rowley]

Full disclosure - this was not my idea, but I thought it was a really good one and worth bringing up here.

_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

[Ryan Sleevi]

Hey Jeremy,

Huge thanks for raising this. Sadly, you’re about to be eaten by a grue [1]

Browsers have definitely had conversations with both WebTrust and ETSI on
this, and I recall some conversations to that effect during the CA/Browser
Forum F2F discussions.

One of the challenges is that ETSI and WebTrust are highly divergent in
this area, in that an ETSI audit is not based on Management’s Assertions in
the way that WebTrust is, but against a set of criteria (largely) CAs (via
ETSI) developed for themselves, in order to meet certain European legal
obligations.

To date, the policy has left it flexible for that reason. Instead, we’ve
tried to work very closely to explain the concerns to both WebTrust and to
ETSI. To address these concerns, the WebTrust TF went back and examined
their relevant professional obligations (e.g. under ISAE 3000 or AICPA’s
AT-C standards) and developed illustrative reports that WebTrust-licensed
auditors could use to opine on and disclose other relevant matters, such as
incidents. While it’s certainly true that the WebTrust TF does not
presently place an obligation on the auditor to disclose these matters,
either via the (“externally” maintained) aforementioned professional
standards or via the (TF-maintained) audit criteria, a competent and
capable auditor should be familiar both with the illustrative reports and
the policy requirements of the relevant report recipients. CAs contracting
with their auditors can certainly ensure that the auditor understands the
requirements placed upon the CA by its report recipients, and thus ensure
the auditor will disclose information they are professionally empowered to
disclose, and as demonstrated via the illustrative report.

There are several subtle differences in the implications of the reporting,
as to whether it’s disclosed via Management’s Assertion or disclosed as
other findings, and much more stark differences with respect to ETSI
reporting, so I’m not sure a one-size-fits-all approach works here. That
said, I absolutely entirely welcome any and all additional disclosures that
are made by CAs in their management letters, and that’s something commonly
discussed with WebTrust in trying to find angles for formalizing various
reporting and disclosure.

This community is wholly open, so perhaps the best thing CAs can do is
ensure their auditors actively follow, and perhaps even engage with, this
community. We’ve definitely had folks from the audit community involved and
seeking insight and input (without violating their professional obligations
re: confidentiality or their professional opinion forming requirements), so
perhaps it’s worth making sure your auditor does or commits to do so. I
know Scott Perry was one of the original firms involved here, before the
CPA licensing, as they were recognized by Mozilla as a competent auditor
independent from Mozilla when they first proposed becoming an auditor many
years ago (15+?) Perhaps they should revisit and be more closely following?

I realize that answer is a bit “No, but yes, but no” - but it’s exactly the
sort of discussion worth having, and it’s been a conversations browsers
have been having for years now with the auditors, in trying to address
auditor quality control issues. Hopefully, some of the auditors, and
particularly the WTTF members who I do know lurk here, may want to chime
in, especially if I’ve botched any of the summarization :)

[1]
http://mentalfloss.com/article/29885/eaten-grue-brief-history-zork

[clemen...@tuv-austria.com]

Dear all,
just a short note on that with regard to auditing and Audit Attestations based upon ETSI: throughout the audit we check the incidents of the current audit period as documented by the CA (have they been addressed at a sufficient level, have the measures taken proven that they are sufficient in depth and coverage, etc.). We are going to write a finding in case incidents seem to be not or not fully covered. We do have a corresponding section 7.9 in ETSI EN 319 401 dealing with that.
Furthermore we agreed with the browsers to have a specific section in the Audit Attestation: any buzilla bugs reported in the audit period are going to be listed in the Attestation including their status (closed, open/reason).

Hence, from my perspective we would not really need an additional requirement in the Mozilla policy for that. And please let's keep in mind that the best way to treat such things is to add them to a CA/B Forum requirements document rather than to a browsers root store policy as that affects only CAs being part of that Root Store rather than the whole community.

Clemens

[jeffw...@gmail.com]

Reposting as I don't see my original response to this thread:

Sorry for the delay in responding, but I first wanted to gather feedback from the members of the WebTrust Task Force. Keep in mind any audit/assurance engagement entails professional judgment, which of course may vary from firm to firm. We are certainly seeing a trend in audit reports whereby “Other Matters” are described that do not modify/qualify the auditor’s opinion but do allow the auditor to make mention of the issues. Some firms use this section to list items noted in Bugzilla, for instance, to demonstrate that these types of issues were addressed in the course of the audit.

Depending on the firm, some firms have been listing all issues noted in Bugzilla, while others are listing only those that are significant. As a Task Force, it is not our position, or even a possibility, for us to mandate how these should be handled as professional judgment will dictate their treatment based on the respective firms’ risk management policies. That being said, we have as part of our draft practitioner guidance commentary that the browser community prefers seeing all these types of issues listed as either modifications/qualifications, or described in “Other Matters” and encourage practitioners to use this approach.

In most cases, management’s Assertion accompanies the audit report, and there is greater flexibility for what goes into them. Our professional standards require certain items be present in the Assertion, but other items that management wants to add are permissible to include. Except in the rare reporting scenario when an assertion by management is not provided, most of the Task Force members agree there would not be any issue with your proposed requirement to require a CA's management to detail Bugzilla or similar issues relevant to the current audit period in their assertion to the auditors.

As far as the management representation letter, which is a required written communication to the auditors at the conclusion to the audit, regulatory non-compliance would normally form part of the management representation that is needed for each audit, and the assertion can form part of that representation. Keep in mind that the management representation letter is not part of the deliverable that is viewed publicly as is the case with the auditor’s opinion and management assertion.

To demonstrate that the auditor has addressed the significance of the relevant issues, we can propose that the standard WebTrust reports have an Exhibit that sets out the issues identified by reference to Bugzilla, as an example. The audit report can then reflect the relevant significance either as a report qualification or through an "Other Matters" paragraph that provides further commentary. The main issue is that, even by identifying and addressing them in the audit report, there is still the potential for disagreements to exist as to significance due to auditor’s professional judgment. Of course, in our new SOC-like report, these issues will be discussed in the system description, audit report and potentially an unaudited management comment section.

Thanks,

Jeff Ward

[Wayne Thayer]

Thanks for the response Jeff.

On Fri, Sep 6, 2019 at 4:17 PM jeffwardpki--- via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On Wednesday, August 21, 2019 at 11:46:37 PM UTC-5, Jeremy Rowley wrote:

> Reposting as I don't see my original response to this thread:
>
> Sorry for the delay in responding, but I first wanted to gather feedback
> from the members of the WebTrust Task Force. Keep in mind any
> audit/assurance engagement entails professional judgment, which of course
> may vary from firm to firm. We are certainly seeing a trend in audit
> reports whereby “Other Matters” are described that do not modify/qualify
> the auditor’s opinion but do allow the auditor to make mention of the
> issues. Some firms use this section to list items noted in Bugzilla, for
> instance, to demonstrate that these types of issues were addressed in the
> course of the audit.
>
> Depending on the firm, some firms have been listing all issues noted in
> Bugzilla, while others are listing only those that are significant. As a
> Task Force, it is not our position, or even a possibility, for us to
> mandate how these should be handled as professional judgment will dictate
> their treatment based on the respective firms’ risk management policies.
> That being said, we have as part of our draft practitioner guidance
> commentary that the browser community prefers seeing all these types of
> issues listed as either modifications/qualifications, or described in
> “Other Matters” and encourage practitioners to use this approach.
>
>

I would go beyond "all issues noted in Bugzilla" to "all issues, including
those noted in Bugzilla".

In most cases, management’s Assertion accompanies the audit report, and
> there is greater flexibility for what goes into them. Our professional
> standards require certain items be present in the Assertion, but other
> items that management wants to add are permissible to include. Except in
> the rare reporting scenario when an assertion by management is not
> provided, most of the Task Force members agree there would not be any issue
> with your proposed requirement to require a CA's management to detail
> Bugzilla or similar issues relevant to the current audit period in their
> assertion to the auditors.
>
>

By doing this, a CA can eliminate any question about disclosure of
incidents to auditors, which is terrific and encouraged. As a Mozilla
requirement it has the problem - as Ryan pointed out - that ETSI audits
don't include management assertions, at least not today.

> As far as the management representation letter, which is a required
> written communication to the auditors at the conclusion to the audit,
> regulatory non-compliance would normally form part of the management
> representation that is needed for each audit, and the assertion can form
> part of that representation. Keep in mind that the management
> representation letter is not part of the deliverable that is viewed
> publicly as is the case with the auditor’s opinion and management assertion.
>
> To demonstrate that the auditor has addressed the significance of the
> relevant issues, we can propose that the standard WebTrust reports have an
> Exhibit that sets out the issues identified by reference to Bugzilla, as an
> example. The audit report can then reflect the relevant significance
> either as a report qualification or through an "Other Matters" paragraph
> that provides further commentary. The main issue is that, even by
> identifying and addressing them in the audit report, there is still the
> potential for disagreements to exist as to significance due to auditor’s
> professional judgment. Of course, in our new SOC-like report, these issues
> will be discussed in the system description, audit report and potentially
> an unaudited management comment section.
>
>

This also sounds like a great idea, and one that is more in line with the
solution Clemens described for ETSI - disclosing major and minor
conformities in all attestation reports. I'm much less interested in the
categorization of an issue as a qualification versus an "other matter" than
I am with having it disclosed.

Under this approach, Mozilla could require that all audit reports include a
section listing all incidents that occurred during the audit period. Given
that Mozilla already places specific requirements on audit reports in
section 3.1.4, our policy is a good location for this requirement. I filed
https://github.com/mozilla/pkipolicy/issues/187 to track this idea.

Thanks,
>
> Jeff Ward
>
>