Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

[Ben Wilson]

Historically, Mozilla Policy required that CAs "provide attestation of
their conformance to the stated verification requirements and other
operational criteria by a competent independent party or parties with
access to details of the CA's internal operations."
https://wiki.mozilla.org/CA:CertificatePolicyV1.0 "Competency" was "for
whom there is sufficient public information available to determine that the
party is competent to judge the CA's conformance to the stated criteria. In
the latter case the 'public information' referred to should include
information regarding the party's:

- knowledge of CA-related technical issues such as public key
cryptography and related standards;
- experience in performing security-related audits, evaluations, or risk
analyses; *and*
- honesty and objectivity."

Today, section 3.2 of the MRSP
<https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#32-auditors>
states, "In normal circumstances, Mozilla requires that audits MUST be
performed by a Qualified Auditor, as defined in the Baseline Requirements
section 8.2," but under section 2.3
<https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#23-baseline-requirements-conformance>,
"Mozilla reserves the right to accept audits by auditors who do not meet
the qualifications given in section 8.2 of the Baseline Requirements, or
refuse audits from auditors who do."

Section 8.2 of the Baseline Requirements states an auditor must have:
1. Independence from the subject of the audit;
2. The ability to conduct an audit that addresses the criteria specified in
an Eligible Audit Scheme (see Section 8.1);
3. Employs individuals who have proficiency in examining Public Key
Infrastructure technology, information security tools and techniques,
information technology and security auditing, and the third-party
attestation function;
4. (For audits conducted in accordance with any one of the ETSI standards)
accredited in accordance with ISO 17065 applying the requirements specified
in ETSI EN 319 403;
5. (For audits conducted in accordance with the WebTrust standard) licensed
by WebTrust;
6. Bound by law, government regulation, or professional code of ethics; and
7. Except in the case of an Internal Government Auditing Agency, maintains
Professional Liability/Errors & Omissions insurance with policy limits of
at least one million US dollars in coverage

It is proposed in Issue #192
<https://github.com/mozilla/pkipolicy/issues/192> that information about
individual auditor's qualifications be provided--identity, competence,
experience and independence. (For those interested as to this independence
requirement, Mozilla Policy v.1.0 required either disclosure of the
auditor's compensation or the establishment that the auditor "is bound by
law, government regulation, and/or a professional code of ethics to render
an honest and objective judgement regarding the CA.")

While subsection 3 of BR 8.2 requires "individuals who have proficiency in
examining Public Key Infrastructure technology, information security tools
and techniques, information technology and security auditing, and the
third-party attestation function," that fact needs evidence in order to be
established. The proposed resolution of this Issue #192 intends to
accomplish that.

This proposal to require disclosure of individual auditor qualifications is
very similar to the approach adopted by the U.S. Federal PKI
<https://www.idmanagement.gov/wp-content/uploads/sites/1171/uploads/fpki-annual-review-requirements.pdf>
(see Appendices B-1 and C). E.g., "Did each Audit Opinion Letter identify
the auditor and the individuals performing the audit?" In practice, the
information about auditor qualifications could be in the form of a separate
document, such as a curriculum vitae.

Some initial, draft language to address this issue is located here:
https://github.com/BenWilson-Mozilla/pkipolicy/commit/d0da7cb2b6db38e66c3a72e5c1db0e78e91d8df6

A new subsection 3. would be added to the list of audit requirements that
would require "[the] name(s) and qualifications of individuals performing
the audit, as required by section 3.2" and a new paragrpah would be added
to section 3.2 that would say, "A Qualified Auditor MUST have relevant IT
Security experience, or have audited a number of CAs, and be independent
and not conflicted. Individuals have competence, partnerships and
corporations do not. Audit documentation of individual auditor
qualifications MUST be provided to Mozilla that is sufficient for Mozilla
to determine the competence, experience, and independence of the Qualified
Auditor. Mozilla will review each individual auditor’s credentials and
ensure that any Qualified Auditor has the collective set of skills required
by section 8.2 of the Baseline Requirements."

Please provide your comments and suggestions in response to this email.

Thanks,

Ben

[Clemens Wanko]

Hi Ben,
in order to avoid for every single audit the compilation work for the auditor (in person) on his qualification, independence, etc. as well as the need to crosscheck the statements he made, that was covered for the EU ETSI/eIDAS scheme by the accreditation of the body (organization; example: I am member/employee of TUV Austria CERT as accredited body) employing and using that auditor (in person) for a specific audit. The body will receive/keep its accreditation only in case it proves in annual audits with its accreditation body, that the following auditor related aspects were covered in the audit projects performed throughout the past audit period:

ISO/IEC17065 - I listed the relevant chapters only:
6 Resource requirements ............................... 31
6.1 Certification body personnel ....................... 31
6.1.1 General .......................................................... 31
6.1.2 Management of competence for personnel involved in the certification process ......................................................... 31
6.1.3 Contract with the personnel ........................ 33
6.2 Resources for evaluation............................. 33
6.2.1 Internal resources ........................................ 33
6.2.2 External resources (outsourcing) ............... 33

…amended by guidance documentation in the following areas:
Annex A (informative) Principles for product certification bodies and their certification activities ................................... 63
A.1 General .......................................................... 63
A.2 Impartiality .................................................... 63
A.3 Competence .................................................. 65
A.4 Confidentiality and openness ..................... 65
A.4.1 General .......................................................... 65
A.4.2 Confidentiality .............................................. 65
A.4.3 Openness ...................................................... 65
A.4.4 Access to information .................................. 65
A.5 Responsiveness to complaints and appeals .......................................................... 65
A.6 Responsibility ............................................... 67

For ETSI/eIDAS auditors the ISO/IEC17065 is detailed by the following mandatory ETSI EN 319 403 (updated version: ETSI EN 319 403-1) requirements. I listed the relevant chapters only:
“Electronic Signatures and Infrastructures (ESI); Trust Service Provider Conformity Assessment; Part 1: Requirements for conformity assessment bodies assessing Trust Service Providers”:
6 Resource requirements ........................................................................................................................... 11
6.1 Conformity Assessment Body personnel ......................................................................................................... 11
6.1.1 General ........................................................................................................................................................ 11
6.1.2 Management of competence for personnel involved in the audit process................................................... 11
6.1.2.0 General requirements ............................................................................................................................ 11
6.1.2.1 Management of competence.................................................................................................................. 11
6.1.2.2 Training of audit teams ......................................................................................................................... 12
6.2 Resources for evaluation .................................................................................................................................. 12
6.2.0 General requirements .................................................................................................................................. 12
6.2.1 Internal resources ........................................................................................................................................ 12
6.2.1.0 General requirement .............................................................................................................................. 12
6.2.1.1 Competence of Conformity Assessment Body personnel ..................................................................... 12
6.2.1.2 Competences for all functions ............................................................................................................... 12
6.2.1.3 Competences for application review ..................................................................................................... 13
6.2.1.4 Competences and prerequisites for auditing.......................................................................................... 13
6.2.1.5 Competences for review ........................................................................................................................ 13
6.2.1.6 Competences for certification decision ................................................................................................. 14
6.2.1.7 Competences for Technical Experts ...................................................................................................... 14
6.2.1.8 Audit team ............................................................................................................................................. 14
6.2.1.9 Audit team leader .................................................................................................................................. 15
…
Annex A (informative): Auditors' code of conduct .............................................................................. 25

For audits performed by ISO17065/ETSI EN 319 403 accredited bodies it is therefore ensured, that the requested auditors properties you refer to are guaranteed. We deem it redundant to require the auditor acting for the accredited body to provide corresponding statements again. Furthermore we believe that repetition would not provide additional/better assurance.
Apart from that we expect GDPR trouble in case the accredited body publishes personal auditor information.

All the best
Clemens (TUV Austria & ACAB’c member)

[Ryan Sleevi]

Hi Clemens,

I think this fundamentally misunderstands the proposal. As Ben mentioned,
and as countless other schemes have highlighted, competency is with
individuals, not organizations. While the eIDAS Scheme is relevant for
eIDAS qualification, I think it's important to highlight that browsers are
not, nor have they ever, relied upon the Qualified Trust List, nor on the
eIDAS Framework, as they are fundamentally separate trust frameworks. I
understand you see this redundant, but given that browsers are not relying
on the Supervisory Body function, since they are different trust
frameworks, it's absolutely vital for transparency to be able to provide
that information.

While something may be acceptable for the eIDAS Certification scheme,
audits exist to support those consuming them, and it's important to make
sure they are addressed. WebTrust equally has an approach like you
describe, but what is being suggested here is that is not sufficient for
the need and use case, and I fully support that. I can understand that
professional accountability is scary, because it might mean that audits
that might be acceptable for eIDAS are rejected as unacceptable for
Mozilla, but again, that's a reflection of the different nature of trust
frameworks.

I find the appeal to redundancy and the NAB, and further, the suggestion of
GDPR, to be a bit insulting to this community. This opposition to
transparency fundamentally undermines the trust in ETSI provided audits, or
this appeal to the eIDAS scheme, which has limited relevance given it's a
fundamentally different audit scheme, beggars belief. If you'd like to
raise Fear/Uncertainty/Doubt about GDPR, I believe you owe this community a
precise and detailed explanation about what you believe, relevant to the
auditor professional experience, would be problematic.

The suggestion here that this is somehow unique or untowards is deeply
concerning. I note, for example, that BSI's own C5 controls are designed
around transparency, and Section 4.4.9 on auditor qualification similarly
places provisions for transparency.

Without making an appeal to either the NAB or the Supervisory Body, both of
which are relevant for eIDAS but not acting in a function for browser trust
frameworks (nor can they), what alternative would you propose to help
community members here have verifiable evidence and confidence in the
auditor's qualifications?

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

[Tim Hollebeek]

Ben,

We're concerned that these changes could unintentionally prevent many new auditors from being able to conduct audits, despite being Qualified Auditors. The problem is that CAs will understandably be very hesitant to use a new auditor, as they cannot risk having an audit conducted, and then not accepted by Mozilla. An unfortunate effect of that is that CAs will likely only consider auditors who have a previous history of being accepted as qualified.

One potential solution is to allow CAs to ask Mozilla to "pre-vet" a potential auditor they would like to use. Mozilla policy already has a process in the next paragraph to provide opinions on auditors who "do not fit the definition of Qualified Auditor" that could potentially be re-used. Unfortunately, as the paragraph is currently worded, it can only be used for auditors who are *NOT* Qualified, which is really weird.

It would be helpful to clarify that CAs MAY use the same procedure to work with Mozilla to determine whether a particular auditor is Qualified or not, prior to the beginning an engagement.

-Tim

[Ryan Sleevi]

That sounds like a great idea, and sounds like a good compliment to an
approach that several (Root) CAs have taken, of requiring all their
externally-operated subordinate CAs review their auditors and audit scope
with the root CAO, before finalizing the audit engagement.

An example of how the public review has been done in the past is available
at
https://groups.google.com/g/mozilla.dev.security.policy/c/sRJOPiePUvI/m/oRmRffaQmcYJ

[Wojtek Porczyk]

On Thu, Nov 05, 2020 at 11:48:20AM -0500, Ryan Sleevi via dev-security-policy wrote:
> competency is with individuals, not organizations.

[snip]

> I find the appeal to redundancy and the NAB, and further, the suggestion of
> GDPR, to be a bit insulting to this community. This opposition to
> transparency fundamentally undermines the trust in ETSI provided audits, or
> this appeal to the eIDAS scheme, which has limited relevance given it's a
> fundamentally different audit scheme, beggars belief. If you'd like to
> raise Fear/Uncertainty/Doubt about GDPR, I believe you owe this community a
> precise and detailed explanation about what you believe, relevant to the
> auditor professional experience, would be problematic.

Not the original poster, but
1) I understand that the very general language of OP, which you dismiss as
FUD, is because this is "consult your own lawyer" area;
2) contrary to what you have written, the onus is on Mozilla to demonstrate
the compliance with GDPR and not the other way around.

If Mozilla (or you personally, in your capacity as peer, doesn't matter)
intend to keep track of competency of people (like "physical people" and not
corporations), those people (at least those, who perform audits in Europe)
have certain rights from Mozilla under GDPR. You can't have it both ways
-- either you keep trust in organisations and ignore GDPR, or you keep trust
in people, and then you have all those GDPR requirements. Those are not hard
to fulfill, but they would have to be thought through before the policy takes
effect. I have found nothing in either the proposed change, or your response,
that this problem has been thought through.

For example, art. 13 of GDPR specifies that the data subject (the auditor) is
to be provided with information that the data about her/him is processed. In
the spirit of transparency, could you post an example notice which would be
sent to the auditor in question?

What would be the legal basis? (art. 6) If (e) or (f), the auditor has a right
to object; what would happen after the objection?

Have Mozilla appointed a representative in the EU (art. 27)? (I just checked
and I have found only "Attn: Legal" address in USA). If not, why? If yes,
what's his/her name and contact details?


--
pozdrawiam / best regards
Wojtek Porczyk

I do not fear computers,
I fear lack of them.
-- Isaac Asimov

[Ryan Sleevi]

On Thu, Nov 5, 2020 at 7:00 PM Wojtek Porczyk <wo...@invisiblethingslab.com>
wrote:

> On Thu, Nov 05, 2020 at 11:48:20AM -0500, Ryan Sleevi via
> dev-security-policy wrote:
> > competency is with individuals, not organizations.
>
> [snip]
>
> > I find the appeal to redundancy and the NAB, and further, the suggestion
> of
> > GDPR, to be a bit insulting to this community. This opposition to
> > transparency fundamentally undermines the trust in ETSI provided audits,
> or
> > this appeal to the eIDAS scheme, which has limited relevance given it's a
> > fundamentally different audit scheme, beggars belief. If you'd like to
> > raise Fear/Uncertainty/Doubt about GDPR, I believe you owe this
> community a
> > precise and detailed explanation about what you believe, relevant to the
> > auditor professional experience, would be problematic.
>
> Not the original poster, but
> 1) I understand that the very general language of OP, which you dismiss as
> FUD, is because this is "consult your own lawyer" area;
> 2) contrary to what you have written, the onus is on Mozilla to demonstrate
> the compliance with GDPR and not the other way around.
>

I think this is significantly misstating things.

Without opining on the legal statements you've offered, the reality is that
fundamentally, if we cannot achieve reasonable evidence to trust the
auditor - specifically, the individuals that make up the audit team (both
the audit members and any relevant technical experts that may have
contributed) - then there's no objective reason to accept the audit. The
concerns you raised are secondary to that, and so the objection to
something *cannot* be provided simply means that it would limit the
utility, reliability, and trustworthiness of those audits and potentially
make them unacceptable. The need for objective, transparent understanding
about the skills and competencies is as paramount as the need for an
objective, transparent understanding about the CA itself, and for which
audits exist. The attestation of the CAB is demonstrably insufficient for
purpose, in the same way as a CA saying "I solemnly swear I'm up to good"
would not somehow invite trust.

I understand that the appeal is "Well, the NAB oversees the CAB, you see,
and EA oversees the NABs, and through the power of Regulation 765/2008,
trust is imbued", but that of course fails the very basic test of having
concrete, objective data for the consumer of the audit (e.g. Mozilla, but
also this broader community). It entirely outsources the supervision,
without providing any evidence of meeting the requirement, for a core
product security requirement. Rather than accept such risk, it becomes just
as reasonable to reject such audits.

I highlight this because to suggest something *cannot* be done is to
unquestionably invite the assertion of FUD. If there are *challenges* to
doing so, we should try to find solutions. But outright dismissal, or
suggesting somehow that transparency is not necessary because *handwave*
this other scheme *handwave* doesn't inspire confidence, nor does it
achieve the necessary transparency. This is clearly not insurmountable, as
discussed previously, and in the worst case, might mean that rather than
arbitrarily accepting audits, Mozilla would contractually negotiate with
potential auditors to ensure the necessary skills and qualifications were
met (e.g. as widely practiced in other industries).

[Clemens Wanko]

Hi Ryan, hi all,
three things to comment on that:

1. How is the EU ETSI audit scheme thought and what is it intended to provide to Mozilla and the CA/Browser ecosystem?
The European scheme of technical standards for CA/TSP developed by ETSI was made and is constantly adopted to integrate CA/Browser requirements. The main standards I am referring to are: ETSI EN 319 401, EN 319 411-1 and EN 319 411-2. With regard to auditor guidance specifically for the CA/Browser ecosystem, ETSI even developed and maintains the TS 119 403-2 “Part 2: Additional requirements for Conformity Assessment Bodies auditing Trust Service Providers that issue Publicly-Trusted Certificates”.
For auditors using such technical standards Europe has established a well thought through scheme based upon organizations (accredited audit bodies) which job it is – amongst others – to ensure auditors (the natural person) qualification, independence, etc. This scheme turned out to be of high trustworthiness, reliability, robustness, etc. And it works very well over here since years.

2. Transparency
The transparency lies in the European scheme with independent skilled bodies auditing each other in order to ensure conformant implementation of technical standards as well as of operational requirements for audit bodies as well as for the single auditor (natural person). Not only the requirements for the qualification/independence/etc. of the auditor (natural person) are clearly defined within the ISO/ETSI but also the management requirements of the body in order to ensure that they are kept upright – btw. BSI C5 controls, section 4.4.9 is meant similar to that.
Every accredited body is audited at least once a year by its NAB which checks conformant audit processing (e.g. according to ISO/IEC17065 in combination with ETSI EN 319 403).
Now, requesting more of transparency with regard to auditor qualification for Mozilla insinuates this immediately translates into more confidence in the auditor. Please lets be careful here. If we like the community to follow that, shouldn’t be clearly stated:
- which documents/evidences are expected from the auditor to be handed in to Mozilla,
- what criteria (details need to be published) shall Mozilla use to check the documents/evidences against,
- a statement of how Mozilla comes to the conclusion: auditor is acceptable / not acceptable,
- how Mozilla shall organize themselves to perform this task (skilled staff members are required),
- how that can get organized in a way that it is compatible to CA projects (see Tim Hollebeeks mail),
and finally:
what information is expected from Mozilla to be published for every single auditor (natural person) to show auditors qualification and make it transparent.
The tasks related to auditor qualification validation and management actually are performed by the participants in the European scheme already (and apart from that, not very different also in the WebTrust world).

3. Requiring all their externally-operated subordinate CAs review their auditors
You said:
quote <<< That sounds like a great idea, and sounds like a good compliment to an approach that several (Root) CAs have taken, of requiring all their externally-operated subordinate CAs review their auditors and audit scope with the root CAO, before finalizing the audit engagement.

An example of how the public review has been done in the past is available at https://groups.google.com/g/mozilla.dev.security.policy/c/sRJOPiePUvI/m/oRmRffaQmcYJ >>>

If we like to suggest that, wouldn’t we then not also need to state at least
1st based upon what rule-set we like the CA to review auditors,
2nd what competences are required at CA level to validate auditors and how those shall be established and maintained,
3rd how the CA is required to do that with regard to process and timing?
In order to be clear and transparent, things like these would need to be defined at CA level then.

Best regards
Clemens

[Jeff Ward]

Thanks Ben. Perhaps it would be helpful to provide context to how CPA Canada evaluates auditor qualifications. CPA Canada assesses a WebTrust practitioner’s qualifications both on an annual basis, and upon each WebTrust engagement being performed. The annual renewal process includes listing individual auditors’ qualifications and any other relevant information about the firm that changed since the previous renewal. CPA Canada also requires similar information for each engagement performed and reviews it prior to issuing any WebTrust seal. CPA Canada posts on its website a listing of firms that are deemed qualified to perform WebTrust engagements based on this process, but the details are not made public.

Audit reports, whether for WebTrust, financial statements, or other forms of engagement reports providing assurance to users of the information, do not include specific audit team members’ names. Simply stated, this desire to include individual auditor’s qualifications in a public report is not consistent with any other compliance reporting methods or reporting requirements for CAs, or any other auditee for that matter.

Perhaps there is a way to have a communication protocol established with CPA Canada to identify those firms known to be deficient in technical competency that the community could use in evaluating the worthiness of an audit opinion when circumstances warrant. If that is something that has merit, a discussion with CPA Canada directly would be the next step to explore this option to provide the user community the information they desire.

Thanks,

Jeff

[Ryan Sleevi]

On Fri, Nov 6, 2020 at 12:31 PM Jeff Ward via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Audit reports, whether for WebTrust, financial statements, or other forms
> of engagement reports providing assurance to users of the information, do
> not include specific audit team members’ names. Simply stated, this desire
> to include individual auditor’s qualifications in a public report is not
> consistent with any other compliance reporting methods or reporting
> requirements for CAs, or any other auditee for that matter.

Hi Jeff,

Could you help me square this statement with the practical examples? For
example, here's a report [1] from a WebTrust TF member, Ernst and Young,
demonstrating how this works in practice. You can see there hasn't been an
issue for years [2][3], even with the direct involvement of individuals on
the WebTrust TF in preparing such an audit.

So I'm having difficulty squaring your statement that they "do not", when
we've got examples from long-standing members of the WebTrust TF that
demonstrate that, in practice, they do. Could you help highlight what's
inconsistent here?

Alternatively, and as mentioned to ETSI, here's an example of an ISAE 3000
based audit scheme, similar to WebTrust, that also discloses the relevant
professional qualifications and skills to clients [4], as discussed in
4.4.8 and 4.4.9.

[1] https://www.oversight.gov/sites/default/files/oig-reports/18-19.pdf
[2]
https://www.oversight.gov/sites/default/files/oig-reports/Assessment%20Report%2019-12%20GPO%20Federal%20PKI%20Compliance.pdf
[3] https://www.oversight.gov/sites/default/files/oig-reports/17-27.pdf
[4]
https://www.bsi.bund.de/EN/Topics/CloudComputing/Compliance_Criteria_Catalogue/Compliance_Criteria_Catalogue_node.html

[Ryan Sleevi]

On Fri, Nov 6, 2020 at 12:00 PM Clemens Wanko via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Hi Ryan, hi all,
> three things to comment on that:
>
> 1. How is the EU ETSI audit scheme thought and what is it intended to
> provide to Mozilla and the CA/Browser ecosystem?

The European scheme of technical standards for CA/TSP developed by ETSI was
> made and is constantly adopted to integrate CA/Browser requirements. The
> main standards I am referring to are: ETSI EN 319 401, EN 319 411-1 and EN
> 319 411-2. With regard to auditor guidance specifically for the CA/Browser
> ecosystem, ETSI even developed and maintains the TS 119 403-2 “Part 2:
> Additional requirements for Conformity Assessment Bodies auditing Trust
> Service Providers that issue Publicly-Trusted Certificates”.
> For auditors using such technical standards Europe has established a well
> thought through scheme based upon organizations (accredited audit bodies)
> which job it is – amongst others – to ensure auditors (the natural person)
> qualification, independence, etc. This scheme turned out to be of high
> trustworthiness, reliability, robustness, etc. And it works very well over
> here since years.
>

I'm sure something is lost in translation, but this does not address any of
the concerns raised. The ETSI standards here are not relevant; voluntary
standards, in and of themselves, are not binding. The fact that a standard
exists is not relevant, since what matters is how the standard is adhered
to and supervised.

Your subsequent statement is simply "Well, they're auditors, people trust
them to do this", without any form of meaningful engagement on the why.
"It's their job to ensure independence" - yes, but that says nothing about
the performance of the job, that your measure of independence is consistent
with another measure, etc. Your entire appeal here simply is "We say we're
trustworthy, so of course we're trustworthy", which would be farcical if it
wasn't presented so seriously.

> 2. Transparency
> The transparency lies in the European scheme with independent skilled
> bodies auditing each other in order to ensure conformant implementation of
> technical standards as well as of operational requirements for audit bodies
> as well as for the single auditor (natural person).

This is, again, a restatement of "Trust us, because we declared we're
trustworthy". For something such as trust, there's no question of "but
verify". Your appeal to SDOs is not relevant here, because as you and I are
both aware, the SDO just makes (voluntary) standards, and doesn't enforce
them. And this gets to the heart of the matter: the question about how each
of these dimensions are interpreted is something you would prefer the CABs
to self-declare, and the suggestion here is "Sure, you can say that, but
you need to also help us understand why that's true".

> Not only the requirements for the qualification/independence/etc. of the
> auditor (natural person) are clearly defined within the ISO/ETSI but also
> the management requirements of the body in order to ensure that they are
> kept upright – btw. BSI C5 controls, section 4.4.9 is meant similar to
> that.
> Every accredited body is audited at least once a year by its NAB which
> checks conformant audit processing (e.g. according to ISO/IEC17065 in
> combination with ETSI EN 319 403).
>

This is the first time in this e-mail that you remotely come to
establishing a "why". As we both know full well, the authority of the CAB
(and its assessment along these relevant axis in 319 403) is imbued by the
NAB. The authority of the NAB is imbued by EA, established by Regulation
765/2008. The root of trust is, simply stated, "The state mechanisms of the
European Union trusts us, ergo, we are defacto trustworthy".

As a long-time participant in this group, I'm sure you can understand and
appreciate that the "Trust us, we're the government" argument rarely
improves trust. For CAs, we work on the basis of concrete evidence; after
all, this is why we have audits in the first place, even for government
CAs. The argument you're making here is that EA imbued sovereign member
states the ability to establish their NAB, and the NABs are responsible for
supervising the CABs. If we have a complaint with the CAB, the argument
goes, we should complain to the NAB, consistent with the ISO 17065
framework that EN 319 403 is based on.

Yet this misguided argument overlooks a host of salient details, no doubt
because of the convenience in doing so. Unlike, say, the ISAE 3000 approach
to audits, the approach taking by this EA/NAB/CAB chain is much more
limited with respect to a host of professional duties, and only directly
applicable within the contents of 319 403, and the relevant (supervised)
reports, such as 319 411-1. Thus, there's easy opportunity for there to be
a divergence in needs, by a CAB asserting that they are qualified within
the aegis of 319 403/319 411-1 with the necessary technical skills, and
which an individual supervisory body unfortunately accepts, while still
being vastly below the quality expected by the broader Mozilla community.
There is no opportunity for relief in the framework of 17065, because the
fact that the NAB/SB have determined the CAB's skills and reporting as
suitable for eIDAS is all that is required.

All of this, again, rests in a presumption that we "trust, but verify". To
accept the argument that EA/NAB are inviolate and incapable of making
decisions misaligned with community needs is to equally be making the
argument that every government CA should be automatically added, because
they're the government. That's not how this works. That's not how any of
this works.

> Now, requesting more of transparency with regard to auditor qualification
> for Mozilla insinuates this immediately translates into more confidence in
> the auditor. Please lets be careful here. If we like the community to
> follow that, shouldn’t be clearly stated:
>

I can understand and appreciate why, as a CAB used to performing audits
under the ETSI framework, you prefer that everything be a checklist.
Unfortunately, trust is not a checklist, but a preponderance of evidence of
past behavior that can lead to accurate prediction of future behavior. I
can trust you to do X today, because you did X yesterday, the day before,
and the day before that.

The basis for trust is that the CAB should build the evidence as to *why*
they are trustworthy. What makes the CAB relevant for this problem?

To date, your appeal has mostly been "Well, we address this other problem,
and they're satisfied, so clearly, we're qualified to address this
problem", but that's comparing apples to orangutans. Concrete evidence as
to /why/ and /how/ are relevant to being able to say "Yes, that's good
enough", but also to help identify trends and patterns, such as "Oh,
auditors with this skill X, or without this skill Y, consistently produce
this result Z, and that is/is not what we want"

I agree that there is benefit from an appropriate set of principles, but I
reject the assertion at face value that the idea of trust should be reduced
to a checklist, whether it's CAs or auditors. As a long-time participant,
you know exactly that the process of trust is not binary, and considers a
host of factors, some discretionary. Reducing these complex decisions to
binary decisions is exactly the direction root stores have spent
considerable effort moving away from over the past two decades, because of
the myriad security problems, and it'd be naive to suggest it somehow
appropriate for auditors.

> 3. Requiring all their externally-operated subordinate CAs review
> their auditors
> You said:
> quote <<< That sounds like a great idea, and sounds like a good
> compliment to an approach that several (Root) CAs have taken, of requiring
> all their externally-operated subordinate CAs review their auditors and
> audit scope with the root CAO, before finalizing the audit engagement.
> An example of how the public review has been done in the past is available
> at
> https://groups.google.com/g/mozilla.dev.security.policy/c/sRJOPiePUvI/m/oRmRffaQmcYJ
> >>>
>
> If we like to suggest that, wouldn’t we then not also need to state at
> least
> 1st based upon what rule-set we like the CA to review auditors,
> 2nd what competences are required at CA level to validate auditors and how
> those shall be established and maintained,
> 3rd how the CA is required to do that with regard to process and timing?
> In order to be clear and transparent, things like these would need to be
> defined at CA level then.
>

This is certainly "an" approach that can be used, and I do want to
acknowledge that. But it's not the only way this problem can be solved, and
I worry that this suffers from the many deficiencies that are addressed
earlier in this mail.

[Jakob Bohm]

On 2020-11-06 18:31, Jeff Ward wrote:
> ...

>
> Audit reports, whether for WebTrust, financial statements, or other forms of engagement reports providing assurance to users of the information, do not include specific audit team members’ names. Simply stated, this desire to include individual auditor’s qualifications in a public report is not consistent with any other compliance reporting methods or reporting requirements for CAs, or any other auditee for that matter.
>

Most paper-based auditing schemes for company financial records (the
historic work area of auditors) include, on each report, the personal
signature and corresponding printed name of the responsible auditor,
optionally with an abbreviation of their national qualification level
(such as an abbreviation of "Examplarian State Authorized Public
Accountant"). From there, it would be possible for interested parties
to check that a physical person by that name is/was indeed on the roster
of such authorized individuals, but not if/why the State of Exemplar
decided to so include that person. Furthermore, the auditor person
and/or their company may have voluntarily published further details of
their qualifications (in brochures, on websites etc.) and may have
applicable original degree documents framed and hanging on their walls
for all concerned to readily inspect.

In terms of GDPR, the state would have published rules for how to get
added/removed from the public roster, and each auditor would have the
opportunity, at all times, to retract their self-descriptions and/or
remove some or all of their framed documents from their public office.

A modern equivalent procedure for CA audits could be:

1. Each Auditor has their name and a unique public nickname registered
in a non-public roster at either CPA Canada or the relevant European
counterpart. This is done to fulfill the contractual obligation of
their professional oath of responsibility. The roster organization
might optionally provide alias e-mails based on the nicks.

2. Each non-public roster operates a public online service which will
confirm or deny the presence of a name/nick pair, with appropriate
safeguards against attempts to extract the roster by systematic polling
of made up names.

Unless otherwise stated in public by Mozilla (such as the statements
made a few years ago about certain auditors from E&Y), any auditor on
these rosters shall be presumed sufficiently qualified to sign audits
used by Mozilla.

3. Each auditor person signs his public audit letters with his name,
nick, a reference to the roster-keeping organization and any other
honorific titles he/she may legitimately choose to use. He does this to
satisfy his contractual obligation to provide the CA with that letter.
Any official physical copies will have his physical signature above his
name and may also carry a physical stamp or seal of him or his
organization, as dictated by local legal traditions.

4. Each such public audit letter is submitted to a public repository
operated by the roster-keeping organization, using a procedure that
verifies that the letter was submitted exactly as given, by that named
auditor from their roster. This is done to satisfy the contractual
obligation of the auditor towards the CA in accordance with a
contractual reference to terms of the roster-keeping organization.

5. The roster-keeping organization publishes the public audit letters in
both a traditional paper journal deposited at major public archives and
as an online readily accessible web site with a Merkel hash tree
providing public verification that each letter was in the public record
on or before the stated inclusion date. As hash algorithms fail to
future cracks, the roster-keeping organization retains the ability to
regenerate the signatures using new algorithms, based on its offline
archive of originals, including a signed public statement of said
regeneration. This publication of records that include the identity of
both the actual auditors as well as relevant principal CA Officers is
done to further satisfy the contractual obligations in #4. As is common
in paper-based book-keeping, retractions can be filed as separate
letters of correction, and the retracted documents may be made invisible
to the public without invalidating the hash-tree.
For public access, each public letter is given a unique permanent URL
to which the CA may publicly refer, including in the CADB and on its
website.

6. Each auditor shall submit for publication by the roster organization
a self-authored but roster verified statement of qualifications, usually
just a few paragraphs. Each such statement similarly gets a permanent
URL, but remains visible only until superseded or retracted by either
the auditor or roster-keeper. This publication is done as part of the
auditor's contractual obligations to the roster-keeping organization,
and the ability to retract provides the GDPR right of deletion of any
included details. Links to the current document are published by the
auditor organization (e.g. E&Y) as part of their advertising and as part
of their contractual obligations to the audited CAs.

An example of such a statement could be:

-- Begin example document --
Statement of qualifications of WebTrust auditor Jack F. Honest Esq.
(JAH2):

Jack Fictional Honest graduated with a CPA degree from Harward Law
School in 1975, grade average B+, and worked as classified documents
security inspector for the USAF, reaching the rank of Colonel in 1985.
Jack retired honorably from the army in 1990 to work for DeLoite
auditing, and is now a senior partner in Deloitte's Northern California
Office. Jack also holds a Masters degree in Cryptology from MIT (1998)
and a Bachelors degree in computer software, also from MIT (2009). Jack
was one of the original authors of the IETF public key certificate
standard (PKIX, RFC5280).
-- End of example document --

As previously mentioned, all these statements, if published and not
hidden on the roster website would be verified for truth by the roster-
keeping organization (CPA Canada for WebTrust, some European
organization for eldas), so Relying parties can rely on that information
to be true. Thus Mozilla could trust that an Audit signed
by JAH2 and published on the WebTrust roster, was actually signed by a
WebTrust qualified auditor with these qualifications and not by any
other WebTrust auditor that may never have passed the requirements to
join the WebTrust program, and is not one of the few named auditors that
Mozilla has publically stated they won't accept audits from.




Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

[Dimitris Zacharopoulos]

Can other people, except Ryan, follow this thread? I certainly can't. Too much information, too much text, too many assumptions, makes it impossible to meaningfully participate in the discussion.

[Ryan Sleevi]

These are complex topics, for sure, but that’s unavoidable. Participation
requires a degree of understanding both about the goals to be achieved by
auditing, as well as the relevant legal and institutional frameworks for
these audits. So, admittedly, that’s not the easiest to jump into.

Could you indicate what you’re having trouble following? I don’t know that
we can do much about “too much information”, since that can be said about
literally anything unfamiliar, but perhaps if you would simply ask
questions, or highlight what you’d like to more about, it could be more
digestible?

What would you say your desired outcome from your email to be? Accepting,
for a second that this is a complex topic, and so discussion will
inherently be complex, and so a response such as “make it simpler for me”
is a bit unreasonable.

[Dimitris Zacharopoulos]

I will try to further explain my thoughts on this. As we all know,
according to Mozilla Policy "CAs MUST follow and be aware of discussions
in the mozilla.dev.security.policy
<https://www.mozilla.org/about/forums/#dev-security-policy> forum, where
Mozilla's root program is coordinated". I believe Mozilla Root store
managers' minimum expectations from CAs are to _read the messages and
understand the content of those messages_. Right now, we have [1], [2],
[3], [4], [5], [6], [7], [8], [9] policy-related threads opened up for
discussion since October 15th.

If every post in these threads contained as much information and
complexity as your recent reply to Clemens, I think it eventually
"abuses" the requirement that CAs must follow discussions in m.d.s.p.
and leads to fatigue. Understanding the complicated English language
used, especially for non-Native English speakers, is a very challenging
and difficult task of its own. Therefore, I think it is unreasonable for
Mozilla Root store managers to expect that CAs will follow and
understand all of these discussions if these threads are bombarded with
long and complicate emails that only very few will be able to read and
understand.

I think sending specific questions is a good advice and I will try to do
that next week, but please try to also consider and respect the fact
that CAs have a finite set of resources to work on these issues, among
other duties. An unexpected increase in the volume of information CAs
must follow, creates a risk that something critical might be missed,
despite the good efforts of CAs having allocated the necessary resources
to monitor these lists and Bugzilla incidents.

I obviously can't suggest anyone to post more or less, each person has
the right to post whatever he/she deems necessary. I just wanted you to
know, as a peer to this Module, that some participants of this Root
Program want to contribute and continue to do so, and it would help
tremendously if some messages were shorter and simpler to read. Perhaps
breaking down your long reply into more than one messages might make
them easier to process, I don't know.

Thanks for listening :-)


Dimitris.



[1]:
https://groups.google.com/g/mozilla.dev.security.policy/c/4fhP4iV4ut4/m/WQknrWbhAAAJ
[2]:
https://groups.google.com/g/mozilla.dev.security.policy/c/ZFLsguJyFDo/m/Tmn5rcXhAAAJ
[3]:
https://groups.google.com/g/mozilla.dev.security.policy/c/oJiMmvAJXdI/m/ZhH6oLwpAAAJ
[4]:
https://groups.google.com/g/mozilla.dev.security.policy/c/3sW3_cRBrfo/m/ErldH8JWAQAJ
[5]:
https://groups.google.com/g/mozilla.dev.security.policy/c/Oqd2iKCFELI/m/f9Kfs0M0BAAJ
[6]:
https://groups.google.com/g/mozilla.dev.security.policy/c/DChXLJrMwag/m/uGpEqiEcBgAJ
[7]:
https://groups.google.com/g/mozilla.dev.security.policy/c/nMrORsPPcds/m/hVahATyTBwAJ
[8]:
https://groups.google.com/g/mozilla.dev.security.policy/c/rbSFMYKlfI4/m/3kvOhydWAQAJ
[9]:
https://groups.google.com/g/mozilla.dev.security.policy/c/xk3BanrcljY/m/8dFyM-5pAQAJ

[Ryan Sleevi]

On Sat, Nov 7, 2020 at 4:52 AM Dimitris Zacharopoulos <ji...@it.auth.gr>
wrote:

>
> I will try to further explain my thoughts on this. As we all know,
> according to Mozilla Policy "CAs MUST follow and be aware of discussions in
> the mozilla.dev.security.policy
> <https://www.mozilla.org/about/forums/#dev-security-policy> forum, where
> Mozilla's root program is coordinated". I believe Mozilla Root store

> managers' minimum expectations from CAs are to *read the messages and
> understand the content of those messages*. Right now, we have [1], [2],

> [3], [4], [5], [6], [7], [8], [9] policy-related threads opened up for
> discussion since October 15th.
>
> If every post in these threads contained as much information and
> complexity as your recent reply to Clemens,
>

This seems like a strawman argument, ht I don’t think it’s intentional.

You’re arguing that “if things were like this hypothetical situation, that
would be bad”. However, they aren’t like that situation, as the evidence
you provided shows. This also goes back to the “what is your desired
outcome from your previous mail”, and trying to work out what a clear call
to action to address your concerns. Your previous message, especially in
the context of your (hypothetical) concern, reads like you’re suggesting
“Mozilla shouldn’t discuss policy changes with the community”. I think
we’re all sensitive and aware of the desire not to have too many parallels
discussions, which is exactly why Ben’s been only introducing a few points
a week, to facilitate that and make progress without overwhelming.

As it relates to this thread, or any other thread, it seems the first order
evaluation for any CA is “Will the policy change”, followed by “What do I
need to do to meet the policy?”, both of which are still very early in this
discussion. You’re aware of the policy discussion, and you’re aware a
decision has not been made yet: isn’t that all you need at this point?
Unlike some of the other proposals, which require action by CAs, this is a
proposal that largely requires action by auditors, because it touches on
the audit framework and scheme. It seems like, in terms of expectations for
CAs to participate, discussing this thread with your auditor is the
reasonable step, and working with them to engage here.

Hopefully that helps. Your “but what if” is easily answered as “but we’re
not”, and the “this is a lot, what do I need to do” is simply “talk with
your auditor and make sure they’re aware of discussions here”. That seems a
very simple, digestible call to action?

[Jeff Ward]

Sure Ryan, the answer is quite simple. When I used the word "public" in my post, I should have been more clear as to the nuance of this concept. Public reports by definition are generally distributed (available to anyone). When reports are restricted in distribution and only intended for a certain user or users as specified in the report, they are no longer public. In each of the EY examples you reference, they all state in the last paragraph before the EY signature, "This report is intended solely for the information and use of GPO-CA and the Federal PKI Policy Authority and is not intended to be, and should not be, used by anyone other than GPO-CA and the Federal PKI Policy Authority." When reports are not generally distributed and available to the general public, the specifics of individuals performing the audit will not be present. When my firm issues reports for FPKI, we also provide the listing of individuals in a letter that is not public information.

Jeff

[Ryan Sleevi]

On Sat, Nov 7, 2020 at 9:21 AM Jeff Ward via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Sure Ryan, the answer is quite simple. When I used the word "public" in
> my post, I should have been more clear as to the nuance of this concept.
> Public reports by definition are generally distributed (available to
> anyone). When reports are restricted in distribution and only intended for
> a certain user or users as specified in the report, they are no longer
> public. In each of the EY examples you reference, they all state in the
> last paragraph before the EY signature, "This report is intended solely for
> the information and use of GPO-CA and the Federal PKI Policy Authority and
> is not intended to be, and should not be, used by anyone other than GPO-CA
> and the Federal PKI Policy Authority." When reports are not generally
> distributed and available to the general public, the specifics of
> individuals performing the audit will not be present. When my firm issues
> reports for FPKI, we also provide the listing of individuals in a letter
> that is not public information.
>

Thanks Jeff,

This is useful for confirming that there is a clearly viable path forward
here. As you know, I appreciate the nuance here regarding public reporting,
as well as reports that are restricted in distribution but still made
public (e.g. as part of a regulatory function, such as the OIG in this
case). I think we agree in the substance: that this is possible, and are
merely working out the details here with regards to reporting.

For example, Mozilla could require that, in addition to the "traditional"
WebTrust reporting , Mozilla be named as part of a restricted distribution
report that contains these details. Alternatively, Mozilla could require
that, as part of participating within their root program, CAs ensure such
reports include as restricted distribution those Subscribers, Relying
Parties, and Application Vendors that would rely upon the CAs' services,
much like widely practiced in industry today with respect to cloud
providers.

Would you agree that it's fair to say that it is not fundamentally that
auditors cannot report such information, but focused here on the details of
how that report is delivered to Mozilla?

[Jeff Ward]

Thanks Ryan, I do agree that this information is better placed in a separate communication as you suggest. As we already worked through the restricted use issue with the detailed controls report, we could adopt that same limited distribution language in the added communication you suggest. So now you have me thinking. This separate communication could also be used to discuss other items, especially the locations in scope and visited during the audit. Without having this separate communication, this information will likely be added as another exhibit to the report, especially for the larger, more complex CAs, making a long report even longer in volume. (Side note: length of some reports is an issue when the WebTrust seal is applied). I think if this information was put into the separate communication, we could solve another problem herein by taking this information that is candidly sensitive (specific locations) and move it to a more restricted report. I'll be interested to hear your thoughts on this approach as well.

Jeff

[Dimitris Zacharopoulos]

On 7/11/2020 3:12 μ.μ., Ryan Sleevi wrote:
>
>
> On Sat, Nov 7, 2020 at 4:52 AM Dimitris Zacharopoulos
> <ji...@it.auth.gr <mailto:ji...@it.auth.gr>> wrote:
>
>
> I will try to further explain my thoughts on this. As we all know,
> according to Mozilla Policy "CAs MUST follow and be aware of
> discussions in the mozilla.dev.security.policy
> <https://www.mozilla.org/about/forums/#dev-security-policy> forum,
> where Mozilla's root program is coordinated". I believe Mozilla

> Root store managers' minimum expectations from CAs are to _read
> the messages and understand the content of those messages_. Right

> now, we have [1], [2], [3], [4], [5], [6], [7], [8], [9]
> policy-related threads opened up for discussion since October 15th.
>
> If every post in these threads contained as much information and
> complexity as your recent reply to Clemens,
>
>
> This seems like a strawman argument,  ht I don’t think it’s intentional.
>
> You’re arguing that “if things were like this hypothetical situation,
> that would be bad”. However, they aren’t like that situation, as the
> evidence you provided shows. This also goes back to the “what is your
> desired outcome from your previous mail”, and trying to work out what
> a clear call to action to address your concerns. Your previous
> message, especially in the context of your (hypothetical) concern,
> reads like you’re suggesting “Mozilla shouldn’t discuss policy changes
> with the community”. I think we’re all sensitive and aware of the
> desire not to have too many parallels discussions, which is exactly
> why Ben’s been only introducing a few points a week, to facilitate
> that and make progress without overwhelming.

To the contrary, I want more people to be able to participate in these
discussions, which is precisely why I "complained" about the size of
your response to Clemens :-) Keeping our replies to reasonable levels,
with a mindset that this is an International Internet community and
people might be interested to participate (even auditors that are not
native-English speakers), I believe is a good thing.

I also see that Ben has introduced a lot of policy proposal topics for
discussion in a short period of time, but I don't know what the
expectations about their "discussion time" are. Should anyone just pick
any topic and start a discussion? That might introduce a lot of parallel
discussions and I'm not sure if this is desirable by Ben. Perhaps we
need some coordination on these topics, for example "please send
feedback for topics 1 and 2 before the end of week X. If no feedback is
received, we'll deem the proposal accepted", something like that, before
moving to other topics.

>
> As it relates to this thread, or any other thread, it seems the first
> order evaluation for any CA is “Will the policy change”, followed by
> “What do I need to do to meet the policy?”, both of which are still
> very early in this discussion. You’re aware of the policy discussion,
> and you’re aware a decision has not been made yet: isn’t that all you
> need at this point? Unlike some of the other proposals, which require
> action by CAs, this is a proposal that largely requires action by
> auditors, because it touches on the audit framework and scheme. It
> seems like, in terms of expectations for CAs to participate,
> discussing this thread with your auditor is the reasonable step, and
> working with them to engage here.
>
> Hopefully that helps. Your “but what if” is easily answered as “but
> we’re not”, and the “this is a lot, what do I need to do” is simply
> “talk with your auditor and make sure they’re aware of discussions
> here”. That seems a very simple, digestible call to action?
>

It helps me understand your point of view but it seems that you don't
acknowledge the need to keep these emails to a reasonable and digestible
size, regardless if the intended recipients are auditors, CAs, Relying
Parties. You seem to dismiss my point and the fact that some messages on
this list have been, in fact, very long and very complicated which makes
participation and contributions very difficult. I trust that we are both
interested in truly meeting Mozilla's goal for an open Internet
community (which includes contributions from International
participants), so please help the community by trying to break down
complicated responses into simpler ones, and let's all try to use
shorter answers and to the point.

Indeed, this particular policy change proposal seems to mainly affect
Auditors, but individual members of this community (either representing
CAs or as Relying Parties) might also be interested to participate, just
as Auditors and Relying Parties may participate in discussions around
policy change proposals that affect CAs. FWIW, I think changing the
rules for auditors also affects CAs because it creates an opportunity
for CAs to have engagements with individual auditor persons, as long as
they are accepted by Mozilla.

[Ben Wilson]

Hi Dimitris,

I intend to introduce the remaining discussion topics over the next three
weeks. I did not announce an end to the discussion period on purpose, so
that we can have as full of a discussion as possible. Also, in the next
three weeks, I intend to start summarizing the discussions and coming up
with new suggested language on those issues that have been discussed. I
expect that during December we will start to solidify the amendments to
MRSP (v.2.7.1), and that in January I'll announce a "last call" on the
amendments. Following that I will "summarize a consensus that has been
reached, and/or state the official position of Mozilla" - see
https://wiki.mozilla.org/CA/Updating_Root_Store_Policy.

Part of the discussion that will still need to take place deals with
implementation deadlines, timing, etc. Let's discuss that now for the
non-controversial items, and then in late December / early January for
those that are more contentious (assuming they remain in this batch of
changes).

Sincerely yours,
Ben Wilson
Mozilla Root Store

[Clemens Wanko]

Hi Ryan, hi all,
well, isn’t the point to make here just, that there are multiple ways to ensure proper auditor qualification? No matter which way you like to go however, you must define the details of your regime: what is the criteria you require the auditor to fulfill, how do you organize that these criteria are checked, how do you ensure the quality of this process, how do you publish any results with regard to the auditors qualification, etc.

Now, what we have in place with the EU ETSI scheme (or with WebTrust) is a regime which provides
- normative options specifically adopted to address CA/B-Forum/Browser requirements as well as
- a system to ensure auditor qualification is there and kept upright over time.
The regime is there, proven and flexible to make proper use out of it. And constant effort is made from ETSI as well as from the auditors (ACAB’c) to the Forum in order to incorporate CA/B Forum requirements.

Certainly, there are always alternatives. But the alternative should be clearly defined and described in order to allow an evaluation of all the options before taking a decision. In the current case I like to understand the alternatives you suggest for Mozilla.
Best regards
Clemens

[Dimitris Zacharopoulos]

Thank you Ben, this is really helpful.

Dimitris.

[Ryan Sleevi]

On Mon, Nov 9, 2020 at 11:53 AM Clemens Wanko via dev-security-policy <

dev-secur...@lists.mozilla.org> wrote:

> Hi Ryan, hi all,

> well, isn’t the point to make here just, that there are multiple ways to
> ensure proper auditor qualification? No matter which way you like to go
> however, you must define the details of your regime: what is the criteria
> you require the auditor to fulfill, how do you organize that these criteria
> are checked, how do you ensure the quality of this process, how do you
> publish any results with regard to the auditors qualification, etc.

Clemens,

I see you doubled down on this approach, but I already responded
previously. I think this mindset to compliance does a great disservice, and
also substantially misrepresents, the values and principles at play here.
That is, you've again anchored on the notion here tha compliance should be
a checklist - this exact approach and mindset that has led to countless
security issues and incidents. This is the fundamentally wrong approach
here that I think bears calling out, and it's worth again, emphasizing,
that no, it doesn't have to be like how you describe, and also
(unsurprisingly) overlooks the downsides.

If you examine the posts I referenced previously, you'll see this in
action, so I do hope you will. So your questions are a bit nonsense here,
because they start from a conceptually bad foundation.

<snip>

> Certainly, there are always alternatives. But the alternative should be
> clearly defined and described in order to allow an evaluation of all the
> options before taking a decision. In the current case I like to understand
> the alternatives you suggest for Mozilla.
>

You've turned this thread into a broader discussion of ETSI standards, and
while many criticisms could be fairly stated, I think that detracts from
this thread, and ignores the very thing it was started to discuss. I'd like
to reorient you back to the original purpose, of bringing transparency to
the auditor skills and competencies. It would seem your position is that
there should be no independent evaluation, by affected software vendors, of
the skills and competencies of the auditor or how they perform the audit.
You would like us to rely solely on the NAB and Supervisory Body for
carrying that out, even for a totally unrelated audit scheme (the eIDAS
Regulation), which can incidentally make use of largely-unrelated standards
for audits (the EN 319 403 series). You seem to argue that's superior to
any form of transparency or accountability, and seem to reject the notion
that, were auditors skills and qualifications known and part of the report,
it can tangibly lead to improvements on the requirements.

Frankly, I think that idea is nonsense. We know, from the Supervisory
Bodies involved in the eIDAS Regulation, that there are vast differences in
quality and expectation from CABs. Browsers have shared those same concerns
to ENISA, and ACAB'c seems to recognize it as well, from its very
existence. Yet you seem to reject efforts to improve that, and suggest that
we simply "trust the process" that it'll get sorted out. We have ample
evidence, from Certificate Transparency, about how bringing transparency
reveals systemic issues and flaws. Yet, rather than embrace this for
audits, it's seemingly rejected with unsubstantiated roadblocks.

You've not responded, in substance, to my previous message, and so I'm not
sure how to interpret that, especially since this largely repeats the same
points.

[Kathleen Wilson]

> It is proposed in Issue #192
> <https://github.com/mozilla/pkipolicy/issues/192> that information about
> individual auditor's qualifications be provided--identity, competence,
> experience and independence. (For those interested as to this
independence
> requirement, Mozilla Policy v.1.0 required either disclosure of the
> auditor's compensation or the establishment that the auditor "is bound by
> law, government regulation, and/or a professional code of ethics to
render
> an honest and objective judgement regarding the CA.")

I am very much in favor of increasing transparency about the
qualifications of the auditors providing audit statements for CAs in our
program. However, I think that we need to spend more time figuring out a
few things before adding such a requirement to our policy. Therefore, I
think we should add this to our list of things to spend some focused
time to figure out in early 2021, and move this item to the next version
of Mozilla’s root store policy.

Below are some of the questions we need to be able to answer before
adding this requirement to Mozilla's root store policy.

Please do NOT respond to these questions now. We will have future
discussions about this when we are ready.

- What information is needed and in what format to demonstrate each
individual auditor's qualifications?
- What are the criteria to be considered and what is sufficient to be
considered a qualified auditor?
- How do auditors apply to be considered qualified auditors?
- How can new participants become involved in this space and become
qualified auditors?
- What is the process to determine if an auditor is qualified?
- Does every auditor signing their name or listed in an audit statement
need to be verified as a qualified auditor? Or just the lead auditor?
- How are the qualifications of the auditors communicated in conjunction
with the audit statement(s)?
- Who is responsible for verifying auditor qualifications?
- Who is responsible for maintaining the list of known qualified auditors?
- How do CAs find out if their auditors are qualified?

I look forward to having these discussions in full later, but I think
this effort is too large in scope for version 2.7.1 of Mozilla's Root
Store Policy.

Thanks,
Kathleen

[Kathleen Wilson]

PS: In the meantime, we will continue to verify auditor qualifications
as described here:
https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications

[Ryan Sleevi]

On Thu, Nov 12, 2020 at 7:27 PM Kathleen Wilson via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> I am very much in favor of increasing transparency about the
> qualifications of the auditors providing audit statements for CAs in our
> program. However, I think that we need to spend more time figuring out a
> few things before adding such a requirement to our policy. Therefore, I
> think we should add this to our list of things to spend some focused
> time to figure out in early 2021, and move this item to the next version
> of Mozilla’s root store policy.
>

I think that's a reasonable place, but I do worry that there's a risk that
either progress won't be made until then, and we'll be in the similar
situation of needing more time.

Given that this relates to audits and audit statements, does it make sense
to actually go forward with trying to find a narrowly banded approach, with
a forward dated requirement, such as 12 months after 2.7.1?

The benefit to this approach is that it allows the expectation to be
factored into contractual agreements with auditors, which may be done
several years in advance, as well as provides the necessary signals to
audit firms, that this is a thing of real and concrete value that Mozilla
is committed to. It also allows Mozilla to assess progress, throughout the
year, on the full set of requirements. If we think about other changes that
have required some degree of details to be worked out (e.g. the SHA-1
deprecation, RSA-1024 deprecation, certificate lifetimes), the benefit of
setting a clear expectation ahead of time helps move the conversation from
debating "if" to discussing "how", which can result in more productive
engagement.

Equally, I think there's a lot that can be borrowed from how approaches to
transparency have been done in other regards. For example, with the
introduction of CAA, there was "merely" a requirement to disclose, which
later turned into more concrete criteria and objectives. Similarly, with
respect to organizational validations, an approach being taken right now
for the EV Guidelines is to disclose, with the intent of using such
disclosures to better inform, analyze, and develop concrete requirements,
based on those collective disclosures and interpretations.

In this regard, the principles from Mozilla's 1.0 Certificate Policy
provide a small minimum, along with some of the language from, say, the
FPKI, regarding technical competencies. The basis here is simply for the
auditor to *disclose* why they believe they meet the criteria or objectives
set. This avoids the need to address part of your questions (e.g. "How do
auditors apply to be considered qualified auditors"), because it leaves the
current policies and presumptions in place, but introduces the disclosure
requirement for why the auditor is relevant and reliable for the report.

This is why I took exception to ETSI's approach, which I worry your
questions jump to as well, which is the first step to answering those
questions is understanding the existing set of qualifications and how those
relate to Mozilla's goals and principles, without seeking to establish a
full "qualification regime" out of the gate. By focusing on disclosure, it
allows us to evaluate both "is this what we expect/want", as well as better
address some of the issues (e.g. "We see auditors with skill X provide much
better reports than skill Y; we should consider making X a required skill")

As it relates to Ben's proposal, I think the language he's proposed largely
works, and we can avoid the set of questions you're worried about, by
removing the language "Mozilla will review each individual auditor's

credentials and ensure that any Qualified Auditor has the collective set of

skills required by Section 8.2 of the Baseline Requirements". Additionally,
introducing a forward-dated requirement (e.g. 12 months) helps work through
any of the delivery issues Jeff highlighted, in a way that's mutually
workable. This ensures transparency, without adding a hurdle here. Issues
related to auditors that Mozilla may lose trust in are already covered in
Section 2.3 of the policy, with
https://github.com/mozilla/pkipolicy/issues/146 existing to provide greater
clarity, but I think this can be seen as a contingency.

[Kathleen Wilson]

On 11/13/20 1:43 PM, Ryan Sleevi wrote:
> In this regard, the principles from Mozilla's 1.0 Certificate Policy
> provide a small minimum, along with some of the language from, say, the
> FPKI, regarding technical competencies. The basis here is simply for the
> auditor to *disclose* why they believe they meet the criteria or objectives
> set. This avoids the need to address part of your questions (e.g. "How do
> auditors apply to be considered qualified auditors"), because it leaves the
> current policies and presumptions in place, but introduces the disclosure
> requirement for why the auditor is relevant and reliable for the report.

I think it is reasonable to update section 3.2 of Mozilla's Root Store
Policy in v2.7.1 to re-add information that appears to have been lost
during the efforts to remove duplication with the BRs. And we could
consider adding some incremental changes to improve transparency and
clarify expectations regarding auditor experience.

For example, we could begin by updating section 3.2 to the following,
which is a combination of the versions 2.7 and 2.4.1
(https://github.com/mozilla/pkipolicy/blob/2.4.1/rootstore/policy.md) of
Mozilla's Root Store Policy. And then see if there are incremental
updates to this that will improve transparency while keeping the audit
statements that we add to the CCADB as fully public-facing documents.

===

3.2 Auditors

Mozilla requires that audits MUST be performed by a competent,
independent, qualified party.

The burden is on the CA to prove that it has met the below requirements.
However the CA MAY request a preliminary determination from us regarding
the acceptability of the criteria and/or the competent, independent,
qualified party or parties by which it proposes to meet the requirements
of this policy.

By "competent party" we mean a person or other entity who is authorized
to perform audits according to the stated criteria (e.g., by the
organization responsible for the criteria or by a relevant government
agency) or for whom there is sufficient public information available to

determine that the party is competent to judge the CA’s conformance to
the stated criteria. In the latter case the "public information"

referred to SHOULD include information regarding the party’s:

- knowledge of CA-related technical issues such as public key
cryptography and related standards;
- experience in performing security-related audits, evaluations, or risk

analyses; and
- honesty and objectivity.

By "independent party" we mean a person or other entity who is not
affiliated with the CA as an employee or director and for whom at least
one of the following statements is true:
- the party is not financially compensated by the CA;
- the nature and amount of the party’s financial compensation by the CA
is publicly disclosed; or
- the party is bound by law, government regulation, and/or a

professional code of ethics to render an honest and objective judgement
regarding the CA.

By "qualified party" we mean a person or other entity who meets the
requirements of section 8.2 of the Baseline Requirements. If a CA wishes
to use auditors who do not fit the definition in section 8.2 of the
Baseline Requirements, they MUST receive written permission from Mozilla
to do so in advance of the start of the audit engagement. Mozilla will
make its own determination as to the suitability of the suggested party
or parties, at its sole discretion.

==

Thanks,
Kathleen

[Ben Wilson]

Here is my attempt to reword section 3.2 based on combining MRSP version
2.4.1 with version 2.7.
My approach was to align the concepts of "competent", "independent" and
"qualified" with their more-accepted meanings.
Version 2.4.1 and earlier versions of the Mozilla Root Store Policy mixed
some of these concepts together.

3.2 Auditors

Mozilla requires that audits MUST be performed by a competent, independent,
qualified party.

The burden is on the CA to prove *establish* that it*s auditor* has me*e*t
*s* the below requirements *below*.

However*,* the CA MAY request a preliminary determination from us regarding

the acceptability of the criteria and/or the competent, independent,
qualified party or parties by which it proposes to meet the requirements of
this policy.

By "competent party" we mean a person or other entity *group of persons* who

is authorized to perform audits according to the stated criteria (e.g., by

the organization responsible for the criteria or by a relevant agency) or

for whom there is sufficient public information available to determine that

the party is competent *has sufficient education, experience, and ability*

to judge the CA’s conformance to the stated criteria.

In the latter case, "Public information" referred to SHOULD include

information regarding the party’s:
- knowledge of CA-related technical issues such as public key cryptography
and related standards;
- experience in performing security-related audits, evaluations, or
risk analyses;
and

- honesty and objectivity *ability to deliver an opinion as to the CA’s
compliance with applicable requirements*.

By "independent party" we mean a person or other entity *group of persons* who

is not affiliated with the CA as an employee or director and for whom at
least one of the following statements is true:

the party is not financially compensated by the CA;

the nature and amount of the party's financial compensation by the CA is
publicly disclosed; or

the party is bound by law, government regulation, and/or a professional
code of ethics to render an honest and objective judgement regarding the CA.

By "qualified party" we mean a person or other entity or group of persons who
meets *meeting *the requirements of section 8.2 of the Baseline

[Clemens Wanko]

Hi Ben,
looking at what was suggested so far for section 3.2, it seems that the BR combine and summarize under "qualified" in the BR section 8.2 what you and Kathleen describe with the definitions for "competent" and "independent" parties.

Based upon that, MRSP section 3.2 could be structured in the following way:

***** 1st: definition of "competent party" ******
By "competent party" we mean...

***** 2nd: definition of "independency" ******
By "independent party" we mean...

***** 3rd: now refer to the BR summarizing 1 and 2 up in the term "qualified assessor/auditor" *****
By "qualified party" we mean a person or other entity or group of persons who meet *is meeting * the combination of the requirements defined above for a "competent party" and an "independent party" and as such meets *meeting * the requirements of section 8.2 of the Baseline Requirements.


Further following that idea and syncing it with the wording also used by the BR, the current suggestion for MRSP section 3.2 could be revised/amended as follows:

*****
3.2 Auditors
Mozilla requires that audits MUST be performed by a competent, independent and herewith qualified party.
[...]
By "competent party" we mean a person or other entity *group of persons* who has the proficiency and is authorized to perform audits according to the stated criteria (e.g., by the organization responsible for the criteria or by a relevant agency) and for whom is sufficient public information available to determine and evidence that the party is competent *has sufficient education, experience, and ability* to judge the CA’s conformance to the stated criteria.
In the latter case, "Public information" referred to SHOULD *** -> SHALL - Why not being more strict here?*** include information regarding the party’s:
- evidence of being bound by law, government regulation, or professional code of ethics;

- knowledge of CA-related technical issues such as public key cryptography and related standards;

- experience in performing security-related audits, evaluations, and risk analyses; and

- honesty and objectivity *ability to deliver an opinion as to the CA’s compliance with applicable requirements*.

[...]
*****

Best regards
Clemens

[Ben Wilson]

Thanks, Clemens. I'll take a look.

Also, apparently my redlining was lost when my message was saved to the
newsgroup.

I'll see if I can re-post without the text formatting of strikeouts and
underlines.

On Tue, Jan 26, 2021 at 10:24 AM Clemens Wanko via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Hi Ben,
> looking at what was suggested so far for section 3.2, it seems that the BR
> combine and summarize under "qualified" in the BR section 8.2 what you and
> Kathleen describe with the definitions for "competent" and "independent"
> parties.
>
> Based upon that, MRSP section 3.2 could be structured in the following way:
>
> ***** 1st: definition of "competent party" ******
> By "competent party" we mean...
>
> ***** 2nd: definition of "independency" ******
> By "independent party" we mean...
>
> ***** 3rd: now refer to the BR summarizing 1 and 2 up in the term
> "qualified assessor/auditor" *****

> By "qualified party" we mean a person or other entity or group of persons

> who meet *is meeting * the combination of the requirements defined above
> for a "competent party" and an "independent party" and as such meets
> *meeting * the requirements of section 8.2 of the Baseline Requirements.
>
>
> Further following that idea and syncing it with the wording also used by
> the BR, the current suggestion for MRSP section 3.2 could be
> revised/amended as follows:
>
> *****

> 3.2 Auditors

> Mozilla requires that audits MUST be performed by a competent, independent
> and herewith qualified party.
> [...]

> By "competent party" we mean a person or other entity *group of persons*

> who has the proficiency and is authorized to perform audits according to

> the stated criteria (e.g., by the organization responsible for the criteria

> or by a relevant agency) and for whom is sufficient public information
> available to determine and evidence that the party is competent *has

> sufficient education, experience, and ability* to judge the CA’s
> conformance to the stated criteria.

> In the latter case, "Public information" referred to SHOULD *** -> SHALL -

> Why not being more strict here?*** include information regarding the
> party’s:

> - evidence of being bound by law, government regulation, or professional
> code of ethics;

> - knowledge of CA-related technical issues such as public key cryptography
> and related standards;

> - experience in performing security-related audits, evaluations, and risk

> analyses; and
> - honesty and objectivity *ability to deliver an opinion as to the CA’s
> compliance with applicable requirements*.

> [...]
> *****
>
> Best regards
> Clemens
>

[Ben Wilson]

On second thought, I think that Mozilla can accomplish what we want without
modifying the MRSP
<https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#32-auditors>
(which says audits MUST be performed by a Qualified Auditor, as defined in
the Baseline Requirements section 8.2), and instead adding language to
https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications that
explains what additional information we need submitted to determine that an
auditor is "qualified" under Section 8.2 of the Baseline Requirements.

In other words (paraphrasing from BR 8.2), we would need evidence that the
persons or entities:
1. Are independent from the subject of the audit;
2. Have the ability to conduct an audit that addresses the criteria;
3. Have proficiency in examining Public Key Infrastructure technology,

information security tools and techniques, information technology and
security auditing, and the third-party attestation function;

4. Are accredited in accordance with ISO 17065 applying the requirements
specified in ETSI EN 319 403 *OR* 5. Are licensed by WebTrust;
6. Are bound by law, government regulation, or professional code of ethics
(to render an honest and objective opinion); and
7. Maintain Professional Liability/Errors & Omissions insurance with policy
limits of at least one million US dollars in coverage.

We do some of this already when we check on an auditor's status to bring an
auditor's record current in the CCADB. The edits that we'll make will just
make it easier for us to go through the list above.

Thoughts?

Ben

On Tue, Jan 26, 2021 at 1:36 PM Ben Wilson <bwi...@mozilla.com> wrote:

> Thanks, Clemens. I'll take a look.
>
> Also, apparently my redlining was lost when my message was saved to the
> newsgroup.
>
> I'll see if I can re-post without the text formatting of strikeouts and
> underlines.
>

> On Tue, Jan 26, 2021 at 10:24 AM Clemens Wanko via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
>
>> Hi Ben,
>> looking at what was suggested so far for section 3.2, it seems that the
>> BR combine and summarize under "qualified" in the BR section 8.2 what you
>> and Kathleen describe with the definitions for "competent" and
>> "independent" parties.
>>
>> Based upon that, MRSP section 3.2 could be structured in the following
>> way:
>>
>> ***** 1st: definition of "competent party" ******
>> By "competent party" we mean...
>>
>> ***** 2nd: definition of "independency" ******
>> By "independent party" we mean...
>>
>> ***** 3rd: now refer to the BR summarizing 1 and 2 up in the term
>> "qualified assessor/auditor" *****

>> By "qualified party" we mean a person or other entity or group of persons

>> who meet *is meeting * the combination of the requirements defined above
>> for a "competent party" and an "independent party" and as such meets
>> *meeting * the requirements of section 8.2 of the Baseline Requirements.
>>
>>
>> Further following that idea and syncing it with the wording also used by
>> the BR, the current suggestion for MRSP section 3.2 could be
>> revised/amended as follows:
>>
>> *****

>> 3.2 Auditors
>> Mozilla requires that audits MUST be performed by a competent,

>> independent and herewith qualified party.
>> [...]

>> By "competent party" we mean a person or other entity *group of persons*

>> who has the proficiency and is authorized to perform audits according to

>> the stated criteria (e.g., by the organization responsible for the criteria

>> or by a relevant agency) and for whom is sufficient public information
>> available to determine and evidence that the party is competent *has

>> sufficient education, experience, and ability* to judge the CA’s
>> conformance to the stated criteria.

>> In the latter case, "Public information" referred to SHOULD *** -> SHALL

>> - Why not being more strict here?*** include information regarding the
>> party’s:

>> - evidence of being bound by law, government regulation, or professional
>> code of ethics;

>> - knowledge of CA-related technical issues such as public key
>> cryptography and related standards;

>> - experience in performing security-related audits, evaluations, and risk

>> analyses; and
>> - honesty and objectivity *ability to deliver an opinion as to the CA’s
>> compliance with applicable requirements*.

[Ryan Sleevi]

I'm not sure this approach is very clear about the edits you're making, and
whether pull requests or commits might be clearer, as Wayne did in the
past. If there is a commit, happy to look at it and apologies if I missed
it.

I'm not sure this addresses the issue as raised, or at least, "or entities"
seems to create the same issues that are trying to be addressed, by
thinking in terms of "legal entities" rather than qualified persons.

Your discussion about "auditor's" and "auditor's status" might be misread
as "Audit firm", when I think the issue raised was thinking about "person
performing the audit". The individual persons aren't necessarily licensed
or accredited (e.g. #4/ #5), and may not be the ones that retain PL/E&O
insurance (#7). Further, the individuals might be independent, but the firm
not (#1)

So I think you're really just left with wanting to have a demonstration as
to the members of the audit team and how individual members meet (#2, #3,
#6). Is that right? I think Kathleen's proposal from November got close to
that, and then the remainder is clarifying the language that you've
proposed for 2.7.1, namely "Individuals have competence, partnerships and
corporations do not".

I think the expectation goal is that "Individually, and as an audit team,
they are independent (#1)" (e.g. you can't have a non-independent party
running the audit with a bunch of independent parties reporting to them,
since they're no longer independent), while that collectively the audit
team meets #2/#3, with the burden being to demonstrate how the individuals
on the team meet that.

Is that what you were thinking? Or is my explanation a jumbled mess :)

[Ben Wilson]

Thanks. My current thinking is that we can leave the MRSP "as is" and that
we write up what we want in
https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications, which
is, as you note, information about members of the audit team and how
individual members meet #2, #3, and #6.




On Thu, Jan 28, 2021 at 12:44 PM Ryan Sleevi <ry...@sleevi.com> wrote:

[Ryan Sleevi]

On Thu, Jan 28, 2021 at 3:05 PM Ben Wilson <bwi...@mozilla.com> wrote:

> Thanks. My current thinking is that we can leave the MRSP "as is" and
> that we write up what we want in
> https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications,
> which is, as you note, information about members of the audit team and how
> individual members meet #2, #3, and #6.
>

Is this intended as a temporary fix until the issue is meaningfully
addressed? Or are you seeing this as a long-term resolution of the issue?

I thought the goal was to make the policy clearer on the expectations, and
my worry is that it would be creating more work for you and Kathleen, and
the broader community, because it puts the onus on you to chase down CAs to
provide the demonstration because they didn't pay attention to it in the
policy. This was the complaint previously raised about "CA Problematic
Practices" and things that are forbidden, so I'm not sure I understand the
distinction/benefit here from moving it out?

I think the relevance to MRSP is trying to clarify whether Mozilla thinks
of auditors as individuals (as it originally did), or whether it thinks of
auditors as organizations. I think that if MRSP was clarified regarding
that, then the path you're proposing may work (at the risk of creating more
work for y'all to request that CAs provide the information that they're
required to provide, but didn't know that).

If the issue you're trying to solve is one about whether it's in the audit
letter vs communicated to Mozilla, then I think it should be possible to
achieve that within the MRSP and explicitly say that (i.e. not require it
in the audit letter, but still requiring it).

Just trying to make sure I'm not overlooking or misunderstanding your
concerns there :)

>

[Ben Wilson]

All,

I've modified the proposed change to MRSP section 3.2 so that it would now
insert a middle paragraph that would read:

"A Qualified Auditor MUST have relevant IT Security experience, or have
audited a number of CAs, and be independent and not conflicted. Individuals
have competence, partnerships and corporations do not. Each Audit Report
MUST be accompanied by documentation provided to Mozilla of individual
auditor qualifications sufficient for Mozilla to determine the competence,
experience, and independence of the Qualified Auditor."

See
https://github.com/BenWilson-Mozilla/pkipolicy/commit/57063dc07f5b753184c94dbf5d0d30d0b9b90789

The basis for further interpretation of the above language would still be
section 8.2 of the Baseline Requirements. ("In normal circumstances,
Mozilla requires that audits MUST be performed by a Qualified Auditor, as
defined in the Baseline Requirements section 8.2").

Section 3.1.4 still remains with a proposed subsection 3 - "name(s) and
qualifications of individuals performing the audit, as required by section
3.2."

I anticipate that additional guidance for how CAs should submit this
information will be made available here on the wiki -
https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications.

<https://github.com/BenWilson-Mozilla/pkipolicy/commit/57063dc07f5b753184c94dbf5d0d30d0b9b90789>
Ben

[Jeff Ward]

I wanted to clarify a couple of points. Firms must be independent to do audit/assurance work. If independence is impaired, for example, by one person in the firm performing management functions, the entire firm is no longer independent. Firms have the responsibility to monitor activities of its professionals, which also includes personal investments, to ensure they remain independent.

Also, WebTrust practitioners provide information on the firm and the professionals used on these engagements. The information provided is closely aligned with the Auditor Qualifications you are describing. As you know, CPA Canada provides a listing of qualified audit firms on its website. Working closely with them could also help in instances where auditor qualifications are in question.

And one last item, thank you for hearing us on the listing of auditors performing the engagement. The only place I am aware that lists the audit partner in a comparable world is the signing audit partner on public company audits in the US, which is available on the SEC website. Other than that, I am not aware of any other team member being listed. We have seen listings of team members and related experience summarized on a non-publicly issued letter to management in the US Federal space.

Hope this helps!

Jeff

[Ryan Sleevi]

On Mon, Feb 15, 2021 at 2:03 PM Jeff Ward via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> I wanted to clarify a couple of points. Firms must be independent to do
> audit/assurance work. If independence is impaired, for example, by one
> person in the firm performing management functions, the entire firm is no
> longer independent. Firms have the responsibility to monitor activities of
> its professionals, which also includes personal investments, to ensure they
> remain independent.
>
> Also, WebTrust practitioners provide information on the firm and the
> professionals used on these engagements. The information provided is
> closely aligned with the Auditor Qualifications you are describing. As you
> know, CPA Canada provides a listing of qualified audit firms on its
> website. Working closely with them could also help in instances where
> auditor qualifications are in question.
>
> And one last item, thank you for hearing us on the listing of auditors
> performing the engagement. The only place I am aware that lists the audit
> partner in a comparable world is the signing audit partner on public
> company audits in the US, which is available on the SEC website. Other
> than that, I am not aware of any other team member being listed. We have
> seen listings of team members and related experience summarized on a
> non-publicly issued letter to management in the US Federal space.

Jeff,

https://www.oversight.gov/sites/default/files/oig-reports/18-19.pdf

Is an example, which is an audit of the U.S. Government Printing Office,
provided by a WTTF member, against the US Federal PKI CP. This doesn’t meet
the criteria you mentioned (public company, SEC), and itself was provided
several years ago.

It is directed to a set of named parties, and made publicly available by
those parties, using the WebTrust for CAs criteria. On page 4 (report)/6
(FPKI submission)/9 (PDF page), you can see an enumerated list of audit
participants and their applicable skills, summarized.

Since you mentioned “a comparable world”, the BSI C5 controls, which
provide a valuable model for improvements in transparency and thoroughness
of reporting (aka the so called “detailed controls” report), notes this
within Section 3.5.1 of the Controls [1]

“As part of the reporting, it must be specified which of the professional
examinations/certifications are held by the audit team (e. g. in the
section “Independence and quality assurance of the auditor”). Upon request,
appropriate documents (e. g. certificates etc.) must be submitted to the
client.”

Could you clarify whether you and the WTTF considered these two cases? The
former is an example of using an assurance scheme the FPKIMA has said on
its own is insufficient, namely WTCA, but with additional reporting can be
made sufficient. The latter is an example of a scheme specifically adapted
for cloud/vendor security controls against an ISAE 3000 reporting scheme,
which is nearly identical to WTBRs in that regard. It was unclear if y’all
were simply not familiar with these cases, or if you believe there is
substantive differences in the proposal here that may require addressing.

[1]
https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/Publications/CloudComputing/ComplianceControlsCatalogue-Cloud_Computing-C5.pdf?__blob=publicationFile&v=3

>

[Jeff Ward]

Correct Ryan. This is one of the examples you provided previously. This report is of course restricted:
"This report is intended solely for the information and use of {CA} and the Federal PKI Policy Authority and is not intended to be, and should not be, used by anyone other than {CA} and Federal PKI Policy Authority."

As you know, this report then is not generally / publicly distributed as it is a restricted use report. This restriction does not appear in the public company audit opinions, hence the reference. I called out the federal space with this very example in mind. So yes, I am aware of and quite familiar with this scenario. It is more about the restricted use (or in this case the lack thereof) as it is the framework being used. The very point you are referencing an opinion that you are using outside of the restriction sums up my argument. I can't speak for all audit firms as this is more of a risk management issue, but my firm would be fine issuing this report in a restricted manner, unless it became known it would not actually be used in the restricted manner. That defeats the whole purpose. I've issued auditor qualifications in this manner in a management letter, which is also restricted. To my knowledge, that letter has never been made available to those outside of the restricted use, as it appears to have been in this case. So unfortunately your statement is not true. This report is not, and was never meant to be "made publicly available".

[Ryan Sleevi]

Apologies for belaboring the point, but I think we might be talking past
eachother.

You originally stated “The only place I am aware that lists the audit

partner in a comparable world is the signing audit partner on public

company audits in the US, which is available on the SEC website.” I gave
two separate examples of such, and you responded to one (FPKI) by saying
the report was not public (even when it is made available publicly), and
the other I didn’t see a response to.

This might feel like nit-picking, but I think this is a rather serious
point to work through, because I don’t think you’re fully communicating
what you judge to be a “comparable world”, as it appears you are dismissing
these examples.

I can think of several possible dimensions you might be thinking are
relevant, but rather than assume, I’m hoping you can expand with a few
simple questions. Some admittedly seem basic, but I don’t want to take
anything for granted here.

1) Are you/the WTTF familiar with these audit schemes?

2) Are you aware of schemes that require disclosing the relevant skills and
experience of the audit team to the client? (E.g. as done by BSI C5 audits
under ISAE 3000)

3) Are you aware of such reports naming multiple parties for the use of the
report (e.g. as done by FPKI audits)

4) Are you aware of schemes in which a supplier requires a vendor to be
audited, and ensures that the customer of supplier are able to access such
audits as part of their reliance upon supplier? (Note, this doesn’t have to
be limited to ISMS systems)

I’m trying to understand what, given the prevalence of these practices,
makes these instances *not* a comparable world, since it seems that helps
move closer to solutions.

[Jeff Ward]

Ryan, I hope you are not suggesting I am dodging you points. That would be absurd. Let me use different words as comparable world seems to be tripping you up. You are not providing a general/public distribution example to make your point so it is baseless. You are using a restricted opinion from EY and neither Ryan Sleevi nor Google are listed as the two intended users. The closest I have seen to support your desire to name individual auditors is in public company audit reports, which are housed on the SEC website. To be clear, of your two examples, one is an opinion, which is restricted, and the other represents the guidelines. Perhaps you have seen a public/general distribution report from your second opinion as I do not see it in this thread. I am aware, as mentioned previously, of the Federal PKI program desiring to know more about team members, but that is not listed in a non-restricted report, in a public/general distribution format.

EY did the FPKI audit. I am not sure why you keep tagging the as a WTTF member. They are a global firm so if you are implying only they know the standards/rules (which I hope you are not) would be misleading. But to answer you question #1, yes. We even spoke last about this in our TF meeting last week and every member had the same response, including the one you have referenced. #2 answered previously. We are not arguing who wants what. The fact this information is desired is not being debated, rather how it is reported to the user. #3 question is unclear. I am not aware of any report that restricts an opinion to certain users specifically, in the case you mentioned the CA and FPKI that allows additional users to get this information. SOC2 for example has a broader restriction which allows the reports to go to a class or classes of users. Your example is not that case. #4 I am definitely aware of this requirement. A public/general distribution report can be shared with anyone. The restriction dictates who gets the opinion. This is the main point you are not understanding Ryan. For example, I if perform an audit of a company and restrict it to them and one other user, say their bank, the engagement letter / statement of work would clearly reflect this restriction. In addition, the standard terms would require the company to get permission to issue the report beyond the specified users. The example you raise in this question is certainly covered under the broad type of restriction that SOC2 provides, as they would be knowledgeable about the subject matter. The EY report example you provided does not include the broader use. And not to belabor this point, but the restriction precludes its public/general distribution. The words matter. When the distribution of a report is tightly binding two parties, as in your example, you can't distribute it broader. Restricted reports by definition are different.

This is a long dialogue supporting Ben's approach. Firms will most likely be willing to provide the qualifications of auditors in a non-public manner as Ben has suggested.

Jeff

[Ryan Sleevi]

On Mon, Feb 15, 2021 at 6:07 PM Jeff Ward via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Ryan, I hope you are not suggesting I am dodging you points. That would
> be absurd. Let me use different words as comparable world seems to be
> tripping you up.

I'm not trying to suggest you're dodging intentionally, but as I said, I
think we were, and to a degree still are, talking past each other. I think
your reply captured that you don't think I understood your phrasing,
which I didn't, but I also don't think you understood why I was trying to
unpack that phrase more.

> You are not providing a general/public distribution example to make your
> point so it is baseless. You are using a restricted opinion from EY and
> neither Ryan Sleevi nor Google are listed as the two intended users.

I'm not sure why you're bringing in Google with respect to
https://wiki.mozilla.org/CA/Policy_Participants . I'm also not sure how
this relates to the specific questions I asked, which were an attempt to
reset the conversation here to try and make progress on finding solutions.

> The closest I have seen to support your desire to name individual auditors
> is in public company audit reports, which are housed on the SEC website.
> To be clear, of your two examples, one is an opinion, which is restricted,
> and the other represents the guidelines. Perhaps you have seen a
> public/general distribution report from your second opinion as I do not see
> it in this thread. I am aware, as mentioned previously, of the Federal PKI
> program desiring to know more about team members, but that is not listed in
> a non-restricted report, in a public/general distribution format.

Sure, and the purpose of the questions was an attempt to reset the
conversation to a point where we're not talking past each other, by trying
to build up from basics. I know we both know there are many different audit
criteria that are used, and the WTTF member organizations are global
organizations. I don't want to assume that those who focus on the WTTF are
as familiar with, say, the BSI requirements or other audit criteria, much
the same way it wouldn't be reasonable to expect I know as much about AI as
PKI :)

The example report I provided wasn't about calling out an individual
member, but rather to highlight an area that, objectively, seems
comparable, particularly to Ben's proposed language, to better understand
what you meant by that, and whether you had concerns with the new approach
still.

You've anchored here on the public disclosure part, which I can understand
and appreciate the risk perceived by having auditor names attached to the
reports, instead of just firms. I see most of your answers focus on a
specific form of disclosure, but that wasn't what my previous mail was
trying to get to yet. I was trying to work back to understand where
"comparable" criteria diverge, so we can figure out possible paths.

To be clear, I support Ben's proposal as a very useful and meaningful
interim step. Your original message was a bit ambiguous about where you
stand, in large part because your use of "comparable" left quite a bit
ambiguous, and whether you thought the changes were sufficient. I'm quite
glad to hear that it sounds workable for those performing WebTrust audits,
because that helps make forward progress.

However, I still think it'd be useful to view this as an interim step, and
it certainly feels like you might be saying this as far as things can go.
In the cloud services space, we regularly see end users expecting audits of
their cloud provider, and those transitively apply to dependencies of that
cloud provider (e.g. datacenter security, software vendors used by the
cloud provider, etc). This is similarly true with respect to software
supply chain security: working out not just first order dependencies, but
working through second and third-order dependencies as well, to build up a
holistic risk picture. Browsers rely on and delegate to CAs, so it's no
surprise that browsers expect audits for CAs. Yet browsers have their own
customers, and need to provide those customers the assurances they need,
and their customers need, etc.

My Question #5 was an attempt to unpack "comparable" further, because these
are hardly unique problems to software vendor/service provider
relationships. It's not even unique to "cloud" or "PKI", as supply chain
assurances are pretty universal, as shown by goods such as coconut milk,
coffee, chocolate, or diamonds. You answered in the context of "public
report", but I wasn't trying to go there yet. I was trying to figure out
where "comparable" split off, and what schemes were and weren't
comparable, so that we can continue to make improvements. Now that it
seems we have a path forward for direct disclosure, I think it's reasonable
to spend time thinking about how to bring that assurance to browser's
users/customers, such as this community, in the same way we help bring
transparency to cloud provider customers and their customers. We don't have
to do it right now, in this thread, but I also don't think we can say this
is as far as we can go, which I took this most recent reply to be
suggesting.

[Watson Ladd]

I can click on the URL and read it. This seems to be the very definition of public, even if the audit report says it is not for reliance upon by the general public. I fully appreciate that this may be a technicality in the world of auditing, but it is very confusing to those of us who are less familiar.

> Jeff

[Ben Wilson]

All,

I have edited the proposed resolution of Issue #192
<https://github.com/BenWilson-Mozilla/pkipolicy/commit/70db4d97f6748075fa760555c1eabb81bd7914ee>
as follows:

Subsection 3 of MRSP Section 3.1.4. would read:

"The publicly-available documentation relating to each audit MUST contain
at
least the following clearly-labelled information: ...

3. name of the lead auditor and qualifications of the team performing the
audit, as required by section 3.2;
..."

Section 3.2 would read:

"A Qualified Auditor MUST have relevant IT Security experience, or have

audited a number of CAs, and be independent and not conflicted. People have

competence, partnerships and corporations do not. Each Audit Report MUST be

accompanied by documentation provided to Mozilla of the audit team
qualifications
<https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications>

sufficient for Mozilla to determine the competence, experience, and
independence of the Qualified Auditor."

The wiki page linked above will provide further details on how to submit
documentation of the audit team's qualifications (which may be separate
from the audit letter itself).

Ben

<https://github.com/BenWilson-Mozilla/pkipolicy/commit/70db4d97f6748075fa760555c1eabb81bd7914ee>

On Mon, Feb 15, 2021 at 6:01 PM Watson Ladd via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On Monday, February 15, 2021 at 3:07:12 PM UTC-8, Jeff Ward wrote:

> I can click on the URL and read it. This seems to be the very definition
> of public, even if the audit report says it is not for reliance upon by the
> general public. I fully appreciate that this may be a technicality in the
> world of auditing, but it is very confusing to those of us who are less
> familiar.
>
> > Jeff

[Ben Wilson]

All,
Kathleen and I discussed the language of this proposal and have modified it
for MRSP section 3.2 as follows: "A Qualified Auditor MUST have relevant
IT Security experience, or have audited a number of CAs, and be independent.

Each Audit Report MUST be accompanied by documentation provided to Mozilla

of the audit team qualifications sufficient for Mozilla to determine the
competence, experience, and independence of the auditor."
Ben

On Thu, Feb 18, 2021 at 11:27 AM Ben Wilson <bwi...@mozilla.com> wrote:

> All,
>
> I have edited the proposed resolution of Issue #192
> <https://github.com/BenWilson-Mozilla/pkipolicy/commit/70db4d97f6748075fa760555c1eabb81bd7914ee>
> as follows:
>
> Subsection 3 of MRSP Section 3.1.4. would read:
>
> "The publicly-available documentation relating to each audit MUST contain
> at
> least the following clearly-labelled information: ...
>

> 3. name of the lead auditor and qualifications of the team performing the
> audit, as required by section 3.2;
> ..."
>
> Section 3.2 would read:

>
> "A Qualified Auditor MUST have relevant IT Security experience, or have

> audited a number of CAs, and be independent and not conflicted. People

> have competence, partnerships and corporations do not. Each Audit Report

> MUST be accompanied by documentation provided to Mozilla of the audit team
> qualifications
> <https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications>

> sufficient for Mozilla to determine the competence, experience, and
> independence of the Qualified Auditor."
>

> The wiki page linked above will provide further details on how to submit
> documentation of the audit team's qualifications (which may be separate
> from the audit letter itself).
>
> Ben
>
>
> <https://github.com/BenWilson-Mozilla/pkipolicy/commit/70db4d97f6748075fa760555c1eabb81bd7914ee>
>
>
>
> On Mon, Feb 15, 2021 at 6:01 PM Watson Ladd via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
>
>> On Monday, February 15, 2021 at 3:07:12 PM UTC-8, Jeff Ward wrote:

[Wanko Clemens]

Dear Ben, Dear Kahtleen,



we suppose, there are no other changes intendent then apart from what you say below?

If the rest of section 3.2 remains as it is, the specific matter of the ETSI auditor qualification would be addressed through the referrer back to BRG section 8.2. It would then read like this, which is okay for us:



"3.2 Auditors

In normal circumstances, Mozilla requires that audits MUST be performed by a Qualified Auditor, as defined in the Baseline Requirements section 8.2.

A Qualified Auditor MUST have relevant IT Security experience, or have audited a number of CAs, and be independent. Each Audit Report MUST be accompanied by documentation provided to Mozilla of the audit team qualifications sufficient for Mozilla to determine the competence, experience, and independence of the auditor."

Otherwise a link to the Wiki would be necessary for clear definition of the details for the auditor qualification.



Apart from that: is it also intended to change section 3.1.4?



All the best

Clemens



-----Ursprüngliche Nachricht-----
Von: mozilla.dev.s...@googlegroups.com <mozilla.dev.s...@googlegroups.com> Im Auftrag von Ben Wilson
Gesendet: Dienstag, 9. März 2021 00:31
An: mozilla-dev-security-policy <mozilla-dev-s...@lists.mozilla.org>
Betreff: EXT: Re: Policy 2.7.1: MRSP Issue #192: Require information about auditor qualifications in the audit report

All,

Kathleen and I discussed the language of this proposal and have modified it for MRSP section 3.2 as follows: "A Qualified Auditor MUST have relevant IT Security experience, or have audited a number of CAs, and be independent.

Each Audit Report MUST be accompanied by documentation provided to Mozilla of the audit team qualifications sufficient for Mozilla to determine the competence, experience, and independence of the auditor."

Ben





On Thu, Feb 18, 2021 at 11:27 AM Ben Wilson < <mailto:bwi...@mozilla.com> bwi...@mozilla.com> wrote:



> All,

>

> I have edited the proposed resolution of Issue #192

> <https://github.com/BenWilson-Mozilla/pkipolicy/commit/70db4d97f674807

> 5fa760555c1eabb81bd7914ee>

> as follows:

>

> Subsection 3 of MRSP Section 3.1.4. would read:

>

> "The publicly-available documentation relating to each audit MUST

> contain at least the following clearly-labelled information: ...

>

> 3. name of the lead auditor and qualifications of the team performing

> the audit, as required by section 3.2; ..."

>

> Section 3.2 would read:

>

> "A Qualified Auditor MUST have relevant IT Security experience, or

> have audited a number of CAs, and be independent and not conflicted.

> People have competence, partnerships and corporations do not. Each

> Audit Report MUST be accompanied by documentation provided to Mozilla

> of the audit team qualifications

> < <https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications> https://wiki.mozilla.org/CA/Audit_Statements#Auditor_Qualifications>

> sufficient for Mozilla to determine the competence, experience, and

> independence of the Qualified Auditor."

>

> The wiki page linked above will provide further details on how to

> submit documentation of the audit team's qualifications (which may be

> separate from the audit letter itself).

>

> Ben

>

>

> <https://github.com/BenWilson-Mozilla/pkipolicy/commit/70db4d97f674807

> 5fa760555c1eabb81bd7914ee>

>

>

>

> On Mon, Feb 15, 2021 at 6:01 PM Watson Ladd via dev-security-policy <

>> <mailto:dev-secur...@lists.mozilla.org> dev-secur...@lists.mozilla.org

>> <https://lists.mozilla.org/listinfo/dev-security-policy> https://lists.mozilla.org/listinfo/dev-security-policy

>>

>

[Ben Wilson]

All,
As discussed previously, here is a draft amendment to the Audit Statements
wiki page for your review and comment:
https://wiki.mozilla.org/CA/Audit_Statements#Providing_Auditor_Qualifications
Sincerely yours,
Ben

[Ben Wilson]

All,

Here, for your review and comment, is the final version of the wiki page
guidance on providing auditor qualifications. I appreciate the input we
received from ETSI and WebTrust audit groups on this current version.

https://wiki.mozilla.org/CA/Audit_Statements#Providing_Auditor_Qualifications

Please also let me know if you have any questions.

Thanks,

Ben

On Fri, Mar 26, 2021 at 3:20 PM Ben Wilson <bwi...@mozilla.com> wrote:

> All,