Proposal: Add section 5.1 to the Common CCADB Policy

[Kathleen Wilson]

All,

I will greatly appreciate your thoughtful and constructive feedback on
the following proposal to add a section to the Common CCADB Policy,
https://www.ccadb.org/policy

Proposal: Add section 5.1 to the Common CCADB Policy, as follows.
~~
5.1 Audit Statement Content

CCADB uses an Audit Letter Validation (ALV) tool to automatically parse
and validate audit statements. This system eliminates manual processing,
but it requires audit statements to follow some basic rules in order to
function properly. If the audit statement fails to meet any of the
following requirements, the CA will be asked to work with their auditor
to provide an audit statement that passes ALV.

Audit statements listed in the CCADB must contain at least the following
clearly-labelled information in English:

1. Name of the organization performing the audit;
2. Full name of the CA that was audited;
3. SHA-256 fingerprint of each root and intermediate certificate that
was in scope of the audit (see format specifications below);
4. List of the CA policy documents (with version numbers) referenced
during the audit;
5. Whether the audit is for a period of time or a point in time;
6. Date the audit statement was written (see date format specifications
below);
7. Start date and end date of the period that was audited, for those
that cover a period of time (this is not the period the auditor was
on-site);
8. Point-in-time date, for those that are for a point in time;
9. Full names and version numbers of the audit standards that were used
during the audit; and
10. For ETSI, a statement to indicate if the audit was a full audit, and
which parts of the criteria were applied, e.g. DVCP, OVCP, NCP, NCP+,
LCP, EVCP, EVCP+, QCP-w, Part1 (General Requirements), and/or Part 2
(Requirements for trust service providers).

ETSI Audits: Audits conducted by certified ETSI auditors must have their
audit statement uploaded to their auditor’s website. CAs provide the URL
to the audit statements on the auditor’s website, and ALV will verify
those URLs against a known list of audit locations.

WebTrust Audits: Audits conducted by certified WebTrust auditors must
have a WebTrust Seal. CAs enter the URL to the WebTrust Seal into the
CCADB, and upon saving of the record, the CCADB automatically converts
the URL to point to the corresponding PDF file via integration with CPA
Canada.
- For qualified WebTrust audits, CAs may attach the audit statement to a
Bugzilla Bug and provide that URL. Additionally, the CA needs to provide
an explanation about the findings and timeframe for resolution of the
findings.

Format Specifications for SHA-256 Fingerprints:
- MUST: No colons, no spaces, and no linefeeds
- MUST: Uppercase letters
- SHOULD: be encoded in the document (PDF) as “selectable” text, not an
image

Format Specifications for Dates: The following formats are accepted by ALV
- Month DD, YYYY example: May 7, 2016
- DD Month YYYY example: 7 May 2016
- YYYY-MM-DD example: 2016-05-07
- Month names in English
- No extra text within the date, such as “7th” or “the”

~~

Thanks,
Kathleen

Relevant links:
- https://github.com/mozilla/www.ccadb.org/issues/33
-
https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy#314-public-audit-information
-
https://social.technet.microsoft.com/wiki/contents/articles/31635.microsoft-trusted-root-certificate-program-audit-requirements.aspx#E_Audit_Attestation
- https://www.ccadb.org/cas/fields#uploading-documents

[Ryan Sleevi]

Some comparisons, from the Browser/Root Program Alignment proposal
circulated at
https://github.com/cabforum/documents/compare/master...sleevi:2019-10-Browser_Alignment

On Wed, Oct 30, 2019 at 1:52 PM Kathleen Wilson via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> All,
>
> I will greatly appreciate your thoughtful and constructive feedback on
> the following proposal to add a section to the Common CCADB Policy,
> https://www.ccadb.org/policy
>
> Proposal: Add section 5.1 to the Common CCADB Policy, as follows.
> ~~
> 5.1 Audit Statement Content
>
> CCADB uses an Audit Letter Validation (ALV) tool to automatically parse
> and validate audit statements. This system eliminates manual processing,
> but it requires audit statements to follow some basic rules in order to
> function properly. If the audit statement fails to meet any of the
> following requirements, the CA will be asked to work with their auditor
> to provide an audit statement that passes ALV.
>
> Audit statements listed in the CCADB must contain at least the following
> clearly-labelled information in English:
>
> 1. Name of the organization performing the audit;
>

Suggestion: name /and address/ of the organization performing the audit

> 2. Full name of the CA that was audited;
> 3. SHA-256 fingerprint of each root and intermediate certificate that
> was in scope of the audit (see format specifications below);
>

Microsoft policy actually requires the disclosure of the full hierarchy
(c.f.
https://docs.microsoft.com/en-us/security/trusted-root/program-requirements
3.4)

This may help, or harm, the approach used with ALV. There is benefit in
disclosing what is known-and-out-of-scope, but this may cause ALV to
believe that it was in-scope. I've seen CAs disclose explicitly what's out
of scope (e.g. Amazon), and I find this hugely valuable. You can see the
proposed wording from the BR is actually more explicit:

"""the full PKI hierarchy of all certificates that are capable of being
used to issue new certificates, identified by Distinguished Name and the
SHA-256 fingerprint of each and every certificate, and including all Roots,
Subordinate CA Certificates, and Cross Certificates, clearly identifying
which were certificates (and associated keys) were in-scope and
out-of-scope of the audit;"""

> 4. List of the CA policy documents (with version numbers) referenced
> during the audit;
> 5. Whether the audit is for a period of time or a point in time;
> 6. Date the audit statement was written (see date format specifications
> below);
>

The existing Mozilla policy is slightly clearer that this date will always
be after the start/end date for the period, which helps resolve some of the
confusion for period of time.

e.g.
"9. the date the report was issued, which will necessarily be after the end
date or point in time date; and"

> 7. Start date and end date of the period that was audited, for those
> that cover a period of time (this is not the period the auditor was
> on-site);
> 8. Point-in-time date, for those that are for a point in time;
> 9. Full names and version numbers of the audit standards that were used
> during the audit; and
> 10. For ETSI, a statement to indicate if the audit was a full audit, and
> which parts of the criteria were applied, e.g. DVCP, OVCP, NCP, NCP+,
> LCP, EVCP, EVCP+, QCP-w, Part1 (General Requirements), and/or Part 2
> (Requirements for trust service providers).


> ETSI Audits: Audits conducted by certified ETSI auditors must have their
> audit statement uploaded to their auditor’s website. CAs provide the URL
> to the audit statements on the auditor’s website, and ALV will verify
> those URLs against a known list of audit locations.
>
> WebTrust Audits: Audits conducted by certified WebTrust auditors must
> have a WebTrust Seal. CAs enter the URL to the WebTrust Seal into the
> CCADB, and upon saving of the record, the CCADB automatically converts
> the URL to point to the corresponding PDF file via integration with CPA
> Canada.
> - For qualified WebTrust audits, CAs may attach the audit statement to a
> Bugzilla Bug and provide that URL. Additionally, the CA needs to provide
> an explanation about the findings and timeframe for resolution of the
> findings.
>

Erm, is CCADB supposed to adopt Bugzilla? I just wasn't sure if this was a
carryover from Mozilla Policy that may not generalize?

Any expectations on authoritative English version? Is that to be left to
root policy?

>
> Format Specifications for SHA-256 Fingerprints:
> - MUST: No colons, no spaces, and no linefeeds
> - MUST: Uppercase letters
> - SHOULD: be encoded in the document (PDF) as “selectable” text, not an
> image
>
> Format Specifications for Dates: The following formats are accepted by ALV
> - Month DD, YYYY example: May 7, 2016
> - DD Month YYYY example: 7 May 2016
> - YYYY-MM-DD example: 2016-05-07
> - Month names in English
> - No extra text within the date, such as “7th” or “the”
>
> ~~
>
> Thanks,
> Kathleen
>
> Relevant links:
> - https://github.com/mozilla/www.ccadb.org/issues/33
> -
>
> https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy#314-public-audit-information
> -
>
> https://social.technet.microsoft.com/wiki/contents/articles/31635.microsoft-trusted-root-certificate-program-audit-requirements.aspx#E_Audit_Attestation
> - https://www.ccadb.org/cas/fields#uploading-documents
>
>
>

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

[Kathleen Wilson]

On 10/31/19 12:52 PM, Ryan Sleevi wrote:
> Some comparisons, from the Browser/Root Program Alignment proposal
> circulated at
> https://github.com/cabforum/documents/compare/master...sleevi:2019-10-Browser_Alignment
>
> On Wed, Oct 30, 2019 at 1:52 PM Kathleen Wilson via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
>
>> All,
>>
>> I will greatly appreciate your thoughtful and constructive feedback on
>> the following proposal to add a section to the Common CCADB Policy,
>> https://www.ccadb.org/policy
>>
>> Proposal: Add section 5.1 to the Common CCADB Policy, as follows.
>> ~~
>> 5.1 Audit Statement Content
>>
>> CCADB uses an Audit Letter Validation (ALV) tool to automatically parse
>> and validate audit statements. This system eliminates manual processing,
>> but it requires audit statements to follow some basic rules in order to
>> function properly. If the audit statement fails to meet any of the
>> following requirements, the CA will be asked to work with their auditor
>> to provide an audit statement that passes ALV.
>>
>> Audit statements listed in the CCADB must contain at least the following
>> clearly-labelled information in English:
>>
>> 1. Name of the organization performing the audit;
>>
>
> Suggestion: name /and address/ of the organization performing the audit

Updated in my copy of the draft:
Name and address of the organization performing the audit;

>
>
>> 2. Full name of the CA that was audited;
>> 3. SHA-256 fingerprint of each root and intermediate certificate that
>> was in scope of the audit (see format specifications below);
>>
>
> Microsoft policy actually requires the disclosure of the full hierarchy
> (c.f.
> https://docs.microsoft.com/en-us/security/trusted-root/program-requirements
> 3.4)
>
> This may help, or harm, the approach used with ALV. There is benefit in
> disclosing what is known-and-out-of-scope, but this may cause ALV to
> believe that it was in-scope. I've seen CAs disclose explicitly what's out
> of scope (e.g. Amazon), and I find this hugely valuable. You can see the
> proposed wording from the BR is actually more explicit:
>
> """the full PKI hierarchy of all certificates that are capable of being
> used to issue new certificates, identified by Distinguished Name and the
> SHA-256 fingerprint of each and every certificate, and including all Roots,
> Subordinate CA Certificates, and Cross Certificates, clearly identifying
> which were certificates (and associated keys) were in-scope and
> out-of-scope of the audit;"""

I will have to look into this. For example, does Microsoft's policy say
that the full CA hierarchy must be disclosed in the audit statement? Or
does their policy just mean that the full CA hierarchy must be disclosed
to Microsoft, which does not necessarily mean public disclosure, and
does not necessarily mean via the audit statement.

Also, I believe that ALV assumes that the SHA-256 fingerprints found in
audit statements are for certs that were in scope of the audit. So the
approach of also listing the SHA-256 fingerprints of certs that were not
in scope might break ALV.

So, it may turn out that we need another requirement saying that SHA-256
fingerprints for certs not in scope of the audit must not be in the
audit statement.

>
>
>> 4. List of the CA policy documents (with version numbers) referenced
>> during the audit;
>> 5. Whether the audit is for a period of time or a point in time;
>> 6. Date the audit statement was written (see date format specifications
>> below);
>>
>
> The existing Mozilla policy is slightly clearer that this date will always
> be after the start/end date for the period, which helps resolve some of the
> confusion for period of time.
>
> e.g.
> "9. the date the report was issued, which will necessarily be after the end
> date or point in time date; and"

Updated in my copy of the draft:
Date the audit statement was written, which will necessarily be after
the audit period end date or point-in-time date (see date format
specifications below);


>
>

Updated in my copy of the draft:
For qualified WebTrust audits, the CA may post the audit statements on
their own website or attach the audit statement to a Bugzilla Bug and
provide that URL.


>

> Any expectations on authoritative English version? Is that to be left to
> root policy?

Just before the list it says: "must contain at least the following
clearly-labelled information in English:"

Do you think it's necessary to say authoritative?
"An authoritative English language version of the publicly-available
audit information MUST be supplied by the Auditor."

Thanks,
Kathleen

[Ryan Sleevi]

Thanks, Kathleen. Snipped the other changes (which sound good), and a few
replies inline below.

On Thu, Oct 31, 2019 at 4:39 PM Kathleen Wilson via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> >> 2. Full name of the CA that was audited;
> >> 3. SHA-256 fingerprint of each root and intermediate certificate that
> >> was in scope of the audit (see format specifications below);
> >>
> >
> > Microsoft policy actually requires the disclosure of the full hierarchy
> > (c.f.
> >
> https://docs.microsoft.com/en-us/security/trusted-root/program-requirements
> > 3.4)
> >
> > This may help, or harm, the approach used with ALV. There is benefit in
> > disclosing what is known-and-out-of-scope, but this may cause ALV to
> > believe that it was in-scope. I've seen CAs disclose explicitly what's
> out
> > of scope (e.g. Amazon), and I find this hugely valuable. You can see the
> > proposed wording from the BR is actually more explicit:
> >
> > """the full PKI hierarchy of all certificates that are capable of being
> > used to issue new certificates, identified by Distinguished Name and the
> > SHA-256 fingerprint of each and every certificate, and including all
> Roots,
> > Subordinate CA Certificates, and Cross Certificates, clearly identifying
> > which were certificates (and associated keys) were in-scope and
> > out-of-scope of the audit;"""
>
>

> I will have to look into this. For example, does Microsoft's policy say
> that the full CA hierarchy must be disclosed in the audit statement? Or
> does their policy just mean that the full CA hierarchy must be disclosed
> to Microsoft, which does not necessarily mean public disclosure, and
> does not necessarily mean via the audit statement.
>
> Also, I believe that ALV assumes that the SHA-256 fingerprints found in
> audit statements are for certs that were in scope of the audit. So the
> approach of also listing the SHA-256 fingerprints of certs that were not
> in scope might break ALV.
>
> So, it may turn out that we need another requirement saying that SHA-256
> fingerprints for certs not in scope of the audit must not be in the
> audit statement.
>

It's unclear Microsoft's position here.
https://docs.microsoft.com/en-us/security/trusted-root/audit-requirements B
indicates that the CA MUST include the entire hierarchy /in/ the scope of
the audit, so it seems the answer is "Yes", but that's not entirely
followed.

The WebTrust Illustrative Guidance provides guidance on how to do this
(e.g. for the non-performance of activities).

The reason I highlight this is that it significantly reduces the
ambiguities that we're seeing with Intermediate ALV and the questionable
reissuance of reports, and/or CA-defined AUP for retroactive reports.
Forcing the disclosure of the explicit scope - and the consideration -
resolves the ambiguity that we presently see, which is "Was it in scope,
and the auditor looked and forgot to say this, or was it out of scope, and
the auditor never looked"

> > Any expectations on authoritative English version? Is that to be left to
> > root policy?
>

> Just before the list it says: "must contain at least the following
> clearly-labelled information in English:"
>

> Do you think it's necessary to say authoritative?
> "An authoritative English language version of the publicly-available
> audit information MUST be supplied by the Auditor."
>

That's a fair question. We've seen some reports provided where it is
provided by a translator independent from the auditor, and that certainly
calls into question the authoritativeness of the audit - e.g. if the
translator mistranslates a phrase, it could affect the level of assurance
and reliance placed upon that audit.

Similarly, we've seen audit reports (along with CP/CPSes) in which they say
that the local-language version controls, and the English version is merely
informative.

So I'm not fully convinced it needs to say authoritative, but that's the
set of scenarios we've seen in the past that are useful to consider.

[Kathleen Wilson]

From Microsoft:

1) Quote: "We do not require CAs to disclose their hierarchy within the
audit letter itself."

2) Summarized: ALV tries to find a match in the Audit Letter for the
SHA256 thumbprint that is sent by CCADB. Listing thumbprints that were
out of scope within an audit letter could cause ALV to produce
inaccurate results. It would be good to state that audit letters MUST
NOT contain the SHA-256 thumbprints for certs that were out of scope.


Thanks,
Kathleen

[Ryan Sleevi]

On Thu, Oct 31, 2019 at 7:20 PM Kathleen Wilson via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> 2) Summarized: ALV tries to find a match in the Audit Letter for the
> SHA256 thumbprint that is sent by CCADB. Listing thumbprints that were
> out of scope within an audit letter could cause ALV to produce
> inaccurate results. It would be good to state that audit letters MUST
> NOT contain the SHA-256 thumbprints for certs that were out of scope.
>

Thanks.

I think it's preferable to avoid that MUST NOT for now, at least within the
CCADB policy. I think it may potentially portend requiring separate audit
letters for different root stores.

[Kathleen Wilson]

All,

The updated proposed section is here:
https://github.com/mozilla/www.ccadb.org/issues/33#issuecomment-548884075

Please let me know if you have any further feedback on this proposed
addition to the Common CCADB Policy.

Thanks,
Kathleen

[Kathleen Wilson]

All,

The proposed section to add to the CCADB Policy (www.ccadb.org/policy)
has been updated and is here:

https://github.com/mozilla/www.ccadb.org/issues/33#issuecomment-558714086

This is the last call for feedback on it.

Thanks,
Kathleen

[Malcolm Doody]

On Tuesday, 26 November 2019 16:53:21 UTC, Kathleen Wilson wrote:
> The proposed section to add to the CCADB Policy (www.ccadb.org/policy)
> has been updated and is here:
>
> https://github.com/mozilla/www.ccadb.org/issues/33#issuecomment-558714086

Typo in "Format Specifications for SHA-256 Fingerprints:"
> HOULD: be encoded in the document (PDF) as select-able text, not an image

SHOULD: be encoded in the document (PDF) as select-able text, not an image

[Kathleen Wilson]

All,

Section 5.1 has been added to the CCADB Policy.

https://www.ccadb.org/policy#51-audit-statement-content

Please let me know if you see any problems with the addition.

Thanks,
Kathleen