Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Misissuance and BR Audit Statements

655 views
Skip to first unread message

Wayne Thayer

unread,
Aug 7, 2018, 1:33:18 PM8/7/18
to mozilla-dev-security-policy
Given the number of incidents documented over the past year [1][2] for
misissuance and other nonconformities, I would expect many of the 2018
period-of-time WebTrust audit statements being submitted by CAs to include
qualifications describing these matters. In some cases, that is exactly
what we’re seeing. One of many positive examples is Deloitte’s report on
Entrust [3] that includes 2 of the 3 issues documented in Bugzilla.

Unfortunately, we are also beginning to see some reports that don’t meet my
expectations. I was surprised by GlobalSign’s clean reports [4] from Ernst
& Young, but after examining their incident bugs, it appears that the only
documented misissuance that occurred during their audit period was placing
metadata in Subject fields. I can understand how this could be regarded as
a minor nonconformity rather than a qualification, but I would have liked
to at least see the issue noted in the reports.

Ernst & Young’s clean reports on Comodo CA [5] is the example that prompted
this message. We have documented the following issues that occurred during
Comodo’s last audit period:
* Misissuance using "CNAME CSR Hash 2" method of domain control validation
(bug 1461391)
* Assorted misissuances and failure to respond to an incident report within
24 hours (bug 1390981)
* CAA misissuance (bugs 1398545,1410834, 1420858, and 1423624 )

I would like to know if Comodo reported these issues to EY. I asked Comodo
this question four weeks ago [6] but have not received a response.

I will acknowledge that ETSI audits are an even bigger problem (Actalis and
SwissSign are recent examples [7][8][9]). Due to the structure of those
audits, there is no provision for issuing a qualified report. WebTrust
audits are theoretically much better in this regard, but only if auditors
actually find and report on issues! I don’t think it is productive to
expect auditors to search Bugzilla for a list of issues to copy into their
reports, but I do think it is reasonable to question the competence and
trustworthiness of the auditor when so many known issues are absent from
their report.

In this particular example, unless additional facts are presented, I plan
to notate the auditor’s record in CCADB with this issue. We have documented
a number of other issues with Ernst & Young - including the
disqualification of their Hong Kong branch - but this is the first issue
I’m aware of from their New York office. We also recently received a very
“good” qualified audit report from EY’s Denmark office on Telia [10].

- Wayne

[1] https://wiki.mozilla.org/CA/Incident_Dashboard
[2] https://wiki.mozilla.org/CA/Closed_Incidents
[3]
https://www.entrustdatacard.com/-/media/documentation/licensingandagreements/entrust_baselinerequirements_2018.pdf?la=en&hash=BC08BAF5AE81B2EE66A2146EE7710FB2F4F33BA6
[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1388488
[5] https://bugzilla.mozilla.org/show_bug.cgi?id=1472993
[6] https://bugzilla.mozilla.org/show_bug.cgi?id=1472993#c5
[7] https://www.actalis.it/documenti-en/actalisca_audit_statement_2018.aspx
[8]
https://it-tuv.com/wp-content/uploads/2018/07/AA2018070301_Audit_Attestation_TA_CERT__SwissSign_Platinum_G2_signed.pdf
[9]
https://it-tuv.com/wp-content/uploads/2018/07/AA2018070303_Audit_Attestation_TA_CERT__SwissSign_Silver_G2_signed.pdf
[10] https://bugzilla.mozilla.org/show_bug.cgi?id=1475115

Ryan Sleevi

unread,
Aug 15, 2018, 5:43:10 AM8/15/18
to Wayne Thayer, mozilla-dev-security-policy
Wayne,

Thanks for raising this. I definitely find it surprising to see nothing
noted on Comodo's report, as you call out.

As another datapoint, consider this recent audit that is reported to be
from DigiCert, by way of Amazon Trust Services' providing the audits for
their externally operated sub-CAs in [A]. The scope of the WebTrust BR
audit report in [B] contains in its scope "DigiCert ECC Extended Validation
Server CA" of
hash FDC8986CFAC4F35F1ACD517E0F61B879882AE076E2BA80B77BD3F0FE5CEF8862,
which [C]. During that time, this CA issued a cert [D] as part of their
improperly configured Onion issuance in [E], which was remediated in early
March, within the audit period for [B]. I couldn't find it listed in the
report.

Looking over that period, there were two other (resolved) DigiCert issues,
[F] and [G], which affect the CAs listed in scope of [B].

I was a bit surprised by this, as like you, I would have expected these to
be called out by both Management's Assertion and the auditor.
http://www.webtrust.org/practitioner-qualifications/docs/item85808.pdf
provides some of the illustrative reports, but it appears to only provide
templates for management on the result of obtaining a qualified report.

[A] https://bugzilla.mozilla.org/show_bug.cgi?id=1482930
[B] https://bug1482930.bmoattachments.org/attachment.cgi?id=8999669
[C] https://crt.sh/?id=23432431
[D] https://crt.sh/?id=351449246
[E] https://bugzilla.mozilla.org/show_bug.cgi?id=1447192
[F] https://bugzilla.mozilla.org/show_bug.cgi?id=1465600
[G] https://bugzilla.mozilla.org/show_bug.cgi?id=1398269#c29
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

Ben Wilson

unread,
Aug 15, 2018, 11:41:10 PM8/15/18
to ry...@sleevi.com, Wayne Thayer, mozilla-dev-security-policy
Thanks, Ryan and Wayne,

Going forward we'll work to improve our management letter disclosures to include reported mis-issuances during the audit period.

Sincerely yours,

Ben

Ben Wilson

unread,
Aug 15, 2018, 11:41:14 PM8/15/18
to ry...@sleevi.com, Wayne Thayer, mozilla-dev-security-policy
Re-sending

Wayne Thayer

unread,
Aug 16, 2018, 12:32:41 AM8/16/18
to Ben Wilson, Ryan Sleevi, mozilla-dev-security-policy
I went ahead and noted these DigiCert audits as a concern on the CCADB
record for Scott S. Perry CPA, PLLC.

I do think it's important for CAs to disclose these issues to their
auditors, but I also expect auditors to discover them.

- Wayne

clemen...@tuv-austria.com

unread,
Aug 16, 2018, 2:11:52 PM8/16/18
to mozilla-dev-s...@lists.mozilla.org
Dear all,
this is a joint response from ETSI ESI and ACABc:

ETSI have published a supplement to its audit requirements specifically to address specific requirements of Mozilla, and other CA/Browser Forum members, for auditing Trust Service Providers that issue Publicly-Trusted Certificates TS 119 403-2. This is available for download at:
https://www.etsi.org/standards-search#search=TS119403-2
With regard to the treatment of non-conformities it says in PTA-4.3-08: The Audit Attestation shall be issued only if no critical non-conformities are identified.

ETSI audits do cover the CA incident management. That includes the whole process including the timely treatment of incidents as well how to guarantee proper and comprehensive responses to incidents. In ETSI EN 310 401 corresponding requirements are not only provided directly by section 7.9 Incident Management but also through the requirement for a ISMS as stated in section 5. Assessing that, the ETSI auditor will look at the CA incident treatment during the Stage 1 as well as during the Stage 2 onsite part of the audit. This includes the treatment of Bugzilla and cert.sh listed issues. Incidents closed by the CA may have resulted in a change in the CA operations. In such cases the auditor checks that the changes are functioning correctly as defined by the CA. In that way the auditor is assessing the incident management as such including possible measures taken to avoid such incidents in the future and at the implemented measures itself.

Another matter is the question of how to handle security related incidents and the counter measures taken by a CA in audit reports. In order to keep the security issue confidential as well as the details of the measures taken by the CA, the accredited CABS (ETSI auditors) decided to document such findings in their detailed audit reports. These detailed reports list all relevant non conformities and the counter measures taken by the CA. It is handed over to the CA in addition to the audit attestation. Based upon that detailed report, the ETSI auditor will compile the Audit Attestation as the browsers have it in their hand. The contents of the Audit Attestation as summary document was agreed upon between ACAB’c and the CA/B Browser Forum. If you regard it helpful to add information about the audit results gained in the area of the CA incident treatment, we can certainly discuss that. We then should reach a common agreement on what exactly we add.

We certainly believe however, that it is not advisable to publish detected weak points. E.g. there might be findings in the way that the CA has NOT correctly treated an incident. In that case the ETSI auditor will document such findings and will not issue a positive report as well as no Audit Attestation. The CA is then obliged to immediately install appropriate counter measures which again will be judged by the auditor. Only if the counter measures are rated sufficient in coverage and suitability by the auditor, he will issue a positive report and an Audit Attestation.

Regards
Clemens

Ben Wilson

unread,
Aug 16, 2018, 2:12:27 PM8/16/18
to Wayne Thayer, Ryan Sleevi, mozilla-dev-security-policy
What about all of the other audit firms?



From: Wayne Thayer <wth...@mozilla.com>
Sent: Wednesday, August 15, 2018 1:09 PM
To: Ben Wilson <ben.w...@digicert.com>
Cc: Ryan Sleevi <ry...@sleevi.com>; mozilla-dev-security-policy <mozilla-dev-s...@lists.mozilla.org>
Subject: Re: Misissuance and BR Audit Statements



I went ahead and noted these DigiCert audits as a concern on the CCADB record for Scott S. Perry CPA, PLLC.



I do think it's important for CAs to disclose these issues to their auditors, but I also expect auditors to discover them.



- Wayne



> dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org>
> https://lists.mozilla.org/listinfo/dev-security-policy
>
_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org>
https://lists.mozilla.org/listinfo/dev-security-policy

Wayne Thayer

unread,
Aug 16, 2018, 5:11:53 PM8/16/18
to Wanko, Clemens, mozilla-dev-security-policy
Thank you for responding on behalf of ETSI ESI and ACABc! I believe that
this is an important topic and I hope that ETSI ESI and ACABc members will
continue to participate in the discussion.

On Thu, Aug 16, 2018 at 11:11 AM clemens.wanko--- via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Dear all,
> this is a joint response from ETSI ESI and ACABc:
>
> ETSI have published a supplement to its audit requirements specifically to
> address specific requirements of Mozilla, and other CA/Browser Forum
> members, for auditing Trust Service Providers that issue Publicly-Trusted
> Certificates TS 119 403-2. This is available for download at:
> https://www.etsi.org/standards-search#search=TS119403-2

>
This is extremely helpful in normalizing ETSI Audit Attestations and
ensuring that they contain the basic information that Mozilla requires, but
it provides no guidance on documenting non-conformities.
>

>
> With regard to the treatment of non-conformities it says in PTA-4.3-08:
> The Audit Attestation shall be issued only if no critical non-conformities
> are identified.
>
> >
Yes, as I mentioned in my original email, this is the fundamental concern I
have with ETSI audits: qualified Audit Attestation reports cannot be
issued. I do not even understand how it is possible for ETSI auditors to
provide an unbroken sequence of period-of-time Audit Attestations if a
"critical non-conformity" occurs. Apparently once the non-conformity is
remediated it is no longer "critical"? This process gives relying parties
no visibility into the problems identified by ETSI audits.
>

> ETSI audits do cover the CA incident management. That includes the whole
> process including the timely treatment of incidents as well how to
> guarantee proper and comprehensive responses to incidents. In ETSI EN 310
> 401 corresponding requirements are not only provided directly by section
> 7.9 Incident Management but also through the requirement for a ISMS as
> stated in section 5. Assessing that, the ETSI auditor will look at the CA
> incident treatment during the Stage 1 as well as during the Stage 2 onsite
> part of the audit. This includes the treatment of Bugzilla and cert.sh
> listed issues. Incidents closed by the CA may have resulted in a change in
> the CA operations. In such cases the auditor checks that the changes are
> functioning correctly as defined by the CA. In that way the auditor is
> assessing the incident management as such including possible measures taken
> to avoid such incidents in the future and at the implemented measures
> itself.
>
> >
Good. But if, for example, a CA were to repeatedly and systematically fail
to respond to incident reports within 24 hours. Would that non-conformity
appear on the Audit Attestation? This has appeared as a qualification on a
number of recently completed WebTrust audit statements.
>

> Another matter is the question of how to handle security related incidents
> and the counter measures taken by a CA in audit reports. In order to keep
> the security issue confidential as well as the details of the measures
> taken by the CA, the accredited CABS (ETSI auditors) decided to document
> such findings in their detailed audit reports. These detailed reports list
> all relevant non conformities and the counter measures taken by the CA. It
> is handed over to the CA in addition to the audit attestation. Based upon
> that detailed report, the ETSI auditor will compile the Audit Attestation
> as the browsers have it in their hand. The contents of the Audit
> Attestation as summary document was agreed upon between ACAB’c and the CA/B
> Browser Forum. If you regard it helpful to add information about the audit
> results gained in the area of the CA incident treatment, we can certainly
> discuss that. We then should reach a common agreement on what exactly we
> add.
>
>
Yes, this would be helpful information, and I agree that there should be a
discussion resulting in auditor guidance on what to report to ensure that
we get consistent information from all auditors.
>

>
> We certainly believe however, that it is not advisable to publish detected
> weak points. E.g. there might be findings in the way that the CA has NOT
> correctly treated an incident. In that case the ETSI auditor will document
> such findings and will not issue a positive report as well as no Audit
> Attestation. The CA is then obliged to immediately install appropriate
> counter measures which again will be judged by the auditor. Only if the
> counter measures are rated sufficient in coverage and suitability by the
> auditor, he will issue a positive report and an Audit Attestation.
>
> >
Any sufficiently serious vulnerability should of course be fixed
immediately. Once it has been fixed, there is value in disclosing the
problem. It provides relying parties with information on the risks they are
taking in trusting the CA and can result in the development of better
practices across the industry. This is consistent with Mozilla's incident
response guidance [1].
>

> Regards
> Clemens
>
> - Wayne

[1] https://wiki.mozilla.org/CA/Responding_To_An_Incident

Wayne Thayer

unread,
Oct 12, 2018, 3:05:37 PM10/12/18
to mozilla-dev-security-policy
Comodo responded to my question about disclosure of incidents to their
auditor with the following statement [1]:

It turns out that we did not disclose these to EY. That was down to
Comodo CA not offering the evidence of these events during the audit
evidence gathering phase. It was not our intention to mislead and we
regret this short-coming on our part.
We have amended our own internal audit process to make sure these are
gathered up as part of the evidence pack to go to the auditors in
future audit cycles.

I think we should make it a Mozilla program requirement that CAs disclose
incidents and misissuance to their auditors [2].

- Wayne

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1472993#c15
[2] https://github.com/mozilla/pkipolicy/issues/154

On Wed, Aug 15, 2018 at 12:09 PM Wayne Thayer <wth...@mozilla.com> wrote:

> I went ahead and noted these DigiCert audits as a concern on the CCADB
> record for Scott S. Perry CPA, PLLC.
>
> I do think it's important for CAs to disclose these issues to their
> auditors, but I also expect auditors to discover them.
>
> - Wayne
>
> On Wed, Aug 15, 2018 at 8:21 AM Ben Wilson <ben.w...@digicert.com>
> wrote:
>
>> Re-sending
>>
>> -----Original Message-----
>> From: Ben Wilson
>> Sent: Wednesday, August 15, 2018 8:34 AM
>> To: 'ry...@sleevi.com' <ry...@sleevi.com>; Wayne Thayer <
>> wth...@mozilla.com>
>> Cc: mozilla-dev-security-policy <
>> mozilla-dev-s...@lists.mozilla.org>
>> Subject: RE: Misissuance and BR Audit Statements
>>
>> Thanks, Ryan and Wayne,
>>
>> Going forward we'll work to improve our management letter disclosures to
>> include reported mis-issuances during the audit period.
>>
>> Sincerely yours,
>>
>> Ben
>>
>> -----Original Message-----
>> From: dev-security-policy <dev-security-...@lists.mozilla.org>
>> On Behalf Of Ryan Sleevi via dev-security-policy
>> Sent: Monday, August 13, 2018 3:57 PM
>> To: Wayne Thayer <wth...@mozilla.com>
>> Cc: mozilla-dev-security-policy <
>> mozilla-dev-s...@lists.mozilla.org>
>> Subject: Re: Misissuance and BR Audit Statements
>>
>> > https://lists.mozilla.org/listinfo/dev-security-policy
>> >
>> _______________________________________________
>> dev-security-policy mailing list
>> dev-secur...@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-security-policy
>>
>
0 new messages