Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Symantec Response T

349 views
Skip to first unread message

Steve Medin

unread,
Apr 10, 2017, 11:00:14 AM4/10/17
to MozPol
Issue T: RA Program Misissuances (January 2010 - January 2017)

Program Background:

Symantec has operated an RA program designed to deliver a superior customer experience in global markets where language skills, understanding of local business requirements, and physical local presence are necessary. RA partners have supported various certificate types, including those for publicly trusted SSL/TLS.

The RA program for publicly trusted SSL/TLS authorized appropriately trained personnel at select RA partners to complete all steps for authentication, review, and certificate issuance.

In 2011, prior to ratification of the CA/Browser Forum Baseline Requirements, Symantec scaled back the scope of the RA program for publicly trusted SSL/TLS to support only those partners whose scale of business and investment in the future success of that business warranted the additional cost associated with supporting the then-new BRs. Since 2013, there have been only 4 RA partners still capable of processing and issuing publicly trusted SSL/TLS certificates: CrossCert, Certisign, Certsuperior, and Certisur.

Symantec has had multiple controls in place to ensure these RAs' compliance with the BRs:

Documentation:
1. Symantec operates an internal Knowledge Base ("KB") for its authentication staff and RA partners that contains detailed step-by-step procedures for performing each of the tasks required to validate the identity asserted in a certificate request.
2. The KB reinforces acceptable and unacceptable sources of validation information and processes using a subset of the information in the BRs.
3. The KB explains request flagging, flag reasons, and flag clearing procedures.

Training & Exams:
1. Topics include BR changes, CPS changes, process changes resulting from industry incidents regardless of the CA involved, and a review of Symantec's procedures that extend the Baseline Requirements.
2. Exams are modified and retaken annually as criteria to renew individual access certificates or after significant internal or external process changes.

Technology During Authentication:
1. Each request is screened for trade compliance, high-risk names, potential phishing (strings used in scam domains, high-profile brands), and other potentially risky content such as "test". Potential failures are flagged, preventing RA issuance, until and unless further review and analysis is completed.
2. Risk flags require manual override by authentication personnel - internal or RA personal as appropriate - for certificate issuance to proceed. Flag clearing privileges are only granted to personnel who are have completed the requisite training and passed appropriate exams.

Technology Pre- and Post- Issuance:
1. Each request is screened to ensure elements outside of the subject information are BR compliant (e.g. SAN fields are complete, proper validity limits are in place, 2048 bit or higher key lengths are used, etc.). This scan is done after Authentication personnel approve the request and before it is issued. These checks cannot be overridden.
2. Daily, we rescan all certificates issued on the prior day using these same checks.

Audit:
1. We requested independent WebTrust audit reports from RAs and assessed them for material findings pursuant to BR 8.4 regarding WebTrust audited Delegated Third Parties. See issue V addressing audits.

Customer-Facing Controls:
1. Symantec supports Certification Authority Authorization, putting control of authorized CAs in the hands of customers.
2. Symantec logs publicly trusted certificates to Certificate Transparency Logs and offers a CT monitor to provide visibility for all customers to enable detection of suspect certificates.

CrossCert Test Certificate Issue:

On January 19, 2017, Andrew Ayer, an independent researcher posted the results of an analysis of public Certificate Transparency logs through which he identified roughly 270 instances of suspicious certificates issued by multiple CAs, including Symantec, containing the words "test" or "example" in the subject information.

Symantec determined that 127 of these certificates were issued from Symantec operated CAs and that all 127 had been issued by the RA CrossCert. All but 31 had already expired or been revoked.

Immediate Response
Andrew Ayer's report was a certificate problem report under BR 4.9.5 which required us to begin an investigation within 24 hours, which we did. We determined that 127 certificates were in scope of the problem report.

1. On January 19, 2017, after becoming aware of this issue, Symantec disabled issuance privileges for all CrossCert staff.
2. On January 20, 2017, Symantec revoked the 31 still valid and active certificates. These certificates had been issued between December 28, 2016 and January 18, 2017.
3. Symantec promptly took over validation and issuance for all pending and new orders submitted through CrossCert. Since then, Symantec's authentication team has continued to fully process these orders independently, without relying on any previous authentication work completed by CrossCert.
4. Symantec disabled customer issuance from enterprise accounts originally authenticated by CrossCert. New issuance from these accounts was blocked until the accounts were re-validated by Symantec personnel.

Root Cause
Following discussions with CrossCert and review of their audit logs, Symantec confirmed that all 127 certificates were issued as part of either internal or customer-specific testing. In mid-2016, CrossCert created a customer-focused feature with good intent to help customers with testing, but they did so without following the proper guidelines.

CrossCert personnel had completed the Symantec-required training mandating that only fully validated certificates be issued. At no point was CrossCert authorized to perform less vetting on certificates for their own internal use or for customer testing. The personnel at CrossCert had also completed exams designed to verify comprehension of this training. Nonetheless, CrossCert management authorized overrides for certificates being used for internal and customer-specific testing, and CrossCert authentication personnel complied with that authorization.

Audit logs confirm that our controls to flag these certificates for compliance failures worked appropriately but CrossCert management overrode those flags. CrossCert did not consult with Symantec on the significance of the compliance failure flags or their decisions to override the flags for any of the certificates. Our flagging process is extensive and a normal part of every business day. For example, our process catches popular domains/trademarks and variants using fuzzy logic that might otherwise be issued by a CA that doesn't test or block trademark abuse. Our flag/hold rate is conservative - as a result, the false positive rate is high and, following manual analysis, most flags are cleared. We appreciate the suggestion that additional flag monitoring and trend analysis would provide greater insight and increase the opportunity to catch issues like this more quickly in the future.

Review of the 127 certificates showed that all issuances occurred in two time frames: Before June, 2012 or after June, 2016. Symantec received unqualified WebTrust for CAs and WebTrust for CAs Baseline and Network Security audits from Ernst & Young KR for the period July 1, 2015-June 30, 2016. Each of the 127 certificates issued for testing purposes were issued either years before these audits or after these most recent audits and so not detectable as part of these independent audits.

Subsequent Investigation
As part of investigating this incident, Symantec began a direct review of the authentication practices at CrossCert and their engagement with Ernst & Young KR. Through that investigation, Symantec identified two additional issues:

1. There were deficiencies in CrossCert's documentation of the validation step in the authentication process, failing to meet both Symantec and Baseline requirements.
2. The CrossCert 2015-2016 audit did not include in its scope all of the CAs from which CrossCert was authorized to issue certificates. E&Y KR stated that CrossCert provided a list of all their issuing CAs but reduced the list of issuing CAs in scope for sampling for budgetary reasons. E&Y KR did not call out this known limit in its report.

These additional findings caused Symantec to seriously question not only CrossCert, but also the adequacy of E&Y KR's WebTrust for CAs Baseline and Network Security audits. This episode also reinforced that we need to more closely monitor and review the scope, methodology, and practices of external auditors.

Remediation
1. Symantec fully terminated CrossCert as an RA partner. They no longer participate in the authentication process or issuance of publicly trusted SSL/TLS certificates.
2. Symantec has hired a team of Korean speaking authentication personnel who have completed the requisite training and now independently process all new orders from this market.
3. Symantec is in the process of fully and independently revalidating active SSL/TLS certificates previously approved by CrossCert, and if necessary, revoking and replacing certificates if we detect any errors. As we posted in response to the assertion about reusing RA validation in m.d.s.policy, we reiterate that we are not doing this. For CrossCert, we are validating each existing certificate as if it was a new request.
4. Symantec announced the decision to wind down the RA program for publicly trusted SSL/TLS.
5. Symantec is in the process of reviewing 100% of the authentication records for the active SSL/TLS certificates approved and issued by Certisign, Certsuperior, and Certisur. If we become aware of a certificate that is misleading, we follow the Baseline Requirements regarding revocation.

It has been suggested that the issues identified above and those associated with Issue V warrant immediate revocation of all certificates issued by each of these RAs. Symantec disagrees with that assessment. Revocation by a CA of certificates, outside of a customer request, can be materially disruptive to the individuals and the organizations that rely on them, and so should be reserved for cases where there are clear security risks.

In the case of CrossCert, Symantec immediately revoked the clearly problematic certificates mis-issued for testing. With respect to individual certificates, the key additional issue identified with CrossCert related to proper record-keeping. Accordingly, using a risk-informed approach, Symantec is fully re-validating these certificates.

In the cases of Certisign, Certsuperior, and Certisur, we relied on at least one unqualified audit for their operations. With respect to their individual certificates, there was no evidence of any problems with the certificates they approved and issued. Accordingly, and again using a risk-informed approach, Symantec is reviewing all of these certificates, which amounts to conducting an audit with 100% sampling.

Symantec will provide updates to the community regarding our revalidation of active CrossCert-issued SSL/TLS certificates and our review of the other RA partners.

Ryan Sleevi

unread,
Apr 11, 2017, 12:18:58 PM4/11/17
to Steve Medin, MozPol
Hi Steve,

A few followup questions.

1) On the basis of the controls Symantec described, at no point was any
mention made of Symantec performing sampling audits to ensure their RA
partners complied with either the RA partner's CP/CPS or Symantec's CP/CPS.
a) Is it fair to conclude that no such examination if done?
b) If no such examination is done, why not?
c) Does Symantec plan to perform any sampling audits its other RA
partners?

2) Symantec states "We requested independent WebTrust audit reports from
RAs and assessed them for material findings pursuant to BR 8.4 regarding
WebTrust audited Delegated Third Parties."
a) For what years did you request an audits? That is, has Symantec
requested such audits since 2011? If not, when was the first year Symantec
requested such audits?
b) Please describe the process and procedures Symantec employees perform
in order to determine the appropriate status of the auditor.
c) Please describe the process and procedures Symantec employees perform
in order to determine that the RA partner's CP/CPS conforms with Symantec's
CP/CPS
d) Please describe what procedures were in place when an RA partner
changed its CP/CPS
e) Please describe what procedures were in place when Symantec changed
its CP/CPS.

3) Symantec states "Symantec supports Certification Authority
Authorization, putting control of authorized CAs in the hands of customers."
a) Why did you mention this, given that Symantec has acknowledged that
its RA partners performed domain validation independent of Symantec.
b) Do you agree that this statement is materially misleading in the
context in which it is provided, namely the discussion of RA security
controls, and may otherwise be inappropriately interpreted as Symantec
suggesting its RA partners were performing an activity that they may not
have been done?

4) Symantec states "Topics include BR changes, CPS changes, process changes
resulting from industry incidents regardless of the CA involved, and a
review of Symantec's procedures that extend the Baseline Requirements."
a) Did Symantec update its RA partners regarding the set of Symantec
issues related to the Test Certificate Misissuance (Issue D)?
b) Was this the same process used to ensure employee compliance, which
was identified deficient with respect to Issue D?
c) What steps did Symantec take, in response to Issue D, to inform its RA
partners about these issues?
d) What steps did Symantec take, in response to Issue D, to review its RA
partners activities, similar to that of its employees activities?
e) What steps did Symantec take, in response to Issue D, to review its
APIs granted to RA partners, similar to the test tool used to cause
misissuance?
f) What other evidence or facts are you able to share that could
establish Symantec as a competent, reliable, or trustworthy CA partner,
given that the current evidence shared meaningfully suggests it is not any
of these.

5) Symantec states "Symantec is in the process of fully and independently
revalidating active SSL/TLS certificates previously approved by CrossCert,
and if necessary, revoking and replacing certificates if we detect any
errors."
a) Does Symantec believe this process is compliant with the Baseline
Requirements and Symantec's CP/CPS?
b) If Symantec believes this process is consistent with the Baseline
Requirements, please specifically reference the sections that permit this.
c) If not, is Symantec requesting an exception to their obligations under
the Baseline Requirements to promptly revoke these certificates within 24
hours?
d) Has Symantec clearly stated that request as such - a request to
violate the Baseline Requirements - rather than as a statement of intent?

6) Symantec states "Symantec is in the process of reviewing 100% of the
authentication records for the active SSL/TLS certificates approved and
issued by Certisign, Certsuperior, and Certisur."
a) Does Symantec believe this process is compliant with the Baseline
Requirements and Symantec's CP/CPS?
b) If Symantec believes this process is consistent with the Baseline
Requirements, please specifically reference the sections that permit this.
c) If not, is Symantec requesting an exception to their obligations under
the Baseline Requirements to promptly revoke these certificates within 24
hours?
d) Has Symantec clearly stated that request as such - a request to
violate the Baseline Requirements - rather than as a statement of intent?

7) Symantec states "It has been suggested that the issues identified above
and those associated with Issue V warrant immediate revocation of all
certificates issued by each of these RAs. Symantec disagrees with that
assessment. "
a) Please provide the reference to the Baseline Requirements to support
this conclusion.

8) Symantec states that it believes revocation "should be reserved for
cases where there are clear security risks."
a) Please explain how this belief is consistent with the Baseline
Requirements, Section 9.6.3, Item 8.
b) Please explain how this belief is consistent with the Baseline
Requirements, Section 4.9.1.1, Item 5.
c) Please explain how this belief is consistent with the Baseline
Requirements, Section 4.9.1.1, Item 9.
d) Is it fair to conclude that Symantec's belief is that it does not have
to follow the Baseline Requirements that it disagrees with?
e) Please provide any supporting evidence related to this belief that
would be useful for Relying Parties to assess whether Symantec is competent
and trustworthy, as the current evidence suggests it is not.

Gervase Markham

unread,
Apr 11, 2017, 12:43:06 PM4/11/17
to mozilla-dev-s...@lists.mozilla.org
On 11/04/17 17:18, Ryan Sleevi wrote:
> 1) On the basis of the controls Symantec described, at no point was any
> mention made of Symantec performing sampling audits to ensure their RA
> partners complied with either the RA partner's CP/CPS or Symantec's CP/CPS.
> a) Is it fair to conclude that no such examination if done?

In various rounds of questioning at the time we were focussing purely on
this incident, I asked Symantec what processes they had in place for
checking that the RAs were doing what they should. Their answer was
"WebTrust audits". So I believe they have already said that no such
examination was done. I'm sure they'd be happy to clarify, though.

Most of your other questions are fairly stated (if perhaps rather broad
in scope - are you expecting a total dump of all their internal
procedure documents?). But particularly:

> d) Is it fair to conclude that Symantec's belief is that it does not have
> to follow the Baseline Requirements that it disagrees with?

Gerv

Ryan Sleevi

unread,
Apr 11, 2017, 1:00:10 PM4/11/17
to Gervase Markham, mozilla-dev-security-policy
On Tue, Apr 11, 2017 at 12:42 PM, Gervase Markham via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> In various rounds of questioning at the time we were focussing purely on
> this incident, I asked Symantec what processes they had in place for
> checking that the RAs were doing what they should. Their answer was
> "WebTrust audits". So I believe they have already said that no such
> examination was done. I'm sure they'd be happy to clarify, though.
>

In attempting to make an objective evaluation of the trustworthiness of
Symantec, in either its past operations or as a future predictor, we
essentially need to understand

1) That Symantec understood the gravity of the situation
2) That Symantec took it seriously and responded appropriately relative to
the trust it was granted
3) That Symantec remains committed to doing so in the future, and with
specific plans to identify and remedy the issues

On the basis of the information provided, I see no reason to believe the
answer to #1 is that they did not (or that they "disagreed"), the answer to
#2 is that "They did not", and the answer to #3 is "They are not, and have
no specific plans".

Symantec is asserting its processes were trustworthy, but the evidence
provided wholly contradicts that conclusion (in my opinion). I'm looking to
develop a meaningful understanding of what Symantec did, so that it can
demonstrate that what it did was reasonable and expected, or to acknowledge
there were deficiencies that have a remediation plan. The current statement
appears to be that the processes were appropriate and no deficiencies -
despite the Baseline Requirements clearly contradicting this - and thus it
seems appropriate to suggest that Symantec should not be trusted and/or
have its trust meaningfully reduced to negate the impact that these
deficient practices have on the ecosystem.

The burden is two-fold:
1) Are the facts correct? It does not appear Symantec has disputed these,
except with respect to the RA partner audits (for which it provided
evidence that supports the current conclusions and refutes their
disagreement)?
2) Are there plans for the future, or an approach to the past, that is
meaningful to consider when evaluating the trustworthiness. At present,
Symantec's not shared such, beyond the RA remediation plans, which are at
conflict with the Baseline Requirements, its CP/CPS, and its Subscriber
Agreements.
0 new messages