Intent to Deprecate and Remove: Trust in existing Symantec-issued Certificates

126308 views
Skip to first unread message

Ryan Sleevi

unread,
Mar 23, 2017, 12:03:20 PM3/23/17
to blink-dev, awha...@chromium.org

Note: Historically, the Google Chrome team has not used the Blink Process for Certificate Authority-related security issues, of which there have been a number over the years. However, we are interested in exploring using this process for such changes, as it provides a greater degree of transparency and public participation. Based on the level of participation and feedback we receive, we may consider using this for the future. However, as CA-related security incidents may require immediate response to protect users, this should not be seen as a guarantee that this process can be used in future incident responses.


Primary eng (and PM) emails:

rsl...@chromium.org awha...@chromium.org


Summary

Since January 19, the Google Chrome team has been investigating a series of failures by Symantec Corporation to properly validate certificates. Over the course of this investigation, the explanations provided by Symantec have revealed a continually increasing scope of misissuance with each set of questions from members of the Google Chrome team; an initial set of reportedly 127 certificates has expanded to include at least 30,000 certificates, issued over a period spanning several years. This is also coupled with a series of failures following the previous set of misissued certificates from Symantec, causing us to no longer have confidence in the certificate issuance policies and practices of Symantec over the past several years. To restore confidence and security of our users, we propose the following steps:

  • A reduction in the accepted validity period of newly issued Symantec-issued certificates to nine months or less, in order to minimize any impact to Google Chrome users from any further misissuances that may arise.

  • An incremental distrust, spanning a series of Google Chrome releases, of all currently-trusted Symantec-issued certificates, requiring they be revalidated and replaced.

  • Removal of recognition of the Extended Validation status of Symantec issued certificates, until such a time as the community can be assured in the policies and practices of Symantec, but no sooner than one year.


Motivation

As captured in Chrome’s Root Certificate Policy, root certificate authorities are expected to perform a number of critical functions commensurate with the trust granted to them. This includes properly ensuring that domain control validation is performed for server certificates, to audit logs frequently for evidence of unauthorized issuance, and to protect their infrastructure in order to minimize the ability for the issuance of fraudulent certs.


On the basis of the details publicly provided by Symantec, we do not believe that they have properly upheld these principles, and as such, have created significant risk for Google Chrome users. Symantec allowed at least four parties access to their infrastructure in a way to cause certificate issuance, did not sufficiently oversee these capabilities as required and expected, and when presented with evidence of these organizations’ failure to abide to the appropriate standard of care, failed to disclose such information in a timely manner or to identify the significance of the issues reported to them.


These issues, and the corresponding failure of appropriate oversight, spanned a period of several years, and were trivially identifiable from the information publicly available or that Symantec shared.


The full disclosure of these issues has taken more than a month. Symantec has failed to provide timely updates to the community regarding these issues. Despite having knowledge of these issues, Symantec has repeatedly failed to proactively disclose them.  Further, even after issues have become public, Symantec failed to provide the information that the community required to  assess the significance of these issues until they had been specifically questioned. The proposed remediation steps offered by Symantec have involved relying on known-problematic information or using practices insufficient to provide the level of assurance required under the Baseline Requirements and expected by the Chrome Root CA Policy.


In January 2015, Symantec-issued certificates represented more than 30% of the valid certificates by volume. While changes in the CA ecosystem have seen that share decrease over the past two years, there is still a significant compatibility risk for an immediate and complete distrust. Further, due to overall TLS ecosystem concerns, we understand that it may take non-trivial effort for some site operators to find suitable solutions, as the need to support older devices may necessitate the use of particular CAs, meaning that distrust of new certificates also has significant compatibility risk.


To balance the compatibility risks versus the security risks, we propose a gradual distrust of all existing Symantec-issued certificates, requiring that they be replaced over time with new, fully revalidated certificates, compliant with the current Baseline Requirements. This will be accomplished by gradually decreasing the ‘maximum age’ of Symantec-issued certificates over a series of releases, distrusting certificates whose validity period (the difference of notBefore to notAfter) exceeds the specified maximum.


The proposed schedule is as follows:

Chrome 59 (Dev, Beta, Stable): 33 months validity (1023 days)

Chrome 60 (Dev, Beta, Stable): 27 months validity (837 days)

Chrome 61 (Dev, Beta, Stable): 21 months validity (651 days)

Chrome 62 (Dev, Beta, Stable): 15 months validity (465 days)

Chrome 63 (Dev, Beta): 9 months validity (279 days)

Chrome 63 (Stable): 15 months validity (465 days)

Chrome 64 (Dev, Beta, Stable): 9 months validity (279 days)


The proposed schedule attempts to avoid making changes in Chrome 63 Stable, as that would be released during the winter holiday production freeze many organizations undergo. This is solely to reduce disruption for site operators and users, and attempts to resume with Chrome 64 following the holiday season. Further, the practical impact of the changes in Chrome 59 and 60 are relatively minimal, due to many of the certificates issued during that period of time being issued using SHA-1, which is no longer supported for certificates in Chrome.


In addition, we propose to require that all newly-issued certificates must have validity periods of no greater than 9 months (279 days) in order to be trusted in Google Chrome, effective Chrome 61. This ensures that the risk of any further misissuance is, at most, limited to nine months, and more importantly, that if any further action is warranted or necessary, that the entire ecosystem can migrate within that time period, thus minimizing the risk of further compatibility issues.


By combining these two steps, we can ensure that the level of assurance in Symantec-issued certificates is able to match what is expected by Google Chrome and the ecosystem, and that the risks posed both from past and possible future misissuance is minimized as much as possible.


Given the nature of these issues, and the multiple failures of Symantec to ensure that the level of assurance provided by their certificates meets the requirements of the Baseline Requirements or Extended Validation Guidelines, we no longer have the confidence necessary in order to grant Symantec-issued certificates the “Extended Validation” status. As documented with both the current and past misissuance, Symantec failed to ensure that the organizational attributes, displayed within the address bar for such certificates, meet the level of quality and validation required for such display. Therefore, we propose to remove such indicators, effective immediately, until Symantec is able to demonstrate the level of sustained compliance necessary to grant such trust, which will be a period no less than a year. After such time has passed, we will consider requests from Symantec to re-evaluate this position, in collaboration with the broader Chromium community.


Compatibility and Interoperability Risk

As with any reduction in trust in a Certificate Authority, this poses a non-trivial degree of compatibility risk. This is because site operators desire to have their certificates recognized in all client browsers, and if one or more browsers fail to trust a given CA, this is prevented from happening.


On the other hand, all site operators expect that certificates will only be issued for their domains upon their request, and the failure to have that assurance significantly undermines the security of HTTPS for both site operators and users.


This compatibility risk is especially high for Symantec-issued certificates, due to their acquisition of some of the first CAs, such as Thawte, Verisign, and Equifax, which are some of the most widely supported CAs. Distrusting such CAs creates further difficulty for providing secure connections to both old and new devices alike, due to the need to ensure the CA a site operator uses is recognized across these devices.


Further, the immediate distrust of a CA, as has been necessary in the past, can significantly impact both site operators and users. Site operators are forced to acquire certificates from other CAs, without having the opportunity and time to research which CAs best meet their needs, and users encounter a substantial number of errors until those site operators act, conditioning them to ignore security warnings. In the event that only a single browser distrusts such a CA, the error is often seen as the browser’s fault, despite it being a failure of the CA to provide the necessary level of assurance, and the site operator to respond in a timely fashion.


Assessing the compatibility risk with both Edge and Safari is difficult, because neither Microsoft nor Apple communicate publicly about their changes in trust prior to enacting them.


While Mozilla conducts their discussions regarding Certificate Authorities in public, and were the first to be alerted of these latest issues, they have not yet begun discussion of the next steps to how best to protect their users. Our hope is that this proposal may be seen as one that appropriately balances the security and compatibility risks with the needs of site operators, browsers, and users, and we welcome all feedback.


Alternative implementation suggestion for web developers

This proposal allows for web developers to continue to use Symantec issued certificates, but will see their validity period reduced. This ensure that web developers are aware of the risk and potential of future distrust of Symantec-issued certificates, should additional misissuance events occur, while also allowing them the flexibility to continue using such certificates should it be necessary.


Usage information from UseCounter:

For a variety of non-technical reasons, we do not currently instrument the usage of CAs. Further, few public metrics exist for intersecting usage information with the validity period, since only certificates valid greater than nine months will be affected outside of their normal replacement cycle. From Mozilla Firefox’s Telemetry, we know that Symantec issued certificates are responsible for 42% of certificate validations. However, this number is not strictly an indicator for impact, as this number is biased towards counting certificates for heavily-trafficked sites, and whose issuance is fully automated and/or whose validity periods will be unaffected, thus significantly overstating impact. By phasing such changes in over a series of releases, we aim to minimize the impact any given release poses, while still continually making progress towards restoring the necessary level of security to ensure Symantec issued certificates are as trustworthy as certificates from other CAs.

Vincent Lynch

unread,
Mar 23, 2017, 12:35:16 PM3/23/17
to blink-dev
"Therefore, we propose to remove [EV] indicators, effective immediately,"


Two questions about this:

1. Can you clarify what "effective immediately" means? Will this occur with the next stable release of Chrome? Or will an update be pushed to the current stable version?

2. Can you confirm that the removal of EV indicators will affect *all* Symantec-issued EV certificates, regardless of their compliance with Chrome's reduced validity schedule?

Ryan

unread,
Mar 23, 2017, 12:48:32 PM3/23/17
to blink-dev


On Thursday, March 23, 2017 at 12:35:16 PM UTC-4, Vincent Lynch wrote:
"Therefore, we propose to remove [EV] indicators, effective immediately,"


Two questions about this:

1. Can you clarify what "effective immediately" means? Will this occur with the next stable release of Chrome? Or will an update be pushed to the current stable version?

As this is a security related incident, the proposal is to land this in Trunk and update Dev and Beta to reflect this, treating this as "Medium" severity per https://www.chromium.org/developers/severity-guidelines by treating this as an address bar spoofing issue.
 
2. Can you confirm that the removal of EV indicators will affect *all* Symantec-issued EV certificates, regardless of their compliance with Chrome's reduced validity schedule?

Correct. Extended Validation certificates are recognition of the processes and controls related to validating the information in the certificates is accurate and correct. The issues highlighted have undermined that confidence. The validity issue is separate and mitigates future issues that may arise or be discovered.

kevin...@ankuraconsulting.com

unread,
Mar 23, 2017, 1:03:16 PM3/23/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org
Can you please clarify if any related CAs are affected, such as GeoTrust and Thawte, are affected?


On Thursday, March 23, 2017 at 12:03:20 PM UTC-4, Ryan Sleevi wrote:

Ryan Sleevi

unread,
Mar 23, 2017, 1:06:41 PM3/23/17
to kevin...@ankuraconsulting.com, blink-dev, awha...@chromium.org, Ryan Sleevi
On Thu, Mar 23, 2017 at 1:03 PM, <kevin...@ankuraconsulting.com> wrote:
Can you please clarify if any related CAs are affected, such as GeoTrust and Thawte, are affected?

All Symantec issued certificates. GeoTrust and Thawte are CAs operated by Symantec, simply afforded different branding.

While this list may need to be updated for some recently created roots, https://chromium.googlesource.com/chromium/src/+/master/net/data/ssl/symantec/README.md may accurately capture the state of impact

pzb...@gmail.com

unread,
Mar 23, 2017, 1:30:36 PM3/23/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org
On Thursday, March 23, 2017 at 9:03:20 AM UTC-7, Ryan Sleevi wrote:

To balance the compatibility risks versus the security risks, we propose a gradual distrust of all existing Symantec-issued certificates, requiring that they be replaced over time with new, fully revalidated certificates, compliant with the current Baseline Requirements. This will be accomplished by gradually decreasing the ‘maximum age’ of Symantec-issued certificates over a series of releases, distrusting certificates whose validity period (the difference of notBefore to notAfter) exceeds the specified maximum.


The proposed schedule is as follows:

Chrome 59 (Dev, Beta, Stable): 33 months validity (1023 days)

Chrome 60 (Dev, Beta, Stable): 27 months validity (837 days)

Chrome 61 (Dev, Beta, Stable): 21 months validity (651 days)

Chrome 62 (Dev, Beta, Stable): 15 months validity (465 days)

Chrome 63 (Dev, Beta): 9 months validity (279 days)

Chrome 63 (Stable): 15 months validity (465 days)

Chrome 64 (Dev, Beta, Stable): 9 months validity (279 days)


The proposed schedule attempts to avoid making changes in Chrome 63 Stable, as that would be released during the winter holiday production freeze many organizations undergo. This is solely to reduce disruption for site operators and users, and attempts to resume with Chrome 64 following the holiday season. Further, the practical impact of the changes in Chrome 59 and 60 are relatively minimal, due to many of the certificates issued during that period of time being issued using SHA-1, which is no longer supported for certificates in Chrome


I'm trying to convert this to concrete terms.

If a sever authentication certificate has a notBefore date of 2017-03-24T00:00:00Z and a notAfter date of 2018-03-23T23:59:59Z (approximately 365 days), and this intent is implemented as described, is it accurate that this certificate will be accepted by Chrome Beta until Chrome 63 and by Chrome Stable 64?
 

In addition, we propose to require that all newly-issued certificates must have validity periods of no greater than 9 months (279 days) in order to be trusted in Google Chrome, effective Chrome 61.


How is "newly-issued" defined?
 

This ensures that the risk of any further misissuance is, at most, limited to nine months, and more importantly, that if any further action is warranted or necessary, that the entire ecosystem can migrate within that time period, thus minimizing the risk of further compatibility issues.


By combining these two steps, we can ensure that the level of assurance in Symantec-issued certificates is able to match what is expected by Google Chrome and the ecosystem, and that the risks posed both from past and possible future misissuance is minimized as much as possible.


 Thanks,
Peter

Andrew Ayer

unread,
Mar 23, 2017, 2:02:14 PM3/23/17
to blink-dev
Hi Ryan,

What restrictions, if any, will be placed on the reuse of validation
information?

Regards,
Andrew

ben.m...@gmail.com

unread,
Mar 23, 2017, 2:07:03 PM3/23/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org
(just to be super clear I'm writing this in a personal capacity and not representing the thoughts of my employer)

As a developer I feel like this plan might be excessively confusing. Given this list of dates, how would I calculate when I have to do something? Do I even remember when my cert was issued? Also, this process seems to aim at a gradual breakage, but from a developer's perspective this is not gradual -- I own a single site, and during one of the chrome releases my site will break the day that release goes to stable.

In addition the lack of a single deadline would seem to make it hard for a campaign to get site owners to replace these certs to be effective. If I were in the shoes of Symantec, I'd start emailing my customers offering them a free renewal of their certificate. If I were another CA I'd start offering Symantec customers a free replacement for their old certificate. If customers are affected over the course of a year it's hard for people to make these kinds of campaigns high profile.

Did you guys consider something where there would be an escalating level of distrust (eg, in one version you get a warning rather than a green lock, the next release a red not secure, then a intersticial)? This would allow attentive operators who might not hear about this change otherwise to figure out that there is an issue before their site breaks completely. It also gives all parties involved a small number of times where they can promote this change.

If other browser vendors might implement a similar policy it could make sense to tie those events to dates rather than releases so that this type of promotion could be more effective.



On Thursday, March 23, 2017 at 9:03:20 AM UTC-7, Ryan Sleevi wrote:

kane...@gmail.com

unread,
Mar 23, 2017, 2:28:26 PM3/23/17
to blink-dev
> Did you guys consider something where there would be an escalating level of distrust (eg, in one version you get a warning rather than a green lock, the next release a red not secure, then a interstitial)? This would allow attentive operators who might not hear about this change otherwise to figure out that there is an issue before their site breaks completely. It also gives all parties involved a small number of times where they can promote this change.

As a developer, this sounds reasonable to me.

Wording & iconography such as this might be reusable for future incidents:

#######
(i) Warning | https://example.com

Your connection to this site is not fully secure

The site presented old identity information which needs to be revalidated. Learn more
#######

- Learn more would link to a help center article saying you need to get your certificate reissued
- Not overly alarmist - use of Info icon, "Warning" instead of "Not Secure"
- Text does not imply operator is at fault, but they do need to take action
- Notice could be upgraded to red in a later release, then to interstitial
- Needs new translation work - "Warning" is not an already used term in the address bar


Kane York

Chris Palmer

unread,
Mar 23, 2017, 2:33:27 PM3/23/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org, ben.m...@gmail.com
On Thursday, March 23, 2017 at 11:07:03 AM UTC-7, ben.m...@gmail.com wrote:

As a developer I feel like this plan might be excessively confusing. Given this list of dates, how would I calculate when I have to do something?

This is a good and important question. A big part of the solution is automating re-issuance. Automation is a good value-add that various CAs and resellers are offering now, because lots of people need it for lots of reasons. I won't endorse any particular vendor, but there are several vendors you can choose from that are quite affordable and who provide automated re-issuance.

Combined with the gradual move to certificates with shorter lifespans anyway (as a way of coping with problems like this, and with the difficulty of certificate revocation generally), automation is a necessity going forward.

PhistucK

unread,
Mar 23, 2017, 2:36:28 PM3/23/17
to Ben Maurer, blink-dev, awha...@chromium.org, Ryan Sleevi
Perhaps this just needs a similar treatment to the one SHA-1 got/mixed content get (simply showing a website as secure, but not emphasizing that it is insecure). No interstitial on one hand, but also not shown as secure on the other hand.
I think this is the best outcome, as Symantec has (repeatedly) demonstrated that it cannot be trusted and not showing a website as secure indicates that it cannot be completely trusted (which is the case).

Also, some warning in the console might help (with a cutoff date, obviously), in anticipation for an interstitial or some other drastic action.


​Also, note that the iconography and taxonomy was simplified. There are three states, if I remember correctly (four, if you count Extended Validation​, but ignore that for now) - Secure (great HTTPS), Not Secure (for form filling HTTP websites, data URLs and so on) and none (HTTP without forms and HTTPS with problems). A warning signal is probably a step back and does not mean a lot to user.


PhistucK

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

lloth...@gmail.com

unread,
Mar 23, 2017, 7:30:44 PM3/23/17
to blink-dev
Could you put the dates those versions hit stable in the original post?

evan...@gmail.com

unread,
Mar 23, 2017, 9:12:35 PM3/23/17
to blink-dev, lloth...@gmail.com
See: https://www.chromium.org/developers/calendar

2017

 Release   Estimated Week of Stable
56 Jan 31st, 2017
57 Mar 14th, 2017
58 Apr 25th, 2017
59 Jun 6th, 2017
60 Aug 1st, 2017
61 Sep 12th, 2017
62 Oct 24th, 2017
63 Dec 12th, 2017

tpg...@gmail.com

unread,
Mar 23, 2017, 9:52:14 PM3/23/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org
I would like more details on the issues. "...allowed least four parties access to their infrastructure in a way to cause certificate issuance..." is pretty vague. Did these 4 parties then issue 30,000 certificates? What's wrong with these certificates exactly?

The issues previously known are 1) "test" certificates issued without domain owner's consent, though Symantec claimed that they were only used internally; 2) certificates issued for unregistered domains.

Google Internet Authority G2 CA that issues many of Google's TLS certificates is signed by GeoTrust Global CA, which presumably is owned by Symantec now. So do Google's own certificates fall into the "gradually untrustworthy" category?

jus...@gerhardt.link

unread,
Mar 23, 2017, 10:53:59 PM3/23/17
to blink-dev
Google semi-recently purchased a trusted root ca certificate.

matt.so...@gmail.com

unread,
Mar 23, 2017, 11:11:00 PM3/23/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org
When will we know if these proposals are accepted and will become reality?

Rik Cabanier

unread,
Mar 23, 2017, 11:19:30 PM3/23/17
to matt.so...@gmail.com, blink-dev, awha...@chromium.org, rsl...@chromium.org
On Thu, Mar 23, 2017 at 8:10 PM, <matt.so...@gmail.com> wrote:
When will we know if these proposals are accepted and will become reality?

Usually, once 3 blink owners send a lgtm (=looks good to me) to this email thread, the intent is accepted and the engineer's changes will be accepted into the chromium code base.
 

--

Ryan Sleevi

unread,
Mar 23, 2017, 11:52:00 PM3/23/17
to Peter Bowen, blink-dev, awha...@chromium.org, Ryan Sleevi
On Thu, Mar 23, 2017 at 1:30 PM, <pzb...@gmail.com> wrote:
I'm trying to convert this to concrete terms.

If a sever authentication certificate has a notBefore date of 2017-03-24T00:00:00Z and a notAfter date of 2018-03-23T23:59:59Z (approximately 365 days), and this intent is implemented as described, is it accurate that this certificate will be accepted by Chrome Beta until Chrome 63 and by Chrome Stable 64?

I believe you meant "accepted by Chrome Beta (until Chrome 63) and by Chrome Stable (until Chrome 64)", correct? If so, that is correct.

Alternatively stated, using the end-goal as the measuring stick, if Chrome 64 were to come out on January 30, 2018 (more or less 6 weeks after December 12, modulo holidays), then the earliest accepted certificate would be one whose notBefore was on or after April 26, 2017 (279 days before January 30, 2018). If Chrome 64 were used on January 31, 2018, then the earliest accepted certificate would be one whose notBefore was issued on or after April 27, 2017. Both of these would only work if their validity periods were less than or equal to 279 days.
 

In addition, we propose to require that all newly-issued certificates must have validity periods of no greater than 9 months (279 days) in order to be trusted in Google Chrome, effective Chrome 61.


How is "newly-issued" defined?

Chrome 60 is estimated to be branched the week of May 25, 2017, as detailed at https://www.chromium.org/developers/calendar . On or after that branch point, in which 'trunk' becomes the intended and eventual Chrome 61 development tree, a change would be landed to require all certificates issued on or after that date have validity periods less than or equal to nine months. For example, a certificate issued on May 26, 2017 would need to expire on or before March 1, 2018. This would work through the normal development cycle of Canary -> Dev -> Beta and be released to Chrome Stable during the week of September 12, 2017. Note these represent estimated dates, as mentioned on the page.

The proposed definition for this is to rely upon the notBefore as an appropriate signal. This does create risk, for example, if a certificate issued on May 26, 2017 has its "notBefore" set to a date prior to the actual issuance date. An alternative definition that may mitigate that risk, but provide site operators less flexibility, is to use the earliest accepted Signed Certificate Timestamp presented with the certificate (either within the certificate, via a stapled OCSP response, or with the TLS extension), and rely on the Trusted Certificate Transparency Logs to provide an accurate timestamp. This provides less flexibility for other Chromium embedders (in that it relies on Certificate Transparency, for which may not be an appropriate dependency for these embedders), and less flexibility for site operators (in that it measures "first disclosed", not necessarily issuance date), but can provide a stronger security guarantee by providing a counter-signature as to the earliest possible time the certificate was 'publicly' known.



As a proposal, I'm especially looking for feedback as to any potential interoperability and compatibility risks, and welcome suggestions on how best to minimize. If the Blink OWNERS would find it helpful and useful, and an appropriate use of WICG, this could be iterated further in a WICG Repository to provide a more formal explainer and pseudo-code to reduce that risk. As I mentioned, this is a particularly new and unique approach for something that has historically been announced post-facto of implementation, and feedback is welcome on how best to minimize risks.

Ryan Sleevi

unread,
Mar 23, 2017, 11:58:24 PM3/23/17
to Andrew Ayer, blink-dev


On Thu, Mar 23, 2017 at 2:01 PM, Andrew Ayer <ag...@andrewayer.name> wrote:
What restrictions, if any, will be placed on the reuse of validation
information?

For better or worse, the reuse of validation information is not web-visible, and as a consequence, is effectively not a quantifiable part of the Web Platform, beyond the audit engagement of a Certificate Authority .

However, this also highlights a gap in incompletely addressing the security concerns. For example, if the information reused was obtained by not following the proper and expected procedures - such as by one of the Delegated Third Parties who were improperly responsible for validating the information that appeared within these 30,000 certificates - then the issuance of a new certificate would fail to meaningfully ensure the information has been appropriately validated.

A way to mitigate these concerns - and ensure the proper level of security - would be to require that any information that appears within certificates issued after a particular date ONLY rely on information which Symantec has independently validated, consistent with the Baseline Requirements.

Ryan Sleevi

unread,
Mar 24, 2017, 12:09:57 AM3/24/17
to Ben Maurer, blink-dev, awha...@chromium.org, Ryan Sleevi
On Thu, Mar 23, 2017 at 2:07 PM, <ben.m...@gmail.com> wrote:
(just to be super clear I'm writing this in a personal capacity and not representing the thoughts of my employer)

As a developer I feel like this plan might be excessively confusing. Given this list of dates, how would I calculate when I have to do something? Do I even remember when my cert was issued? Also, this process seems to aim at a gradual breakage, but from a developer's perspective this is not gradual -- I own a single site, and during one of the chrome releases my site will break the day that release goes to stable.

This is the unfortunate reality of the CA ecosystem, in which the trust afforded to any given site is binary, but the trust afforded to a CA is expressed in gradations, dependent upon how the certificate was validated and when it was validated.

The simple answer would be that, if adopted by Blink OWNERS, then to minimize all parties should strive to replace their certificates with versions whose validity period is less than or equal to 279 days (9 months). By replacing the existing certificates with such a certificate, there should be no negative effect. The remaining dates relate to a ratcheting enforcement that attempts to minimize the breakage from any given release.

Alternatively, should it be that measuring by validity period represents too great an ecosystem concern, an alternative approach would be to express this requirement as "The certificate was issued on or after this date" - such as Chrome 59 rejecting certificates issued prior to 1023 days than the current date. This approach would replace certificates in batches based on issuance. Accompanied with a requirement that certificates issued on or after the proposed Chrome 61 'open' date (the day on or after the Chrome 60 "branch" date of May 25, 2017) have validity periods less than or equal to 9 months, this would also see the eventual harmonization of nine month validity, but affect a different population of certificates.

Would such an approach be easier to understand and communicate?
 
Did you guys consider something where there would be an escalating level of distrust (eg, in one version you get a warning rather than a green lock, the next release a red not secure, then a intersticial)? This would allow attentive operators who might not hear about this change otherwise to figure out that there is an issue before their site breaks completely. It also gives all parties involved a small number of times where they can promote this change.

Yes, this was considered. This is similar to the approach used with SHA-1 deprecation, but in this case, may cause undesired security risk, in that such certificates continue to be used, and thus private information (such as cookies) are still transmitted, and privileged features (such as geolocation) may still be enabled. This proposal was an attempt to balance those security concerns with the interoperability and compatibility risk.

Ryan Sleevi

unread,
Mar 24, 2017, 12:23:36 AM3/24/17
to tpg...@gmail.com, blink-dev, awha...@chromium.org, Ryan Sleevi
On Thu, Mar 23, 2017 at 9:52 PM, <tpg...@gmail.com> wrote:
I would like more details on the issues. "...allowed least four parties access to their infrastructure in a way to cause certificate issuance..." is pretty vague. Did these 4 parties then issue 30,000 certificates? What's wrong with these certificates exactly?

The issues previously known are 1) "test" certificates issued without domain owner's consent, though Symantec claimed that they were only used internally; 2) certificates issued for unregistered domains.

Apologies for not including the fullness of details in the original report, and this may hopefully address any confusion as to "why".

The initial problem report was reported publicly, through Mozilla's dev.security.policy mailing list, at https://groups.google.com/d/msg/mozilla.dev.security.policy/fyJ3EK2YOP8/yvjS5leYCAAJ

In the course of understanding these issues, representatives of Mozilla and Google both addressed follow-up questions to Symantec, as did broader members of the community and peers of the Mozilla Root CA Certificate module.

Symantec's replies are (generally) available at https://bugzilla.mozilla.org/show_bug.cgi?id=1334377 and share further details.

These entities are CrossCert (Korea Electronic Certificate Authority), Certisign Certificatadora Digital, Certsuperior S. de R. L. de C.V., and Certisur S.A.. Each of these entities were authorized by Symantec to perform validation services for information within the certificate, including organizational information and domain names. This process is permitted by the Baseline Requirements, but requires both that the CA accept liability for any issues that emerge through such a relationship, and that the CA ensure these entities are appropriately audited to the equivalent criteria for the validation roles that they perform, so that all certificates issued meet a consistent level of quality.

As demonstrated through the information provided, these four entities did not follow the appropriate practices or did not possess the appropriate and necessary audits from the appropriate parties. Symantec has acknowledged they were actively aware of this for at least one party, failed to disclose this to root programs, and did not sever the relationship with this party.

In effect, each of these parties were able to effect issuance by validating information improperly. At least 30,000 certificates were issued by these parties, with no independent way to assess the compliance of these parties to the expected standards. Further, these certificates cannot be technically identified or distinguished from certificates where Symantec performed the validation role. As a consequence, the insufficient demonstration of compliance, along with the inability to distinguish such certificates, combined with the incomplete identification of the scope of the issues, create a degree of uncertainty related to the entire corpus of certificates, for which the only meaningful way to restore that confidence is to propose a gradual distrusting of the existing certificates, so that all new certificates are fully validated according to the appropriate standards.

The supporting information related to further claims can be founded within the above linked thread and the formal responses.
 

Google Internet Authority G2 CA that issues many of Google's TLS certificates is signed by GeoTrust Global CA, which presumably is owned by Symantec now. So do Google's own certificates fall into the "gradually untrustworthy" category?

As discovered during the follow-up of a previous incident - https://security.googleblog.com/2015/10/sustaining-digital-certificate-security.html - Symantec has cross-signed two organizations who operate wholly independent infrastructure that is audited as such - Apple and Google. This is documented at https://chromium.googlesource.com/chromium/src/+/master/net/data/ssl/symantec/README.md

As such, these certificates are not "Symantec-issued", for any reasonable definition, and are effectively operated by independent third parties, and thus would be proposed to be excluded from these requirements.

This evaluation of trust is not based upon an intrinsic property, but on the public disclosure and ability to independently confirm and assess the policies and practices related to issuance by these two parties.

yuhong...@hotmail.com

unread,
Mar 24, 2017, 12:29:56 AM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org, tpg...@gmail.com
Interestingly, many are 90 days valid, but not all of them (for example, Nest). 

hot73...@gmail.com

unread,
Mar 24, 2017, 12:37:05 AM3/24/17
to blink-dev
Can you elaborate on how the gradual distrust will work? If a Symantec issued cert is valid for three years, will it be immediately treated as invalid if the validity period in Chrome 59 is 33 months?

Or will the 33 month validity period only allow the certificate to be valid for 33 months (and gradually decreasing) from the date of issue effectively ignoring the expiration date as issued?

jussi....@valu.fi

unread,
Mar 24, 2017, 1:42:15 AM3/24/17
to blink-dev

Two questions about this:

1. Can you clarify what "effective immediately" means? Will this occur with the next stable release of Chrome? Or will an update be pushed to the current stable version?

As this is a security related incident, the proposal is to land this in Trunk and update Dev and Beta to reflect this, treating this as "Medium" severity per https://www.chromium.org/developers/severity-guidelines by treating this as an address bar spoofing issue.
 
2. Can you confirm that the removal of EV indicators will affect *all* Symantec-issued EV certificates, regardless of their compliance with Chrome's reduced validity schedule?

Correct. Extended Validation certificates are recognition of the processes and controls related to validating the information in the certificates is accurate and correct. The issues highlighted have undermined that confidence. The validity issue is separate and mitigates future issues that may arise or be discovered.

So, does this mean that e.g. paypal.com will lose its EV status? Doesn't this introduce whole new possibilities for fake/spoof Paypal websites as the original site doesn't have EV status anymore?

userw...@gmail.com

unread,
Mar 24, 2017, 2:29:31 AM3/24/17
to blink-dev, ag...@andrewayer.name, rsl...@chromium.org

So out of curiosity:

Assumptions:
1. Say I am Symantec or any SubCA/RA/DTP/subdivision/... operating under one of their roots.
2. I am not exempt from the measures (Apple/Google subCAs) or one of the naughty third parties (CrossCert/Certisign/Certsuperior/Certisur).
3. No additional violations of the rules by Symantec are discovered (yeah, bear with me here...)

Goal: Have trusted certs in Chrome despite the sanctions.

Possible way to achieve this:

1. non-EV:
a. Existing: To make sure my certificates remain trusted in Chrome, I provide my customers a way to reissue/convert their existing multi-year certs to some with max validity period of 9 months (and I allow them to reissue for as long as the original cert/contract would have lasted). I can do that because the existing validation info lasts for up to 39 months. That cert swap is all that is needed for customer sites to remain trusted and nobody has to worry about transition periods/Chrome updates breaking things. All correct?
b. New: I promptly start to issue new certs with <= 9 months validity. I could, however, still offer multi-year contracts because validation info for my new certs also lasts for 39 (or 27 in 2018) months just like it does for every other CA. Correct?
c. Future: The 9 months validity requirement is basically the full extent of the proposed sanctions, I won't have to create new root or sub CAs and Chrome won't decide to distrust my certs in a year or two. This is not a road to complete or partial distrust unless I fuck up again. Correct?

2. EV:
Not much I can do, as a punishment my existing and new EV certs are effectively DV certs in Chrome58+ for at least a year. I could, however, work with another EV-enabled CA to keep providing EV services. If they perform the validation, that should be allowed. Correct? What if they (cross-)sign an EV subCA of mine with valid audits and stuff? Not allowed?

Message has been deleted

pzb...@gmail.com

unread,
Mar 24, 2017, 9:32:10 AM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org


On Thursday, March 23, 2017 at 9:09:57 PM UTC-7, Ryan Sleevi wrote:
On Thu, Mar 23, 2017 at 2:07 PM, <ben.m...@gmail.com> wrote:
(just to be super clear I'm writing this in a personal capacity and not representing the thoughts of my employer)

Did you guys consider something where there would be an escalating level of distrust (eg, in one version you get a warning rather than a green lock, the next release a red not secure, then a intersticial)? This would allow attentive operators who might not hear about this change otherwise to figure out that there is an issue before their site breaks completely. It also gives all parties involved a small number of times where they can promote this change.

Yes, this was considered. This is similar to the approach used with SHA-1 deprecation, but in this case, may cause undesired security risk, in that such certificates continue to be used, and thus private information (such as cookies) are still transmitted, and privileged features (such as geolocation) may still be enabled. This proposal was an attempt to balance those security concerns with the interoperability and compatibility risk.

I realize API changes usually don't have corresponding UX, but as was called out in the initial email this is using the blink intent process for something that is not technically Blink.

To that end, I think that this this intent seems to be unclear on the resulting user experiences.  There are a variety of gradations of trust reduction available, ranging from simply treating as neutral (currently the ⓘ) to affirmatively insecure (currently red struck through https) to bypassable interstitial to non-bypassable interstitial.    Can you clarity which level of trust is proposed for certificates exceeding the defined maximum validity periods?  Is there any intent to vary this level of trust across the proposed schedule?

Thanks,
Peter

Rick Byers

unread,
Mar 24, 2017, 9:42:10 AM3/24/17
to pzb...@gmail.com, blink-dev, awha...@chromium.org, Ryan Sleevi
I'm following this thread with great interest and trying to build up the context in order to form a well-reasoned opinion.  In the interim I just wanted to clarify one point below:

On Fri, Mar 24, 2017 at 9:32 AM, <pzb...@gmail.com> wrote:

On Thursday, March 23, 2017 at 9:09:57 PM UTC-7, Ryan Sleevi wrote:
On Thu, Mar 23, 2017 at 2:07 PM, <ben.m...@gmail.com> wrote:
(just to be super clear I'm writing this in a personal capacity and not representing the thoughts of my employer)

Did you guys consider something where there would be an escalating level of distrust (eg, in one version you get a warning rather than a green lock, the next release a red not secure, then a intersticial)? This would allow attentive operators who might not hear about this change otherwise to figure out that there is an issue before their site breaks completely. It also gives all parties involved a small number of times where they can promote this change.

Yes, this was considered. This is similar to the approach used with SHA-1 deprecation, but in this case, may cause undesired security risk, in that such certificates continue to be used, and thus private information (such as cookies) are still transmitted, and privileged features (such as geolocation) may still be enabled. This proposal was an attempt to balance those security concerns with the interoperability and compatibility risk.

I realize API changes usually don't have corresponding UX, but as was called out in the initial email this is using the blink intent process for something that is not technically Blink.

Note that for quite some time now we've been using this web platform change process for most chromium changes things that impact web developers, regardless of where the code happens to be located (the division between blink and content is pretty artificial these days).   Here's a few recent examples which I believe are not (or at least primarily not) blink:


That said, I think we're all trying to figure out if this web platform change process will provide value to the web community for tricky PKI-decisions like this.  But at a minimum, the extra transparency and community involvement seems clearly valuable.

lei...@gmail.com

unread,
Mar 24, 2017, 10:17:58 AM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org
Are there any plans to put the "View certificate" button back where it belongs: "Click on the padlock, click on 'view certificate'"?

Half of the times F12 invokes a debugger, and it is counter-intuitive.not just following the padlock.

crypt...@gmail.com

unread,
Mar 24, 2017, 11:08:55 AM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org
Disgusting abuse of your power!  You are punishing 30,000 websites, 99.9%+ of whom are completely legitimate, in order to exact revenge against Symantec.

LEAVE THE INNOCENT BYSTANDERS ALONE!!!!

I propose you block all *new* Symantec certificates until they go back and re-validate (AT THEIR EXPENSE) all the 30,000 websites, and revoke any that are found incorrect.

Be responsible with the power you have, and mindful of the massive collateral damage your actions cause!

You've already just destroyed wosign and startssl wreaking havoc across their entire user base: ***WE*** SUFFER when *you* attack CAs... so STOP IT!!!!

fabri...@gmail.com

unread,
Mar 24, 2017, 12:10:02 PM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org, crypt...@gmail.com


yes, I agree with this, doesn't sound much fair what your proposed solution. Are you aware this move will affect thousands of e-commerce sites and related businesses??! Please, ponder your decision carefully and try to find the best compromise for everyone here.

tbar...@encirca.com

unread,
Mar 24, 2017, 12:23:08 PM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org, crypt...@gmail.com, fabri...@gmail.com
Actually,  this impacts ALL existing EV certs from Symantec.  Not just the 30,000 that were improperly issued.  right?

ad...@adamcaudill.com

unread,
Mar 24, 2017, 12:35:35 PM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org, crypt...@gmail.com
(Standard disclaimer: speaking on my own behalf, not that of my employer, etc.)

> Disgusting abuse of your power!  You are punishing 30,000 websites [...snip...]

Browsers have a substantial responsibility to protect their users, limiting validity and otherwise placing restrictions on a CA is an important tool to minimizing the harm that a CA can do. Browsers, on behalf of users, make decisions about which CAs can be trusted and which ones can't - strict rules and requirements are placed on CAs to ensure that they can be trusted, if these rules are violated, that trust relationship is damaged. If a browser loses faith in a CA, and the controls they have in place to ensure that these requirements are met, they have a responsibility to their users to take action.

If there are signs that controls are defective, if there are signs that a significant number of certificates were issued in violation of these rules, if there is reason to believe that the CA didn't discover issues when they should have, or ignored issues they did discover - there is good reason to lose faith in that CA and all of the certificates they've issued.

> You've already just destroyed wosign and startssl [...snip...]

WoSign/StartSSL showed blatant disregard for the requirements placed on CAs, and in doing so exposed not just their customers to additional risk, but all users of browsers that trusted them. The browsers (not just Chrome) took action to protect their users from such reckless action.

Removing trust, or restricting trust, in a CA is a serious action, with impact to many; for those that operate TLS servers, that impact is often additional work and financial costs, for users (the group browsers are primarily responsible to), the impact is a reduction in risk. CAs that fail their customers by not living up to their obligations should bear the burden of the problems they've introduced, not the browsers that call them out on it. In the case of WoSign, they knowingly performed actions that violated their obligations - they created the situation, they are responsible for failing their customers.

If you want to be mad at someone, be mad at CAs that violate the requirements that keep users safe.
 

guillaume...@gmail.com

unread,
Mar 24, 2017, 12:48:07 PM3/24/17
to blink-dev
If I understand correctly all existing Symantec-issued certificates with a 2 years validity period will be untrusted with September 12th Chrome release, whatever their Not Before date is. Am I right?

I yes, will that generate a big red untrusted certificate error, or will it just not show a green padlock ?

cc1265...@gmail.com

unread,
Mar 24, 2017, 12:48:24 PM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org
 Google Chrome 浏览器将整治所有赛门铁克 SSL/TLS 证书
在一份[声明]中,Google Chrome 团队开发者 Ryan Sleevi 称发现赛门铁克错误颁发三万多个 SSL 证书,并且知晓后在长达一个月的不作为,Chrome 立即将赛门铁克及其所属 CA 的颁发的 SSL EV 证书(浏览器地址栏显示绿锁)降级到普通证书(灰锁),并在未来几个版本的浏览器更新中逐步过期所有赛门铁克证书。赛门铁克证书占所有活跃证书数量的 30%~45%,如果不是考虑到大规模瘫痪整个网络的风险,Chrome 应该一夜间吊销所有赛门铁克证书。
此次事件影响极大,国内的七牛和阿里云等网络巨头已经在过去一段时间里颁发了无数个免费赛门铁克证书。颁发及使用赛门铁克证书的公司应该立刻采取行动,更换证书。
Google的声明原文:https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/eUAKwjihhBs

在 2017年3月24日星期五 UTC+8上午12:03:20,Ryan Sleevi写道:

to...@tobireif.com

unread,
Mar 24, 2017, 1:09:24 PM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org
Thank you for working towards trust and security.

I hope Symantec et al will fulfill the requirements in the future.

Jeffrey Yasskin

unread,
Mar 24, 2017, 1:10:40 PM3/24/17
to Ryan Sleevi, Ben Maurer, blink-dev, awha...@chromium.org
Both the original proposal and the SHA-1-like suggestion allow unwanted certificates to be used for some time; they just differ on which certificates and what kind of use.

Original proposal:
Chrome 59 (Dev, Beta, Stable): 34-39 months blocked; 10-33 allowed
Chrome 60 (Dev, Beta, Stable): 28-39 months blocked; 10-27 allowed
Chrome 61 (Dev, Beta, Stable): 22-39 months blocked; 10-21 allowed
Chrome 62 (Dev, Beta, Stable): 16-39 months blocked; 10-15 allowed
Chrome 63 (Dev, Beta): 10-39 months blocked; no unwanted certificates allowed
Chrome 63 (Stable): 16-39 months blocked; 10-15 allowed15 months validity (465 days)
Chrome 64 (Dev, Beta, Stable): 10-39 months blocked; no unwanted certificates allowed

SHA-1-like, elaborated with a schedule the same length as the OP. I have no idea if keeping the same schedule makes sense when the escalation proceeds along a different axis, and I trust Ryan to change it appropriately.

Chrome 59 (Dev, Beta, Stable): 10-39 months treated as passive mixed content; 10-39 months allowed to use [SecureContext] APIs
Chrome 60 (Dev, Beta, Stable): same
Chrome 61 (Dev, Beta, Stable): 10-39 months treated as HTTP, not allowed to use [SecureContext] APIs. 10-39 months allowed to load.
Chrome 62 (Dev, Beta, Stable): same
Chrome 63 (Dev, Beta): 10-39 months blocked; no unwanted certificates allowed
Chrome 63 (Stable): same as 62
Chrome 64 (Dev, Beta, Stable): 10-39 months blocked; no unwanted certificates allowed

This schedule concentrates the breakage for Chrome in 61 and in 64, but gives each site a longer time to notice and deal with their problems.

One could imagine combining the two schedules, to completely block long-validity certs earlier, but also remove the lock from 10-21-month certs more quickly in order to warn site operators.

HTH,
Jeffrey

crypt...@gmail.com

unread,
Mar 24, 2017, 1:21:02 PM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org, crypt...@gmail.com, ad...@adamcaudill.com
I fail to see anything relevant you've said?  Yes - we are all mad at Symantec, and random google vigilante-employees want to cause extreme pain and damage to that company: fair enough.

However: screwing over 30,000+ innocent bystanders IS NOT THE WAY TO DO IT.

The startcom issue was just one naughty certificate; the destruction of all startcom user websites (including mine) was a decision google made to punish startcom for lying about their business relationship with wosign.  This severely hurt huge numbers of innocent bystanders again, caused irrevocable damage (no other CA offers unlimited wildcard SANs) and massive costs (startcom were 10x+ less expensive that the greedy big American CAs ripping us all of $millions for nothing more than a few DNS lookups and crypto operations).

I'm not supporting Symantec, and not supporting startcom either.

I'm asking that you figure out who the bad guy is, and stop punching us instead of them.  The Google fist is a one-punch killer.  Be ***responsible*** with how you wield that power.  Execute just bad guy, don't commit genocide!!!
Message has been deleted

PhistucK

unread,
Mar 24, 2017, 1:36:48 PM3/24/17
to crypt...@gmail.com, blink-dev, awha...@chromium.org, Ryan Sleevi, ad...@adamcaudill.com
Certificate authorities know they are responsible for the actions of the companies that validate certificates for them. They must ensure that they do not abuse their power. Constantly.

Think about it. If browsers do nothing, users may simply encounter websites that pass the TLS validation, but are actually hosting the content of an attacker and not the actual website.
Why do you expect browsers to jeopardize their users?
And how can browsers tell that the users are protected? They cannot know whether a certain certificate is mis-issued (they can, by manually validating every website, which is unreasonable).

The solution (which is known and common) is not to trust that certificate authority.
Because distrusting it completely will ruin the internet (due to the very large number of websites that use it), browsers are pretty much locked into a really bad state where they have to gradually distrust it.
The safest thing to do would have been to just outright distrust it, but then users would significantly suffer.

So this is a compromise. It is not a good one (it is not very safe), but it is a compromise nonetheless.

Also, remember that the recommendation is to have a short-validity certificate (the shorter the better), if I understand correctly. If websites (or their certificate issuers) are not following the recommendation, they should not be surprised when they have a bad time in a way, right?
Note that this intent does not mention any future plan to completely distrust Symantec, so following the recommendation would have actually helped you not be impacted at all from those measures (barring extended validation scenarios, that is).


PhistucK

--

mega...@megazone.org

unread,
Mar 24, 2017, 2:08:29 PM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org, crypt...@gmail.com, ad...@adamcaudill.com
On Friday, March 24, 2017 at 1:21:02 PM UTC-4, crypt...@gmail.com wrote:
I fail to see anything relevant you've said?  Yes - we are all mad at Symantec, and random google vigilante-employees want to cause extreme pain and damage to that company: fair enough.

You've completely missed the point.  This is NOT about causing 'extreme pain and damage' to Symantec.  This is about *protecting ALL Chrome users!*
 
However: screwing over 30,000+ innocent bystanders IS NOT THE WAY TO DO IT.

It is the ONLY way to do it.
 
The startcom issue was just one naughty certificate; the destruction of all startcom user websites (including mine) was a decision google made to punish startcom for lying about their business relationship with wosign.  This severely hurt huge numbers of innocent bystanders again, caused irrevocable damage (no other CA offers unlimited wildcard SANs) and massive costs (startcom were 10x+ less expensive that the greedy big American CAs ripping us all of $millions for nothing more than a few DNS lookups and crypto operations).

None of that is Google's problem or responsibility.  Startcom misbehaved and destroyed trust in them.  With a CA trust is all or nothing - once they've shown themselves to be untrustworthy you can't trust *anything* they've done.  You MUST distrust ALL certs they've issued. Period.  That's how it works, like it or not.

Same with Symantec.  There is no way for Google to individually validated every cert that Symantec has issued.  The point is not that they've done 30,000 suspicious certs - the point is that those 30,000 certs are strong evidence that Symantec *has not been doing their job!*  That means *ALL* certs Symantec have issued are now suspect and potentially issued in error.  It is NOT just 30,000 certs - that's just what Google has found - it is very likely many times that number, with the majority not having been discovered yet.

Again, unfortunately this is how CAs work.  The evidence is that Symantec has not been doing their job correctly by not properly verifying the owners when they issue certs.  That's what being a CA is all about.  If they've failed in such a fundamental way, unfortunately it means you can't trust them to have done it right in *any* case.  

Yes, this means collateral damage in that valid owners who did nothing wrong will need to get new certs.  But this is not Google's fault nor their problem - the blame falls 100% on Symantec.  They sold themselves as a CA and sold those certs base on being a *trusted* authority.  By behaving in a way that undermined that trust, they effectively defrauded their customers by selling them certs with no backing.  (When a CA issues a cert it all rests on that CA being trusted.)

Google's responsibility is NOT to Symantec's customers - that's Symantec's responsibility - but to their customers, aka Chrome users.  To protect their customers they must downgrade the trust given to Symantec's certs - that's the ONLY viable option given the way the system works.

It seems like you just don't understand how the system is structured.  Google cannot simply block 'bad' certs, because they have no way to know which certs are bad or not.  That was Symantec's job - and they failed.  Google has to either trust Symantec to get it right and accept all of their decisions, or not trust Symantec to get it right, and therefore NOT trust their decisions.  And the evidence shows they did NOT get it right.

Life is not always fair.  It isn't the small site owners fault they trusted Symantec, but if they get caught in the fallout they need to blame Symantec - not Google.  And they need to take their business elsewhere.

Adam Caudill

unread,
Mar 24, 2017, 2:16:25 PM3/24/17
to crypt...@gmail.com, blink-dev, awha...@chromium.org, rsl...@chromium.org
I fail to see anything relevant you've said?

Thanks. Good to know.

> [...snip...] destruction of all startcom user websites (including mine) was a decision google made to punish startcom [...snip...]

I find it interesting that you place blame on Google, when all major browsers took action, and Google's plan more or less mirrored the plan Mozilla had already announced. [Apologies for the typo in the StartCom name in my last email.]

When a CA violates their obligations, the browsers (all of them), have an obligation to take action to protect their users. If there's no faith in the controls that a CA has in place, as was the case with WoSign (including StartCom), the only option left to browsers is to stop trusting the certificates they've issued. Trusting certificates that are issued in violation of CABF baseline requirements / root program requirements put users at risk, that's why these requirements exist. You may take the position that convenience / cheap certificates are more important that ensuring the security of users, I don't.

I don't see an action against a CA as being a punishment against their users, although the impact to them is quite unfortunate, but it is unavoidable if the goal is to eliminate certificates that are of unknown trust. When browsers loose faith in a CA, and their ability to comply with the obligations they accept as being a CA, the only course of action is to eliminate the untrusted certificates from the ecosystem and take steps to ensure future compliance. It would be great if there was a way to easily determine which certificates were issued in compliance with all requirements, and only eliminate those that weren't - a problem that has been discussed many times. Unfortunately, that's not always possible - when that's not possible, that means all certificates that were issued by the CA are suspect.

I'm not supporting Symantec

I'm not taking a position on them here, nor endorsing any plan of action (nor will I); my point is that the browsers have an obligation to eliminate or restrict the trust of CAs that have committed substantial violations of the requirements they accepted. If the browsers have lost faith in a CA, any CA, their obligation is to take actions to protect their users and eliminate the untrusted certificates from the ecosystem.



crypt...@gmail.com

unread,
Mar 24, 2017, 2:31:49 PM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org, crypt...@gmail.com, ad...@adamcaudill.com, mega...@megazone.org
Hard to tell if you're just trolling now.

>This is about *protecting ALL Chrome users!*
This is not true.

Take for example online banks. 90% of them do not use HSTS or HPKP, and more than half operate websites with no SSL whatsoever.  If google honestly wanted to protect users, they would deal with the massive security holes that put users at risk, not fuss around with procedural squabbles. Lets look at the big picture here: it's highly unlikely that even one single Symantec cert will ever cause harm to anyone.

But that's not the point:

Attacking a teeny minority of PERFECTLY HONEST websites because you disagree with their CA's procedures is not about protecting users.  It's about punishing Symantec but causing their customers the greatest amount of pain possible.

Zero customers need to hurt here.

If google wanted - they could make Symantec fix the problem.  The argument is over issuance procedures: make them RE DO those procedures in an audited an compliant way.  Problem fixed: no collateral damage caused.

It appears like someone does not want to do that though: they want to cause the maximum pain, and are disguising their vindictive attack under the fake banner of "protect users".

 
However: screwing over 30,000+ innocent bystanders IS NOT THE WAY TO DO IT.

It is the ONLY way to do it.

Look in a mirror: that's an evil bully staring back at you.  You know full well this can be fixed without hurting customers, if google wanted to.

None of that is Google's problem or responsibility.  Startcom misbehaved and destroyed trust in them.  With a CA trust is all or nothing - once they've shown themselves to be untrustworthy you can't trust *anything* they've done.  You MUST distrust ALL certs they've issued. Period.  That's how it works, like it or not.

Pretend, for a moment, you live in a universe where you were not allowed to hurt bystanders for the sins of the company they buy from.
... but it is still your job to fix the problem.
It's not impossible.
What would you do?

 
Life is not always fair.  It isn't the small site owners fault they trusted Symantec, but if they get caught in the fallout they need to blame Symantec - not Google.  And they need to take their business elsewhere.

Re-read that last sentence you wrote.
And again.
Google what "vindictive" means.

Chris Palmer

unread,
Mar 24, 2017, 2:50:49 PM3/24/17
to blink-dev
Hi all,

This is a difficult problem and to deal with it well we need, facts, figures, and sound arguments. We also need civil discourse. Thanks.

MegaZone

unread,
Mar 24, 2017, 2:57:47 PM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org, crypt...@gmail.com, ad...@adamcaudill.com, mega...@megazone.org
On Friday, March 24, 2017 at 2:31:49 PM UTC-4, crypt...@gmail.com wrote:
>This is about *protecting ALL Chrome users!*
This is not true.

You are, of course, entitled to your opinion.
 
Take for example online banks. 90% of them do not use HSTS or HPKP, and more than half operate websites with no SSL whatsoever.  If google honestly wanted to protect users, they would deal with the massive security holes that put users at risk, not fuss around with procedural squabbles. Lets look at the big picture here: it's highly unlikely that even one single Symantec cert will ever cause harm to anyone.

But that's not the point:

You're right.  This has nothing to do with the subject at hand and was a silly attempt at a red herring.  Forcing the use of SSL/TLS is not at all the same of ensuring it is trustworthy when it is used.  If a user goes to a non-secured site they're fully aware of that and it is their choice.  If they go to an SSL/TLS secured site they have a reasonable expectation of security, and if a CA behaves in a way to undermine that it undermines the entire system.
 
Attacking a teeny minority of PERFECTLY HONEST websites because you disagree with their CA's procedures is not about protecting users.  It's about punishing Symantec but causing their customers the greatest amount of pain possible.

Again, I disagree.  It is about enforcing minimum standards.  And yes, that means Symantec's customers become leverage against them - in the end it is all about money.  Symantec doesn't seem to want to fix the issue, but if they lose revenue they might change their mind and realize it's better business to do the right thing.  Unfortunately, this is what it often comes down to.  It isn't about causing pain, but the pain may be the only means to achieve the end with an unwilling participant.
 
Zero customers need to hurt here.

If google wanted - they could make Symantec fix the problem.  The argument is over issuance procedures: make them RE DO those procedures in an audited an compliant way.  Problem fixed: no collateral damage caused.

It appears like someone does not want to do that though: they want to cause the maximum pain, and are disguising their vindictive attack under the fake banner of "protect users".

This *is* Google making Symantec fix the problem.

Did you miss the part where Google repeatedly approached Symantec about this issue?  Or that Symantec has even responded to this announcement by basically brushing it off?  Google cannot force Symantec to do anything,  Symantec has indicated they have no interest in fixing their broken behavior.  They seem to feel they're 'too big to fail' and that they can just ignore this and Google will have to back off of this decision.

The ONLY option Google has in this case is to revoke their trust in Symantec as a CA.  If Symantec won't fix the issue then Google has to take what action they can.  That may cause Symantec to lose business, aka revenue, as customers seek alternative certs.  If the loss of business looks like it is going to cost them more than the cost of fixing the issue, then they may decide to do the right thing and fix the problem.  But Google cannot *force* anything - the choice is Symantec's, and so far they've decided not to fix it.
 
Look in a mirror: that's an evil bully staring back at you.  You know full well this can be fixed without hurting customers, if google wanted to.

I'm happy with the man in the mirror.

The only way to 'fix' this would be to revoke ALL existing Symantec certs and reissue them with proper verification.  Only Symantec can do that, and they've shown no interest in doing so.  Probably because it would cost them millions and they'd rather play chicken with Google and gamble they can get away without spending the money than do right by their users.  (There is a reason I haven't used any Symantec products in years.)  Like Ford and the Pinto.
 
Pretend, for a moment, you live in a universe where you were not allowed to hurt bystanders for the sins of the company they buy from.
... but it is still your job to fix the problem.
It's not impossible.
What would you do?

Try to work with the company to fix it.  But if they refuse, well, I'd still have to do the right thing and it might suck for the little guy - but that's the lesser of two evils.

You seem to feel that Google should just let Symantec continue to misbehave just to avoid inconveniencing the cert owners.  I do not agree with that.  It is about more than this one CA.  If Symantec is allowed to get away with not fixing their behavior then it undermines the whole system, and why should any other CA do any better if Symantec's approach is good enough?
 
Life is not always fair.  It isn't the small site owners fault they trusted Symantec, but if they get caught in the fallout they need to blame Symantec - not Google.  And they need to take their business elsewhere.

Re-read that last sentence you wrote.
And again.
Google what "vindictive" means.

You keep on using that word.  I do not think it means what you think it means.

It is not vindictive, it is reality.  It is not about 'revenge' or punishment, it is about fixing broken behavior.  If Symantec won't do it because it is the right thing to do, maybe they'll do it because not doing it hurts their bottom line.  That's just how the world works, and sometimes the little guy gets caught in the process.

That's a shame, but again, that's all on Symantec for 1) not doing their job in the first place and 2) not owning up to it and making it right when it was discovered.  If they had done *either* of these then we would have never reached this point in the first place.  Now the choice is either to let them get away with it and undermine the system, or to take action to protect the system by cutting out the rot.  I'm in favor of the latter, even if that means some good tissue has to be excised to make sure all of the cancer is gone.

Chris Palmer

unread,
Mar 24, 2017, 3:07:08 PM3/24/17
to blink-dev
This thread is going to be a centithread :) so let's try to keep it as concise as possible. Please try to post only new information.

Also, the Chromium Code Of Conduct is a good document: https://www.chromium.org/conduct

philli...@gmail.com

unread,
Mar 24, 2017, 3:24:06 PM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org
On Thursday, 23 March 2017 16:03:20 UTC, Ryan Sleevi wrote:


Summary

Since January 19, the Google Chrome team has been investigating a series of failures by Symantec Corporation to properly validate certificates. As.


While I understand it is probably difficult to get consensus support for punitive measures against an organisation that has entrenched itself so deeply into the CA landscape by means of acquisition ('too big to fail' mentality), care should be taken to try to avoid accusations of abuse of market power by Google/the Chrome project, and so far Chrome seems to be taking unilateral action. Is this the result of failed discussions within the CA Browser Forum, or just a decision that the forum is moving too slowly, or?

Has Symantec stopped the on-going abusive issuances?

Given a central plank of the justification for this proposal is how Symantec have responded, what was unsatisfactory - specifically - about Symantec's responses VS what was expected and is this documented anywhere that can be reviewed?

karthikey...@gmail.com

unread,
Mar 24, 2017, 3:30:03 PM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org, crypt...@gmail.com, ad...@adamcaudill.com
1. Is this an official release from Google. Do we take this seriously or its just a blog.
2. Have you published the list of 30K websites. How do we trust this number is valid. Is it exactly 30000?

Rick Byers

unread,
Mar 24, 2017, 3:34:34 PM3/24/17
to Chris Palmer, blink-dev
On Fri, Mar 24, 2017 at 3:06 PM, 'Chris Palmer' via blink-dev <blin...@chromium.org> wrote:
This thread is going to be a centithread :) so let's try to keep it as concise as possible. Please try to post only new information.

Also, the Chromium Code Of Conduct is a good document: https://www.chromium.org/conduct

Thanks Chris.  This debate is definitely an important one to be having in a public and respectful fashion.  

Since this is an issue that involves the whole web ecosystem, I'd personally also encourage posts here which link to relevant discussion elsewhere (even when that discussion elsewhere is not "as concise as possible").  As a Blink API OWNER I'm certainly doing whatever I can to read and digest all the history/debate on this topic across the various locations.  I'm sure we'll see several thoughtful blog/medium posts over the coming week with different people's perspectives and I'm looking forward to trying to digest them all ;-).  I would not want to discourage people from posting such links here to maximize visibility and diversity of perspectives.

Ultimately this specific thread is about the chromium open source project making a decision on whether Ryan's proposal meets our web platform change guidelines around balancing compat/interop risk with benefit to the web.  But there's obviously a much larger set of discussions to be had across the community around all of this which we should try not to conflate too much with the specific question at hand here.


crypt...@gmail.com

unread,
Mar 24, 2017, 3:39:27 PM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org, crypt...@gmail.com, ad...@adamcaudill.com, mega...@megazone.org
Vindictive is taking action designed to force Symantec customers to "take their business elsewhere" without properly exhausting "talks" and other persuasive measures first.
If Symantec are non-compliant: stop trusting every certificate they issue from today onward until they become complaint.  That does not hurt customers.
Easy. No collateral damage. If they still refuse (which I doubt, since their income is halted until the comply), but if they still do, then AND ONLY THEN, is it time to consider the collateral damage approach.  The problem still can be fixed without 3rd party pain at this point.

What exactly am I missing here?

I love the irony that an argument about how google is abusing it's monopolistic power and causing pain to 30,000+ innocent webmaster (and untold millions of end users) drew a "code of conduct" warning against me.  That conduct document embodies a spirit of behavior.  I recommend we ALL follow that spirit, not just with this discussion, but with we how all act in resolving this situation.

crypt...@gmail.com

unread,
Mar 24, 2017, 3:45:49 PM3/24/17
to blink-dev, awha...@chromium.org, rsl...@chromium.org
How do we get someone from google legal department involved?

It's not my job to reign in vigilantes, but it seems reasonable to expect that much of the responses in this thread would make for troublesome reading in a lawsuit which might well come from this action.  Far too many unsubstantiated allegations, and refusals to consider options, and other worrying remarks: I don't know which ones are internet strangers or google employees... if any are the latter: that's a huge problem.

tpg...@gmail.com

unread,
Mar 24, 2017, 3:47:20 PM3/24/17
to blink-dev
I think the immediate removal of EV indicator is very unnecessary. Practically speaking very few users are going to think twice about it. They will not suddenly become educated in what OV means compared to DV and re-evaluate how that fits into their busy lives. It's not going to prevent any attack surely. What it will do is produce confusion, regulatory heartache, and money wasted on fixing perception rather than demonstrable risk.


brian.we...@gmail.com

unread,
Mar 24, 2017, 3:54:26 PM3/24/17
to blink-dev
I have read a few articles about this issue and read statements from both parties and all I have seen from Google seems to be inflated numbers without complete information about the 127 certificates that you seem to inflate to 30,000. Can you even explain where you get this 30,000 number from when Symantec's statement reads

"Google's statements about our issuance practices and the scope of our past mis-issuances are exaggerated and misleading," they wrote. "For example, Google’s claim that we have mis-issued 30,000 SSL/TLS certificates is not true. In the event Google is referring to, 127 certificates—not 30,000—were identified as mis-issued, and they resulted in no consumer harm. We have taken extensive remediation measures to correct this situation, immediately terminated the involved partner’s appointment as a registration authority (RA), and in a move to strengthen the trust of Symantec-issued SSL/TLS certificates, announced the discontinuation of our RA program."

No offense, but where are the numbers of other CA's so we have some comparison? I have heard Google use the word "transparency" but this is looking more like a witch hunt at this point.

Florian Weimer

unread,
Mar 24, 2017, 4:27:14 PM3/24/17
to crypt...@gmail.com, blink-dev, awha...@chromium.org, rsl...@chromium.org, ad...@adamcaudill.com, mega...@megazone.org
> If Symantec are non-compliant: stop trusting every certificate they issue
> from today onward until they become complaint. That does not hurt
> customers.

But isn't this actually more draconian?

Under your proposal, if a Symantec customer needed a new certificate,
they would have to switch at once to a different certificate
authority. This would require immediate business process changes on
the side of the customer. With the gradual decrease in certificate
lifetime, an immediate switch is not enforced.

Ryan Sleevi

unread,
Mar 24, 2017, 5:26:16 PM3/24/17
to jussi....@valu.fi, blink-dev


On Fri, Mar 24, 2017 at 1:42 AM, <jussi....@valu.fi> wrote:
So, does this mean that e.g. paypal.com will lose its EV status?

Yes

Ryan Sleevi

unread,
Mar 24, 2017, 5:35:35 PM3/24/17
to userw...@gmail.com, blink-dev, Andrew Ayer, Ryan Sleevi


On Fri, Mar 24, 2017 at 2:29 AM, <userw...@gmail.com> wrote:
So out of curiosity:

Assumptions:
1. Say I am Symantec or any SubCA/RA/DTP/subdivision/... operating under one of their roots.
2. I am not exempt from the measures (Apple/Google subCAs) or one of the naughty third parties (CrossCert/Certisign/Certsuperior/Certisur).
3. No additional violations of the rules by Symantec are discovered (yeah, bear with me here...)

Goal: Have trusted certs in Chrome despite the sanctions.

Possible way to achieve this:

1. non-EV:
a. Existing: To make sure my certificates remain trusted in Chrome, I provide my customers a way to reissue/convert their existing multi-year certs to some with max validity period of 9 months (and I allow them to reissue for as long as the original cert/contract would have lasted). I can do that because the existing validation info lasts for up to 39 months. That cert swap is all that is needed for customer sites to remain trusted and nobody has to worry about transition periods/Chrome updates breaking things. All correct?

Correct
 
b. New: I promptly start to issue new certs with <= 9 months validity. I could, however, still offer multi-year contracts because validation info for my new certs also lasts for 39 (or 27 in 2018) months just like it does for every other CA. Correct?

Correct
 
c. Future: The 9 months validity requirement is basically the full extent of the proposed sanctions, I won't have to create new root or sub CAs and Chrome won't decide to distrust my certs in a year or two. This is not a road to complete or partial distrust unless I fuck up again. Correct?

Correct
 
2. EV:
Not much I can do, as a punishment my existing and new EV certs are effectively DV certs in Chrome58+ for at least a year. I could, however, work with another EV-enabled CA to keep providing EV services. If they perform the validation, that should be allowed. Correct? What if they (cross-)sign an EV subCA of mine with valid audits and stuff? Not allowed?

Note: This is not a punishment. This is an expression of the uncertainty related to the validation procedures and the process and controls related to those procedures, of which evidence and admission have been provided that demonstrates Symantec has not suitably adhered to the level of diligence expected, and as codified in the Baseline Requirements, which underpin the Extended Validation Guidelines. The removal of EV reflects the uncertainty that has resulted from the sustained period of issues.

With how EV works, an issuing CA (trust anchor) is recognized for "EV trust" given a set of particular OIDs in their certificate policy. If a server certificate bearing one of these policies is presented, and a valid path (appropriate for the policies) can be built to a root enabled for those policies, the certificate is recognized as EV.

While the CA/Browser Forum has defined a generic set of EV policy OIDs, many CAs make use of vendor-specific OIDs that reflect policies in their CP/CPS, which may and often do go above and beyond what the EV Guidelines require, in order to satisfy their business and customer needs. As a consequence, such an issuing CA would have to recognize the sub-CA as being compliant with those issuance policies expressed in the issuing CA's CP/CPS. The issuing CA would need to receive an audit that they believe sufficiently demonstrates the sub-CA's compliance with the issuing CA's policies - which would be the appropriate WebTrust or ETSI set of schemes. As Symantec currently operates WebTrust-audited CAs, this presumably is "WebTrust for CAs", "WebTrust for CAs - SSL", and "WebTrust for CAs - EV" (as they're commonly abbreviated).

Given that these issues are reasonable grounds to expect that the next period of audit should produce "qualifications" (due to these issues of non-compliance), the issuing CA would be accepting the liability, risk, and responsibility to ensure the sub-CA is operating according to both the issuing CA's and the sub-CA's stated CP/CPS, that the CP/CPS appropriately reflect the requirements contained within the "Baseline Requirements" and "Extended Validation Guidelines", and the audits appropriately examine compliance to these statements.

As noted earlier on this thread, Symantec bore this responsibility with their Delegated Third Parties, and by the evidence presented by them, failed to uphold this standard. Should a CA issue such a subordinate CA, they too would bear the responsibility and liability for ensuring it is appropriately complying with the expectations, and failure to do may be reasonable grounds to begin a similar discussion, once the appropriate facts and details were gathered from the issuing CA.