Symantec: Update

[Gervase Markham]

Hi everyone,

Yesterday was May 8th, which was the day I had said we would stop
discussing my proposal of what to do about Symantec and hand it over to
Kathleen for a decision. This didn't happen for two reasons: I had some
personal things to deal with, and also I think the proposal needs some
modification.

Mozilla runs an open and transparent root program, and listens to the
voice of its community. And over the past few days it's been clear that
our community is not impressed with Symantec's engagement, or lack
thereof, with this process. I personally am also not impressed with the
way that getting information from Symantec feels like pulling teeth;
questions are answered at the last possible minute, and despite there
being major outstanding problems with compliance to Mozilla's root
program requirements (issue Y), no effort is made from their side to
proactively engage and start to resolve these issues. It is clear from
the issues list that there are a number of serious concerns, and these
are not being engaged with. Despite the fact that there appear to be
numerous under-audited and unaudited publicly-trusted sub-CAs out there,
and this fact has been known for weeks now, Symantec has not said
anything about the situation to Mozilla, either publicly or privately.
Would we find this acceptable in any other CA?

I am also not happy with simply waiting for the outcome of private
discussions between Google and Symantec in which Mozilla's interests are
not adequately represented. I am keen to move forward, to demonstrate
that delay is not rewarded, and (despite the fact that our process can
be slow) to make sure that timely action is taken based on the results
of our investigations. This is only fair, given that this is what we
have attempted to do for other CAs which we have investigated. We should
treat everyone the same, as far as we can.

I am therefore proposing the following:

* Editing the proposal to withdraw the "alternative" option, leaving
only the "new PKI" option. I no longer have confidence that the
alternative option represents an appropriate response. As some have
pointed out, the "documentation" requirement is actually something
Symantec should have done years ago as part of our intermediate
disclosure process, and which other CAs have made great efforts to
comply with already. The "new PKI" option represents the best way to
reduce the risk from Symantec's under-managed and sprawling existing PKI.

* Engagement here in m.d.s.p. with the community to refine and flesh out
the "new PKI" proposal, based on the Google outline but examining it and
enhancing it to make sure it is practical, both from an implementation
perspective and to reduce disruption to sites as far as possible.

* Discussions within Mozilla as necessary to make sure the appropriate
parts of the organization are briefed on this process.

* Submission of the proposal document to Kathleen at the earliest
possible moment to propose that we have that plan approved as our
requirements of Symantec. (The timeline here is dependent on other
moving parts, but as noted above, delay is to be avoided.)

We may in parallel ask further questions of Symantec, and expect timely
answers (as this is a baseline requirement for participation in our root
program), but this process will not wait around for those answers.

I will begin work on these tasks tomorrow.

Gerv

[Vincent Lynch]

Hi Gervase,

Thank you for the update on Mozilla's process.

I have one question regarding your wording. You write"I am therefore *proposing
*the following," and then you list your changes.

Does this mean that the "alternative" option is officially, 100%, off the
table? Or is this still an option Kathleen is considering?

-Vincent

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>



--
Vincent Lynch

[Gervase Markham]

On 09/05/17 17:29, Vincent Lynch wrote:
> I have one question regarding your wording. You write"I am therefore *proposing
> *the following," and then you list your changes.
>
> Does this mean that the "alternative" option is officially, 100%, off the
> table? Or is this still an option Kathleen is considering?

I apologise for overloading the word "proposing". :-)

My message outlines what I plan to do, and therefore what will be
presented by Kathleen. She will still have the option to order me to go
back to the other option, or write a third option, or do nothing, or
something else :-)

So the alternative option will not be in the paper I present to her.
That's what I'm saying.

Gerv

[Kurt Roeckx]

On Tue, May 09, 2017 at 04:51:12PM +0100, Gervase Markham via dev-security-policy wrote:
> Despite the fact that there appear to be
> numerous under-audited and unaudited publicly-trusted sub-CAs out there,
> and this fact has been known for weeks now, Symantec has not said
> anything about the situation to Mozilla, either publicly or privately.
> Would we find this acceptable in any other CA?

Do we somewhere have the official templates being used to send
reminders of the audit requirements? Kathleen posts a summary of
the email that got send, but I'm not sure if they contain more
text or if the text changes as the period gets longer.

>From the draft templates I could find, I suggest we skip the
first one because it's about being late and there are no audit
reports here. The second template would file a removal bug and start
discussing it here.

Instead of the removal of the roots, I suggest we either ask them
to revoke all the intermediate CAs that do not have the required
audits or that Mozilla adds them to OneCRL.

Did someone try to make a list of all CA certificates that don't
have all the required audit requirements marked in the common CA
database, including other CAs? We really should do this for all
such cases.


Kurt

[Kathleen Wilson]

On Tuesday, May 9, 2017 at 10:03:53 AM UTC-7, Kurt Roeckx wrote:
>
> Do we somewhere have the official templates being used to send
> reminders of the audit requirements?

Unofficial templates:
https://wiki.mozilla.org/CA:Email_templates

The official templates are in Salesforce, but currently match the wiki page.

> Kathleen posts a summary of
> the email that got send, but I'm not sure if they contain more
> text or if the text changes as the period gets longer.

For directly included certs, the email changes as the period gets longer.

So far we only have one email template for intermediate certs:
https://wiki.mozilla.org/CA:Email_templates#Disclosure_Incomplete_Email_Template

I have not yet created automation around notifying CAs of overdue audit statements for intermediate certs.

Kathleen

[Gervase Markham]

On 09/05/17 16:51, Gervase Markham wrote:
> * Editing the proposal to withdraw the "alternative" option, leaving
> only the "new PKI" option.

This has now been done:

https://docs.google.com/document/d/1RhDcwbMeqgE2Cb5e6xaPq-lUPmatQZwx3Sn2NPz9jF8/edit#

> * Engagement here in m.d.s.p. with the community to refine and flesh out
> the "new PKI" proposal, based on the Google outline but examining it and
> enhancing it to make sure it is practical, both from an implementation
> perspective and to reduce disruption to sites as far as possible.

I would appreciate people's comments on the details of the current draft.

Gerv

[Itzhak Daniel]

The next step, if Symantec wish to continue to use their current PKI in the future, should be logging (ASAP) *all* of the certificates they issued to a CT log, then we'll know how deep is the rabbit hole.

[okaphone.e...@gmail.com]

Makes sense to me.

But it does seem to assume that Symantec will cooperate with this. What happens if they decide not to is less clear. Perhaps it would be a good idea to indicate which steps will be taken in any case?

[mono...@gmail.com]

On Wednesday, May 10, 2017 at 7:59:37 PM UTC+2, Itzhak Daniel wrote:
> The next step, if Symantec wish to continue to use their current PKI in the future, should be logging (ASAP) *all* of the certificates they issued to a CT log, then we'll know how deep is the rabbit hole.

already the case since '15

https://security.googleblog.com/2015/10/sustaining-digital-certificate-security.html

although I'm not certain if this applied only to certs issued under the Symantec brand.

[Kurt Roeckx]

On Tue, May 09, 2017 at 07:03:16PM +0200, Kurt Roeckx via dev-security-policy wrote:
>
> Instead of the removal of the roots, I suggest we either ask them
> to revoke all the intermediate CAs that do not have the required
> audits or that Mozilla adds them to OneCRL.

Just to clarify, I believe that under 4.9.1.2 of the BRs, either
point 5, 8 or 9, Symantec is required to revoke those certificates
within 7 days. There is no indication that they follow the BR
requirements, the audit report even says that Symantec does not
control them, just monitor them. They are a clear danger.


Kurt

[Andrew R. Whalley]

On Wed, May 10, 2017 at 2:06 PM, mono.riot--- via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On Wednesday, May 10, 2017 at 7:59:37 PM UTC+2, Itzhak Daniel wrote:

> already the case since '15
>
> https://security.googleblog.com/2015/10/sustaining-
> digital-certificate-security.html

The blog post is dated October 15, but the requirement* only came into
effect June 1st, 2016

> although I'm not certain if this applied only to certs issued under the
> Symantec brand.

Any certs issued by any Symantec CA, regardless of brand, unless the CA is
operated by a 3rd party under its own, separate, audit.

Andrew

*required for the cert to be trusted in Chrome. They are still free to
issue certs that don't comply with the Chrome CT Policy, but those will
cause an interstitial warning in Chrome.

[tmcque...@gmail.com]

Symantec, in previous blog posts on their site, has indicated that they will support their customers [1].

That said, it is fair point that the plan should spell out what happens if symantec does not cooperate. It seems appropriate to have the plan do what it says -- scheduled phase out of the old roots -- with the same timescale. If symantec does not step up to fill their customer needs, I am sure one or more of their competitors will [and remember all this only applies to symantec customers who need publicly trusted certs... one big appeal of the proposal is that non-public uses can remain unaffected].

As the recent Wosign/Startcom experiences teaches, though, if the CA is not cooperative, it is very important for the browsers to step in with messaging. Not sure what form this would take, since most developers I know do not use beta/nightly versions of browsers, so it would need to be something in actual releases. Perhaps a single line with orange background just below URL box that says "in one month, this site will cease to be trusted by major browsers [click here for why]", or somesuch. With the link being very clear: it is the owner of the website that needs to update their certificate.

Just a thought.

1. https://www.symantec.com/connect/blogs/message-our-ca-customers "In the event Google implements its proposal, Symantec will ensure your websites, webservers or web applications continue to work across browsers."

[uri...@gmail.com]

Possibly this is irrelevant, but I have some concerns on how Symantec, it seems to me, is willfully mischaracterizing their certificate compliance issues in their prepared remarks to their investors yesterday.[1]

It makes it sound as if there are some generic certificate industry changes that are coming that might affect them. They do not seem willing to accept public responsibility for their actions and compliance failures.

"As you may be aware, in late March, Google put forth a proposal that, if implemented, would introduce major changes to the processes and operations that are standard across our industry, including our Certificate Authority business. Since that time, we've been engaged in conversations with Google, Mozilla, and other members of the CA community to seek input on our counter proposal that we believe minimizes business disruption for our customers and improves trust in Symantec's CA business. We believe we will find a mutually agreeable path forward that is in the best interest of our customers, and we expect discussions around our proposal to continue and have factored our current expectations around this headwind into our financial outlook."

[1] http://s1.q4cdn.com/585930769/files/doc_financials/2017/Q4/Symantec-4Q17-Prepared-Remarks.pdf

[Jonathan Rudenberg]

> On May 10, 2017, at 11:52, Gervase Markham via dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
>
> I would appreciate people's comments on the details of the current draft.

I don’t think that this proposal goes far enough.

Symantec has demonstrated that they have no interested in engaging with the Mozilla community about these issues. Over the past months, dozens of relevant and important questions have been asked of Symantec by community members, and most of them remain unanswered to this day. In most cases, when questions were answered, it was only after setting a deadline, at the last possible moment of that deadline, and in a format that made it very hard to track responses and ask follow-up questions.

Given this lack of constructive engagement, the recent request that we “pause” making any decisions, and the breathtaking severity of the issues discovered, I believe that the only objective should be to minimize risk to users of the Mozilla root store by removing the Symantec roots as quickly as possible. Trusted roots are a privilege and a responsibility, not a right, and Symantec has demonstrated that they are not capable of fulfilling that responsibility at this time.

With that in mind and taking into account the responses to previous incidents, I believe the following actions should be taken as part of the proposed ‘new PKI’ plan:

1) Immediate removal of EV treatment from all certificates issued by existing Symantec roots.

2) The establishment of a cutoff date a few months from now after which new certificates issued from existing Symantec roots will no longer be trusted based on notBefore. A variant of this is already in the proposal, but the timeline is unclear.

3) Complete removal of existing Symantec roots from the trust store as quickly as possible while limiting user impact, using the Chrome accelerated expiry proposal as a starting point.

Jonathan

[Gervase Markham]

On 11/05/17 13:02, wiz...@ida.net wrote:
> That said, it is fair point that the plan should spell out what happens if symantec does not cooperate.

I think we should cross that bridge when we come to it, which I hope we
won't. Not that I'm not prepared to cross it, but there's no point
devising plans and writing text in advance for a situation which can be
dealt with when and if it occurs.

Gerv

[okaphone.e...@gmail.com]

Better keep your deadlines short then. They seem to be the only times Symantec actually responds to anything asked/said here. :-(

CU Hans

[Michael Casadevall]

On 05/11/2017 09:53 AM, Jonathan Rudenberg via dev-security-policy wrote:
>
>> On May 10, 2017, at 11:52, Gervase Markham via dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
>>
>> I would appreciate people's comments on the details of the current draft.
>
> I don’t think that this proposal goes far enough.

First post on the list but long time lurker, but I feel the need to
weigh in here that I think Jonathah's proposal is much closer to what
has to happen.

Reading through Gervase's document, I'd like to add the following to
this in addition to the existing notes in PKI operations:

- EV certificate roots loose their trust bits effective immediately
(I don't think this can be done via OneCRL so it would be via the
next release)
- Any root stores (new or old) operated by Symantec shall require all
certificates to be posted to a CT log.
- Within three months, require all certificates issued from Symantec to
have SCT embedded in the end point certificate, and mandate this from
the beginning for any root certificates.
- NSS shall only accept certificates with the embedded SCT record in
the certificate.

Certificate transparency was the only way we began to get a real look at
how bad some of these issues are, and I feel that if we're going to
actually continue with Symatec as a CA, then we're going to make
absolutely sure we know how certificates are being utilized.

Mandating the X509v3 extension for TLS certificates means that
downstream servers don't have to be updated for CT awareness, and we
should never be in a case where a Mozilla product is accepting a
certificate that we can't independent review at a further point via the
CT logs. It should also prevent an undisclosed intermediately from being
undetected (as we've seen with Issue Y).


I'd also like to add the following to the transition plans:
- Limit certificate expiration to nine months from the existing roots
for new certificates.
- The above SCT requirement shall come into affect for the old roots no
less that three months from the date the proposal is ratified.
- Create a whitelist of intermediate certificates from the root that
can continue issuing certificates, but cutting off RAs after an initial
six month time period
- Require that Symantec reapply to the root program for a new DV and EV
root certificates, and begin the migration here. Once the new roots are
approved, then they can cross-sign from the old roots to the new ones.

My thought process here is to try and keep impact on WebPKI a minimum,
while making sure that we can externally audit how Symantec is using
their root store for certificates that will be trusted by Mozilla.

I'm concerned that spinning up new roots and having them be in the most
common root stores is going to take a significant period of time and
during that window we're still stuck with the old roots being in
operation. By limiting the branches of the old roots, it should limit
our risk while the new roots come into existence and begin to spread
through the ecosystem.

Winding down the old roots (phase four as described in the proposal) is
going to be a long and slow process so I want to make sure we're making
sure that while we're in the transition period that we've got an
extremely clear picture on what's going on on both sets of roots.

My problem with the Google "sliding scale" is that's its damn hard to
understand when exactly a certificate is good or when it expires since
the dates in the X509 certificate don't necessarily correspond with
reality. By simply capping Symantec certificates to nine months, it puts
us in a position that moving to a new DV/EV root would be required for
them to remain competitive while not drastically affecting the ecosystem
as a whole.

Maybe I'm off-kilter here, but I think this proposal would help keep
impact on WebPKI to a minimum but light a fairly serious fire to get
users moved to the new root stores ASAP. Please let me know if I am
seriously off base with my understanding of the situation or the
technologies involved; WebPKI is a complicated thing to understand :)
Michael

[Jakob Bohm]

On 13/05/2017 12:27, Michael Casadevall wrote:
> On 05/11/2017 09:53 AM, Jonathan Rudenberg via dev-security-policy wrote:
>>
>>> On May 10, 2017, at 11:52, Gervase Markham via dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
>>>
>>> I would appreciate people's comments on the details of the current draft.
>>
>> I don’t think that this proposal goes far enough.
>
> First post on the list but long time lurker, but I feel the need to
> weigh in here that I think Jonathah's proposal is much closer to what
> has to happen.
>

I suspect you may be believing too much of what Google says, see
specific problems below.

> Reading through Gervase's document, I'd like to add the following to
> this in addition to the existing notes in PKI operations:
>
> - EV certificate roots loose their trust bits effective immediately
> (I don't think this can be done via OneCRL so it would be via the
> next release)
> - Any root stores (new or old) operated by Symantec shall require all
> certificates to be posted to a CT log.
> - Within three months, require all certificates issued from Symantec to
> have SCT embedded in the end point certificate, and mandate this from
> the beginning for any root certificates.
> - NSS shall only accept certificates with the embedded SCT record in
> the certificate.

This won't work for the *millions* of legitimate, not-misissued, end
certificates that were issued before Symantec began SCT embedding
(hopefully in the past) and haven't expired before such an early
deadline.

Also, since both Mozilla and Debian-derived systems such as Ubuntu use
the the Mozilla store as the basis for S/MIME checking, it is worth
noting that CT-logging of S/MIME end certs under the current Google-
dominated specifications is a guaranteed spam disaster, as it would
publish all the embedded e-mail addresses for easy harvesting.

>
> Certificate transparency was the only way we began to get a real look at
> how bad some of these issues are, and I feel that if we're going to
> actually continue with Symatec as a CA, then we're going to make
> absolutely sure we know how certificates are being utilized.
>

That is unfortunately true.

> Mandating the X509v3 extension for TLS certificates means that
> downstream servers don't have to be updated for CT awareness, and we
> should never be in a case where a Mozilla product is accepting a
> certificate that we can't independent review at a further point via the
> CT logs. It should also prevent an undisclosed intermediately from being
> undetected (as we've seen with Issue Y).
>

However it would mandate that they be updated with new certificates
instead. A lot easier, but still a mountain not easily moved.

>
> I'd also like to add the following to the transition plans:
> - Limit certificate expiration to nine months from the existing roots
> for new certificates.

I strongly believe the "9 month" rule mysteriously proposed but never
explained by Google was designed specifically to make buying certs from
Symantec all but worthless, chasing away all their customers. People
*paying* for certificates generally don't want to buy from anyone
selling in increments of less than 1 year, preferably 2 or 3. "9
months" is an especially bad duration, as it means the renewal dates
and number of renewals per fiscal year will fluctuate wildly from an
accounting perspective.

> - The above SCT requirement shall come into affect for the old roots no
> less that three months from the date the proposal is ratified.
> - Create a whitelist of intermediate certificates from the root that
> can continue issuing certificates, but cutting off RAs after an initial
> six month time period

Are there any RA's left for Symantec?

> - Require that Symantec reapply to the root program for a new DV and EV
> root certificates, and begin the migration here. Once the new roots are
> approved, then they can cross-sign from the old roots to the new ones.
>
> My thought process here is to try and keep impact on WebPKI a minimum,
> while making sure that we can externally audit how Symantec is using
> their root store for certificates that will be trusted by Mozilla.
>
> I'm concerned that spinning up new roots and having them be in the most
> common root stores is going to take a significant period of time and
> during that window we're still stuck with the old roots being in
> operation. By limiting the branches of the old roots, it should limit
> our risk while the new roots come into existence and begin to spread
> through the ecosystem.

I believe the only reasonable interpretation of the "new root" plan
would be based on cross signing for trust by old Mozilla browsers and
other root programs.

>
> Winding down the old roots (phase four as described in the proposal) is
> going to be a long and slow process so I want to make sure we're making
> sure that while we're in the transition period that we've got an
> extremely clear picture on what's going on on both sets of roots.
>
> My problem with the Google "sliding scale" is that's its damn hard to
> understand when exactly a certificate is good or when it expires since
> the dates in the X509 certificate don't necessarily correspond with
> reality. By simply capping Symantec certificates to nine months, it puts
> us in a position that moving to a new DV/EV root would be required for
> them to remain competitive while not drastically affecting the ecosystem
> as a whole.
>

For starters, we can demand that Symantec never issues misdated
certificates lest they be thrown out instantly. So far the only
misdatings known were a very few that were well documented with very
good technical excuses.

Forcing the stand-up of new roots operated by competent management and
staff is indeed good, but to do that safely (i.e. without false
security theatrics and same old flaws) would require that Symantec
spends several months to get their house in order before generating
the new root keys.

> Maybe I'm off-kilter here, but I think this proposal would help keep
> impact on WebPKI to a minimum but light a fairly serious fire to get
> users moved to the new root stores ASAP. Please let me know if I am
> seriously off base with my understanding of the situation or the
> technologies involved; WebPKI is a complicated thing to understand :)
>

The fundamental questions are:

Should we light a fire under the sloppy Symantec management or under
their innocent users?

And should we allow enough of Symantec to keep standing to not leave
those users who cannot switch stranded with nowhere to go but a 404
webpage?



Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

[Michael Casadevall]

On 05/15/2017 09:32 AM, Jakob Bohm wrote:
> This won't work for the *millions* of legitimate, not-misissued,
> end certificates that were issued before Symantec began SCT
> embedding (hopefully in the past) and haven't expired before such
> an early deadline.
>

Sorry, I could have been more clear here. What I'm proposing is that
after a specific TBD NotBefore date, we require SCTs to be in place on
the certificate to be trusted. Certificates from before that date
would remain trusted as-is (pending any reduction of expiration time).

I don't know if NSS has support for checking of SCTs (I can't pull the
source at the moment to check), but it should fail if the SCT is
missing, and otherwise behave like OCSP validation.

> Also, since both Mozilla and Debian-derived systems such as Ubuntu
> use the the Mozilla store as the basis for S/MIME checking, it is
> worth noting that CT-logging of S/MIME end certs under the current
> Google- dominated specifications is a guaranteed spam disaster, as
> it would publish all the embedded e-mail addresses for easy
> harvesting.
>

I didn't consider the S/MIME use case here. A brief look at the root
store I'd be fine with the SCT restriction only applying when looking
at CKA_TRUST_SERVER_AUTH, and not in other cases. Looking at certdata,
it looks like at least some of the current Verisign/Symantec roots
have both the S/MIME and server auth bits enabled.

While I feel CT would be a nice thing for S/MIME, unfortunately, I
have to agree with this point that we don't need to make spammers
lives easier. That being said, part of me wonders if there would be
other undisclosed intermediates if one could easily evaluate S/MIME
issuances ...

>> Mandating the X509v3 extension for TLS certificates means that
>> downstream servers don't have to be updated for CT awareness, and
>> we should never be in a case where a Mozilla product is accepting
>> a certificate that we can't independent review at a further point
>> via the CT logs. It should also prevent an undisclosed
>> intermediately from being undetected (as we've seen with Issue
>> Y).
>>
>
> However it would mandate that they be updated with new
> certificates instead. A lot easier, but still a mountain not
> easily moved.

See above on NotBefore.

>>
>> I'd also like to add the following to the transition plans: -
>> Limit certificate expiration to nine months from the existing
>> roots for new certificates.
>
> I strongly believe the "9 month" rule mysteriously proposed but
> never explained by Google was designed specifically to make buying
> certs from Symantec all but worthless, chasing away all their
> customers. People *paying* for certificates generally don't want
> to buy from anyone selling in increments of less than 1 year,
> preferably 2 or 3. "9 months" is an especially bad duration, as it
> means the renewal dates and number of renewals per fiscal year will
> fluctuate wildly from an accounting perspective.
>

I can see the point here, but I'm not sure I agree. Every time we keep
digging, we keep finding more and more problems with these roots.
WebPKI depends on all certificates in the root store being
trustworthy, and Symantec as a whole has not exactly shown themselves
to be responsive or willing to communicate publicly on the various
issues brought up on the list.

There's a decent argument to be had to simply disallow new issuance
from the existing roots and allow the current certificates to age out
(in which case imposing SCT embedded as I propose is simple), but I'm
not sure we've gotten a complete picture of how far this rabbit hole goe
s.

There's been a continual pattern of "this is everything", and then we
find another bunch of misissued certificates/undisclosed subCAs/etc.
Can we honestly say that we're comfortable with allowing these roots
to still be active at all?

>> - The above SCT requirement shall come into affect for the old
>> roots no less that three months from the date the proposal is
>> ratified. - Create a whitelist of intermediate certificates from
>> the root that can continue issuing certificates, but cutting off
>> RAs after an initial six month time period
>
> Are there any RA's left for Symantec?
>

TBH, I'm not sure. I think Gervase asked for clarification on this
point, but its hard to keep track of who could issue as an RA. I know
quite a few got killed, but I'm not sure if there are any other subCAs
based off re-reading posts in this thread.

>> - Require that Symantec reapply to the root program for a new DV
>> and EV root certificates, and begin the migration here. Once the
>> new roots are approved, then they can cross-sign from the old
>> roots to the new ones.
>>
>> My thought process here is to try and keep impact on WebPKI a
>> minimum, while making sure that we can externally audit how
>> Symantec is using their root store for certificates that will be
>> trusted by Mozilla.
>>
>> I'm concerned that spinning up new roots and having them be in
>> the most common root stores is going to take a significant period
>> of time and during that window we're still stuck with the old
>> roots being in operation. By limiting the branches of the old
>> roots, it should limit our risk while the new roots come into
>> existence and begin to spread through the ecosystem.
>
> I believe the only reasonable interpretation of the "new root"
> plan would be based on cross signing for trust by old Mozilla
> browsers and other root programs.
>

Won't the cross signature though have to be embedded in Firefox, or
included in a server's SSL bundle for it to actually work?

Older Firefox/NSS won't have the whitelisting provisions in it, so
they'd accept the cross-signature if they're presented with it in the
certificate bundle.

I realize that as I write this though that this might be a technical
impossibility (or at least exceedingly painful) to do so that might
kill this part of my proposal as a matter of course.

> For starters, we can demand that Symantec never issues misdated
> certificates lest they be thrown out instantly. So far the only
> misdatings known were a very few that were well documented with
> very good technical excuses.
>
> Forcing the stand-up of new roots operated by competent management
> and staff is indeed good, but to do that safely (i.e. without
> false security theatrics and same old flaws) would require that
> Symantec spends several months to get their house in order before
> generating the new root keys.
>

Which leaves us in the ugly situation of trusting roots which have
serious trust questions on it. I agree that any case of backdating to
evade a technical control is grounds for the CA death penalty
regardless of customer fallout.

Looking at the points you bring up though, I think we can deal with
the issue as a whole as long as CT is enforced, and that we can
independent confirm that enforcement via embedded SCT since frankly,
I'm not inclined to trust Symantec saying "yes, this is everything"
with regards to certificates they issued.

>
> The fundamental questions are:
>
> Should we light a fire under the sloppy Symantec management or
> under their innocent users?
>

Honestly, part of me is thinking the only reason a full distrust is
not being discussed is because Symantec is "too big to fail" in the
WebPKI world. I don't like the idea of breaking end-users at all, but
I also dislike the idea of weaking trust in the PKI system simply
because the impact of doing something is too big.

On some level, I do think a fire has to be lit. We won't be stuck in
this situation otherwise. How big and how severe is TBDed

> And should we allow enough of Symantec to keep standing to not
> leave those users who cannot switch stranded with nowhere to go but
> a 404 webpage?
>

Removing the EV trust bits at least has the effect of only making the
green bar go away. I think a large part of this comes down to the
question if the RAs could issue EV certificates or not which was still
pending last I checked.

However, going a step beyond that, EV represents that a CA has looked
in-depth to make sure a cert is going to the right person. For example
during the period the cross-signature to the federal PKI was in place,
any fPKI CA could have issued a certificate that would have been green
barred by Mozilla. I can't say with any confidence than any EV
certificate issued by Symantec as things stand actually has had that
check and balances.

We might be able to limit the damage here if we could determine what
was checked by who, and scoping EV removal based on that, but I'm not
even sure we have a clear picture of who could sign with what at the
moment. Short of independently re-validating EVERY certificate, I
don't see a good solution for solving this though.

That's not great for Symantec's customers, but its also far better
than a broken website. It also lights a fairly huge fire on them to
get new roots spun, and then allow for reissues from the old roots to
the new.

My 2 cents,
Michael

[Jakob Bohm]

On 15/05/2017 22:06, Michael Casadevall wrote:
> On 05/15/2017 09:32 AM, Jakob Bohm wrote:
>> This won't work for the *millions* of legitimate, not-misissued,
>> end certificates that were issued before Symantec began SCT
>> embedding (hopefully in the past) and haven't expired before such
>> an early deadline.
>>
>
> Sorry, I could have been more clear here. What I'm proposing is that
> after a specific TBD NotBefore date, we require SCTs to be in place on
> the certificate to be trusted. Certificates from before that date
> would remain trusted as-is (pending any reduction of expiration time).
>

Ok, that's much better.

Yes, that seems to be the trend. But it has nothing to do with if the
"9 month" rule or some other measures are the best remedy.

> There's a decent argument to be had to simply disallow new issuance
> from the existing roots and allow the current certificates to age out
> (in which case imposing SCT embedded as I propose is simple), but I'm
> not sure we've gotten a complete picture of how far this rabbit hole goe
> s.
>

That wouldn't work, see below.

> There's been a continual pattern of "this is everything", and then we
> find another bunch of misissued certificates/undisclosed subCAs/etc.
> Can we honestly say that we're comfortable with allowing these roots
> to still be active at all?
>
>>> - The above SCT requirement shall come into affect for the old
>>> roots no less that three months from the date the proposal is
>>> ratified. - Create a whitelist of intermediate certificates from
>>> the root that can continue issuing certificates, but cutting off
>>> RAs after an initial six month time period
>>
>> Are there any RA's left for Symantec?
>>
>
> TBH, I'm not sure. I think Gervase asked for clarification on this
> point, but its hard to keep track of who could issue as an RA. I know
> quite a few got killed, but I'm not sure if there are any other subCAs
> based off re-reading posts in this thread.
>

RAs (external companies that can decide if Symantec itself should issue
a cert) are completely different from external SubCAs (external
companies that have their own CA and a certificate chain back to a
Symantec root), which are different from internal SubCAs (CA
certificates for Symantec controlled keys, such as the SubCA that
signed normal OV certificates issued in January 2017).

External and Internal SubCAs can be blocked by simple technical means
via TrustDB and OneCRL manipulations. RAs are indistinguishable from
Symantec itself when checking certificates, because the certificates
were in fact signed by Symantec itself.

The standard way this is done is that the old roots (which are trusted
by old browsers) cross-sign the new roots or the subCAs of the new
roots. People buying new certs get (as always) a new cert chain for
their new cert, which contains enough data to pass in browsers that
trust new root, trust the old root, or trust both roots.

Servers with old certs still return the old certificate chain that
leads to the old roots.

Other measures (such as browser embedded SubCA/cross certs) can be used
to reduce how much of the old CA tree Firefox/Thunderbird trust during
the transition.

I think we will need to look separately at two very different issues:

1. Symantec's PKI and the location of the EV trust bit unfortunately
allows non-EV SubCAs to issue EV certs that Firefox marks as green.
This same issue applies to most Mozilla trusted roots, because
Mozilla implemented the EV trust at the root CA level rather than at
a SubCA level.

This can be fixed technically by restricting the EV trust to the
SubCAs that are supposed to issue EV certs, rather than to whatever
general WebPKI root cert resides above it (in order for legitimate EV
certs to be trusted as normal certs by old browsers).

2. If there were any significant failures in the validation of EV
certs signed directly by the dedicated EV SubCAs at Symantec (other
than the one test cert that got some Symantec people fired some time
ago).

> We might be able to limit the damage here if we could determine what
> was checked by who, and scoping EV removal based on that, but I'm not
> even sure we have a clear picture of who could sign with what at the
> moment. Short of independently re-validating EVERY certificate, I
> don't see a good solution for solving this though.
>
> That's not great for Symantec's customers, but its also far better
> than a broken website. It also lights a fairly huge fire on them to
> get new roots spun, and then allow for reissues from the old roots to
> the new.
>
> My 2 cents,
> Michael
>

[Michael Casadevall]

On 05/15/2017 06:05 PM, Jakob Bohm wrote:
>
> Ok, that's much better.
>

Yay for reasonable courses of action. We'll see if it goes into the next
proposal.

>> I can see the point here, but I'm not sure I agree. Every time we keep
>> digging, we keep finding more and more problems with these roots.
>> WebPKI depends on all certificates in the root store being
>> trustworthy, and Symantec as a whole has not exactly shown themselves
>> to be responsive or willing to communicate publicly on the various
>> issues brought up on the list.
>>
>
> Yes, that seems to be the trend. But it has nothing to do with if the
> "9 month" rule or some other measures are the best remedy.
>

There might be a reasonable compromise here, see below.

> RAs (external companies that can decide if Symantec itself should issue
> a cert) are completely different from external SubCAs (external
> companies that have their own CA and a certificate chain back to a
> Symantec root), which are different from internal SubCAs (CA
> certificates for Symantec controlled keys, such as the SubCA that
> signed normal OV certificates issued in January 2017).
>
> External and Internal SubCAs can be blocked by simple technical means
> via TrustDB and OneCRL manipulations. RAs are indistinguishable from
> Symantec itself when checking certificates, because the certificates
> were in fact signed by Symantec itself.
>

I thought the RAs were being issued off their own intermediate branches
and not off Symantec ones. Rechecking crt.sh "C=KR, O=CrossCert,
OU=AccreditedCA, CN=CrossCert Class 1 Server CA" is a separate
intermediate chaining to KISA so I done goofed here. Oops.

Re-reading the issues, I think I got crossed between subCAs missing
audits, and RA issuance.

> The standard way this is done is that the old roots (which are trusted
> by old browsers) cross-sign the new roots or the subCAs of the new
> roots. People buying new certs get (as always) a new cert chain for
> their new cert, which contains enough data to pass in browsers that
> trust new root, trust the old root, or trust both roots.
>
> Servers with old certs still return the old certificate chain that
> leads to the old roots.
>
> Other measures (such as browser embedded SubCA/cross certs) can be used
> to reduce how much of the old CA tree Firefox/Thunderbird trust during
> the transition.
>

Thanks for clearing this up.

> I think we will need to look separately at two very different issues:
>
> 1. Symantec's PKI and the location of the EV trust bit unfortunately
> allows non-EV SubCAs to issue EV certs that Firefox marks as green.
> This same issue applies to most Mozilla trusted roots, because
> Mozilla implemented the EV trust at the root CA level rather than at
> a SubCA level.
>
> This can be fixed technically by restricting the EV trust to the
> SubCAs that are supposed to issue EV certs, rather than to whatever
> general WebPKI root cert resides above it (in order for legitimate EV
> certs to be trusted as normal certs by old browsers).
>

This is a start.

We'd need confirmation from Symantec which subCAs are supposed to be
able to sign EV as I don't believe we have a complete list. That's also
assuming that things are that nice and organized (which is far from a
given). If we can successfully cut a good chunk of the crud away, it
would at least help in mind keeping EV being a reasonable option.

> 2. If there were any significant failures in the validation of EV
> certs signed directly by the dedicated EV SubCAs at Symantec (other
> than the one test cert that got some Symantec people fired some time
> ago).
>

Can we reasonably determine we're good here beyond a reasonable doubt?

The current responses to the last question found new parts of the fPKI
that are chaining via the Issue Y intermediates that appear that they
would be trusted by Mozilla. We also have confirmation that some of the
RAs could issue EVs and could validate certificates.

Symantec said that they independently checked the non-expired EV
certificates, but I think I have (IMHO) justified concerns there might
be additional ones here we don't know about.

Given the number of unknown knows with the EV situation right now, I
think I'm going to wait for more information before pushing any one
specific option, but at a minimium, we need to cut down the parts of the
tree that can sign for EV.

There's also a third issue is "what is the correct response to the
severity of Symantec's trust issues". We're fairly close to CNNIC
territory here, since we've got multiple intermediates that chain to the
Mozilla roots and can issue certificates which are either not BR
compliant, or out of scope of an audit.

In an attempt to try and get this thread to a point where the powers
that be might choose to include it in their proposal, let's try the
following in the addition to what is under PKI concerns in the gdoc:

---
- For any Symantec-owned root certificate in the Mozilla trust program
with the CKA_TRUST_SERVER_AUTH bit enabled, all leaf certificates in the
chain shall contain embedded SCT information. This shall be required for

the certificate to be trusted.

(or in plain English: web certificates need both CT and SCT embedded.
S/MIME certificates are alright without. We separate these into two
separate chains of trust so they can't accidentally intermix.).

- For the purposes of CKA_TRUST_EMAIL_PROTECTION, all CA and subCA
certificates in the chain of trust must have embedded SCT information to
be considered trusted. The leaf certificate shall be exempt from this
requirement.

- A three-day grace period shall be in place from the issuance date of
a certificate to when it must be in the CT logs for validation reasons
(this is in line with other proposals here).

- Certificates issued for server authentication without SCT information
being timely submitted shall be considered equivalent to a misissue.
Exceptions to this policy for legacy customers may be granted on a
case-by-case basis.

- All server authentication certificates shall be submitted to at least
two public CT logs.

- Certificates issued before the effective date shall continue to be
trusted until their expiration/revocation. These certificates shall be
exempt from the embedded SCT requirement unless reissued.

- Symantec shall identify (if possible) subCAs are supposed to be used
for issuing EV certificates. The EV trust bit shall be removed from the
root, and added to the intermediate certificates.

- Certificates issued from the pre-existing roots after the effective
date shall be limited to a maximum lifetime of nine months.

---

This revised proposal achieves the following
- Mozilla never trusts a certificate we can't independently audit and
verify via the CT logs. This also slams the door shut on Issue Y of ever
becoming an issue again since we'd immediately be able to spot it before
the issue snowballs.

- S/MIME and ServerAuth is clearly separated. No chance of a nasty
surprise from an undisclosed CA that has only been issuing S/MIME
certificates but could issue for SA due to a lack of EKUs (since S/MIME
intermediates wouldn't show up in the CT logs as far as I know).

- In the S/MIME case, while we can't drop the entire load into CT to
find undisclosed intermediate certificates, we can mandate that the
embedded SCT for those subCAs has to be there. That should force all the
intermediate certificates that can sign for S/MIME into the logs so we
can at least get *some* idea of the situation here. It at least helps to
ensure that S/MIME intermediates are being disclosed, and we're only
trusting what is being disclosed. I don't know if we can reasonably do
this for the old roots though short of hardcoding all the SCTs into NSS
which is likely undesirable.

- The SCT exemption policy allows for legacy customers to get
certificates for devices that might choke on them (since I've seen some
pretty ugly embedded TLS stacks). We will never trust a certificate
meant for this scenario, keeping in line with the above.

- The old root certificates can't issue for longer that nine months
past the effective date. That should light a serious fire to migrate
ASAP, but prevent existing certificates from being heavily impacted
except in the case of reissue (which should be reissued against the new
roots).

- For customers who pinned the old roots or can't migrate, issuance off
the old certificates remain possible while lighting a fire under them to
fix their infrastructure if they don't wish to reissue every nine months.

- It at least limits some of the potential issues with the current EV
mess. I'm still personally of the opinion that this might need to go
further but at least its a start.

---
Final thoughts:
As far as I can tell and based on my reading, Mozilla products only care
about SERVER_AUTH, CLIENT_AUTH, and EMAIL_PROTECTION, with the other
trust bits being legacy. For CLIENT_AUTH, it might be worth doing
something similar to the S/MIME provision, but I don't know enough to
determine the actual impact here.

While I think there's still discussion to be had on max life+EV, I think
the mandated SCT provisions would go a *long* way in helping restoring
trust in Symantec since it means any potential misissues could be
detected relatively easily, and we never trust anything that can't be
independently verified.

It also goes a good way at lighting the necessary fires under Symantec
management, though some part of me wonders if the max expiration should
be lower.

I realize this might be a fairly large technical burden to get
implemented, so I'm willing to pitch in some way to help if someone is
willing to mentor me.

As always, hoping to hear comments, and the TPTB are willing to consider
it and point out where I've been stupid.
Michael

[Michael Casadevall]

On 05/16/2017 03:50 AM, Michael Casadevall wrote:
> On 05/15/2017 06:05 PM, Jakob Bohm wrote:
>>
>
> - A three-day grace period shall be in place from the issuance date of
> a certificate to when it must be in the CT logs for validation reasons
> (this is in line with other proposals here).
>

> - All server authentication certificates shall be submitted to at least
> two public CT logs.
>

Just realized I had a brainfart when writing this. Don't believe I can
supersede on this list to fix it so sorry for the chatter.

This should say that certificates must be issued with an embedded SCT
which Symantec can get from their own log, and then upload the
certificate to other logs as part of the issuance.

As part of the CT validation, there would be a three day grace period
from the issuance date, to when the certificate can start failing due to
CT failure which should leave a nice bit of padding for the maximum
merge delay on the current public logs.
Michael

[Gervase Markham]

On 15/05/17 21:06, Michael Casadevall wrote:
> Sorry, I could have been more clear here. What I'm proposing is that
> after a specific TBD NotBefore date, we require SCTs to be in place on
> the certificate to be trusted. Certificates from before that date
> would remain trusted as-is (pending any reduction of expiration time).
>
> I don't know if NSS has support for checking of SCTs (I can't pull the
> source at the moment to check), but it should fail if the SCT is
> missing, and otherwise behave like OCSP validation.

Embedding SCTs is not the only way SCTs can be delivered - they can come
in the SSL handshake or via OCSP. Requiring them to be embedded does
have the advantage that certificates now carry an unforgeable timestamp,
and it was something I proposed in a version of Mozilla's now-dormant CT
policy. But for various reasons, it's not necessarily practical to
require it in all circumstances (which is why the CT RFC defines
multiple mechanisms).

Firefox does have some support for checking SCT presence and validity,
but it's not turned on.

>> Are there any RA's left for Symantec?
>
> TBH, I'm not sure. I think Gervase asked for clarification on this
> point, but its hard to keep track of who could issue as an RA. I know
> quite a few got killed, but I'm not sure if there are any other subCAs
> based off re-reading posts in this thread.

Symantec say they have closed their RA program, only Apple and Google
are left in their GeoRoot program, and they have no other programs which
allow third parties to have issuance capability.

>> I believe the only reasonable interpretation of the "new root"
>> plan would be based on cross signing for trust by old Mozilla
>> browsers and other root programs.
>
> Won't the cross signature though have to be embedded in Firefox, or
> included in a server's SSL bundle for it to actually work?

The latter, yes. This is not difficult nor that unusual.

Gerv

[Peter Bowen]

On Fri, May 19, 2017 at 7:25 AM, Gervase Markham via
dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
> On 15/05/17 21:06, Michael Casadevall wrote:
>

>>> Are there any RA's left for Symantec?
>>
>> TBH, I'm not sure. I think Gervase asked for clarification on this
>> point, but its hard to keep track of who could issue as an RA. I know
>> quite a few got killed, but I'm not sure if there are any other subCAs
>> based off re-reading posts in this thread.
>

> Symantec say they have closed their RA program, only Apple and Google
> are left in their GeoRoot program, and they have no other programs which
> allow third parties to have issuance capability.

This is not accurate. They have indicated that the SSP customers have
some level of issuance capability.

Thanks,
Peter

[Gervase Markham]

On 19/05/17 15:28, Peter Bowen wrote:
> This is not accurate. They have indicated that the SSP customers have
> some level of issuance capability.

Oops. Well, they said that a while back, but yes indeed, since then we
have discovered the above fact.

Gerv

[Michael Casadevall]

On 05/19/2017 10:25 AM, Gervase Markham wrote:
> Embedding SCTs is not the only way SCTs can be delivered - they can come
> in the SSL handshake or via OCSP. Requiring them to be embedded does
> have the advantage that certificates now carry an unforgeable timestamp,
> and it was something I proposed in a version of Mozilla's now-dormant CT
> policy. But for various reasons, it's not necessarily practical to
> require it in all circumstances (which is why the CT RFC defines
> multiple mechanisms).
>
> Firefox does have some support for checking SCT presence and validity,
> but it's not turned on.
>

My concern here is right now, we're trying to rebuild trust for
Symantec. We're very much in a "trust but verify" sorta thing, and I
don't think it's an unjustified requirement to do so. I think CT is
about the only thing that has allowed us to reasonably consider keeping
Symantec in the root store at all.

However, for Mozilla's purposes, is there a case where having a SCT in
certificate would either break something, or otherwise be undesirable?

Well, at least with the current state of webpki, mandating an embedded
SCT is probably not practical for everyone. I actually forgot about the
OCSP stapling mechanism for SCTs, though my concern here is not everyone
turns on OCSP stapling. Since both OCSP CT stapling and embedded SCTs
require that the cert be submitting to a log at issuance, part of me
wonders if the right middle ground is this:

As far as I know, I think Microsoft's IIS is the only major web server
that turns OCSP stapling on out of the box.

- By default, Symantec shall issue certificates with embedded SCTs
(soft-fail for failure to validate SCT information)

- If, due to customer demand, a certificate with an embedded SCT can
not be used, said certificate must get the SCT information by a stapled
OCSP response or via TLS extension to be trusted by Mozilla. (hard-fail)

This should cover the general case fairly well, and for the edge cases,
well either its for a special class of device that we don't care about,
or the customer has to do some work to get things working in Mozilla.

Or in other words, if there's a case where an embedded SCT can't fly
here, then we mandate that one of the other two validation options must
be present for things to fly. That being said, for my personal
knowledge, I'd love to know more on the real world practicalities of
embedding SCTs.

Thanks for your feedback.

>>> Are there any RA's left for Symantec?

Following up to this, the question that I should have asked is who can
technically do an issuance of certificates based on Symantec's roots.
SSP customers are a recent discovery. I wonder if there's anything else.
Michael

[Gervase Markham]

On 20/05/17 15:26, Michael Casadevall wrote:
> However, for Mozilla's purposes, is there a case where having a SCT in
> certificate would either break something, or otherwise be undesirable?

I believe we turned the checking on and discovered performance issues,
so we turned it off. I'm not sure if those have since been solved. JC?

> Well, at least with the current state of webpki, mandating an embedded
> SCT is probably not practical for everyone. I actually forgot about the
> OCSP stapling mechanism for SCTs, though my concern here is not everyone
> turns on OCSP stapling. Since both OCSP CT stapling and embedded SCTs
> require that the cert be submitting to a log at issuance,

That's not so. OCSP CT stapling doesn't require the cert be submitted to
a log at issuance. You only need to do it at some point before you
start using it. The same is true of the SSL handshake method.

> - By default, Symantec shall issue certificates with embedded SCTs
> (soft-fail for failure to validate SCT information)

Given that Chrome is requiring CT for all Symantec certificates, one
could argue there's minimal value in Mozilla coming up with its own
CT-related requirements, particularly as Mozilla has not (yet?) deployed
SCT checking in Firefox.

Gerv