Researcher Says API Flaw Exposed Symantec Certificates, Including Private Keys

1460 views
Skip to first unread message

uri...@gmail.com

unread,
Mar 28, 2017, 12:41:47 PM3/28/17
to mozilla-dev-s...@lists.mozilla.org

Nick Lamb

unread,
Mar 28, 2017, 3:46:05 PM3/28/17
to mozilla-dev-s...@lists.mozilla.org
In order for Symantec to reveal anybody's private keys they'd first need to have those keys, which is already, IIRC forbidden in the BRs. So even proof that Symantec routinely had these keys is a big deal. The whole reason things like CSR signing exist is that public CAs have no reason to know anybody's private key, the clue is in the name.

Beyond that I'll note that anybody reading Raymond Chen for a few weeks will learn that not every researcher who thinks they just found a smoking gun turns out to know what a gun actually is, nor sometimes what constitutes smoke. So, evidence is what we need. Neither holes in Symantec security nor overclaims from a person who misunderstood what they were seeing are unlikely.

uri...@gmail.com

unread,
Mar 28, 2017, 11:08:08 PM3/28/17
to mozilla-dev-s...@lists.mozilla.org
For what it's worth, this is the latest post on facebook from the researcher.
https://www.facebook.com/cbyrneiv/posts/10155129935452436

The private key storage issue sounds like a reseller tool, like
https://www.thesslstore.com/ssltools/csr-generator.php
and he found the private key was stored with the reseller, when he accessed the account.

Chris Byrne
March 24 at 8:02pm ·

UPDATED Tuesday March 28th, to clarify some language, and provide additional detail...
..........
Looks like I don't have to stay quiet about this one anymore...
I found out about this problem... and others related to it... back in early 2015. I warned Symantec about it, and worked through some of the issues with them.
At the end of it, they asked for a chance to verify how extensive the issue was, and to fix it. We both agreed it would take up to two years to fix, without creating more chaos and causing more damage... and I said yes.
I agreed to limited non-disclosure of the issue, unless I felt it was critically necessary, or it would be unethical or irresponsible for me not to disclose (for example, if there were a threat to national security, or I discovered a compromise of a client, or any actual criminal compromise arising from it, etc... etc...).
That said, I informed Krebs and a few others about what I had found... and also that I agreed to let them fix it, because it would be less damaging to the world, than exposure would be.
It was one of those issues where there's no GOOD choice... but I looked at the damage immediate exposure would cause, vs. a slowly rolling fix over two years...
I talked to some of my friends on the grey and black sides of things, and did my own checking, to see if there was any chatter about this anywhere, and I couldn't find any... and I concluded that more harm would be done if I disclosed.
Fellow security professional Bobby Kuzma helped me investigate and validate these issues, and he can verify what I state here.
I checked a few months later, and they HAD fixed several of the core problems... though not all of them... So I waited, and so did those who I had informed about the problem, and had verified it for themselves...
Unfortunately, late in 2015, my cancer recurred, and spread to my lymph nodes... and I've been fighting it ever since, so I haven't kept up with the issue.

So... Here's what the post doesn't mention...
If you purchased a Symantec certificate (or a cert from any of their associated subsidiaries and partners) through a third party, from at least as far back as early 2013 until recently; their third party certificate generation, management, and retrieval API allowed those certificates... including in some cases private keys generated by third parties... to be retrieved without proper authentication, or in some cases any authentication at all.
Unless the third party added proper security around it, all you had to do was click a link sent in email, and you could retrieve a cert, revoke a cert, and re-issue a cert... and for some third party vendors, depending on the type of certificate, and how the certificate was issued, you could also retrieve private keys.
Note, this was not just for server certificates for SSL, it also included other certificates, such as personal certificates used for email and other personal encryption. These certs could be requested entirely through the web interface without a separate server issued certificate request, generating both private and public keys, which would then only be protected by a passphrase.

Further, even with first party purchase, for some time in some web interfaces, it was possible for properly authenticated users to edit a URL, and at least retrieve information about first party certificates and their owners, and possibly the certs of others.
This is because third party data retrieval was not limited to certificates issued by that third party. It was available to anyone by entering the UID of any other user.
To clarify... at no time were we able to retrieve private keys directly from Symantec, only from third parties that issued certificates from a web interface without requiring a server generated certificate request.
We were able to confirm that these third party weaknesses extended to retreiving, revoking, and reissuing certificates, in some cases without notification to the legitimate certificate holder.
In one case, we were also able to reissue a certificate without first explicitly revoking it, and the original certificate continued working for at least 48 hours... We did not test further to see how long it would take for the certificate revocation list to propagate, and we're not pale to prove one way or the other if the organ certificate was revoked or not.

It is possible that the certificate was never revoked at all, and another certificate was simply issued, leaving two valid certificates one controlled by the original requestor, and one controlled by the unauthorized party. This would be the worst case scenario.
It is also possible that the original certificate was properly revoked automatically when we reissued the certificate, and it simply took some time for the certificate revocation to take effect, or that our browsers were ignoring or not retreiving on updating the CRL. However even under the best circumstances, there would be a latency period after revocation before the cert hit CRLs, and browsers updated their CRL cache. This latency period could be several days, or longer. Thus this would limit the compromise somewhat, but it would still be a very serous problem.
Also, it was not uncommon at the time for browsers to load sites with revoked certificates, without warning the certificate was revoked, because they had short timeouts for CRL update, and long timeouts before requiring an update... So they would load sites without actually checking a current CRL, because they didn't want to interrupt the user experience, and get complaints or excess support requests etc... These are tunable parameters in most browsers.

There was only so much testing we could do, without actually compromising the certificates of third parties, or without committing a felony, so we were not able to test just how far the exposure and potential for compromise went.
The third party that we tried to work with (before testing with other third party resellers to confirm it wasnt just the one) was entirely incompetent, to the point where they actually said they had no-one who could speak with us about the issue, and directed us to speak with Symantec, which we did.
Symantec told us at the time, that it was a third party security issue, and that it was up to the third parties to provide security around their certificate management... that the URI and UID were not supposed to be exposed to users. That if the third party were doing so, it was against their policies etc...
When I agreed not to disclose the problem until it was fixed or otherwise publicly disclosed, Symantec commited to investigating the full extent of the issue, and that if third parties were violating the conditions, policies, and processes required, that they would deal with those issues appropriately... finding all of the certificates which MAY have been impacted, and then replace them... In discussion, they estimated, and we agreed, that doing so would take at least six months for every cert they could identify, and two years for every cert period.
Given Googles experience and actions here, it appears that Symantec did not fix these issues as they committed to, or at least not within a timely manner. They terminated some third parties participation in their reseller program, and revoked and reissued many thousands of certs, but nowhere near all of those that may have been impacted... Unless they had some way of proving those certs were not compromised, that we are not aware of.

Further, though the interfaces we knew about and had been able to demonstrate were compromised were fixed or shut down, they apparently did not fix their core API issues (they ended their RA program in November of 2016 but users may still have been able to retrieve, revoke, and reissue certificates using the original URI links and UID's) and thus as far as we are able to prove, this critical issue remained at least until November of 2016... and may still remain (I haven't attempted to verify again recently) and third party purchased symantec certs may still be subject to compromise in this manner.
.....
UPDATE: Symantec states that they have tested the issue with their current APIs and interfaces, and are not able to recreate it.
.....
So, at this point, I will publicly disclose what I know about the issue, and release anyone I shared the details of the issue with to do so as well.
My STRONG recommendation, is that anyone who purchased a Symantec certificate from a third party, between early 2013 and late 2016, revoke that cert and have it re-issued... either directly by Symantec, or simply revoking and having another trusted CA issue a different cert... as soon as they are able to do so.
As to first party certificates... I don't know and have not been able to validate how extensive the exposure was, through which interfaces, etc... I do know that they fixed the specific issues that I found in the specific interfacecs I was able to validate, within six months as they agreed to. That said... It would be safer to revoke and re-issue, given the problems that Google themselves identified.
As to end users... I would be extremely wary of any site with a symantec cert issued before late 2016, and take some extra caution regarding any symantec cert period.

Vincent Lynch

unread,
Mar 29, 2017, 12:39:12 AM3/29/17
to mozilla-dev-s...@lists.mozilla.org
On Tuesday, March 28, 2017 at 11:08:08 PM UTC-4, uri...@gmail.com wrote:
> For what it's worth, this is the latest post on facebook from the researcher.
> https://www.facebook.com/cbyrneiv/posts/10155129935452436
>
> The private key storage issue sounds like a reseller tool, like
> https://www.thesslstore.com/ssltools/csr-generator.php
> and he found the private key was stored with the reseller, when he accessed the account.
>

I work for The SSL Store and just wanted to quickly clarify that Urijah is using our site as an example of the *type* of tool that the private key issue was related to.

But our specific tool does not store the user's private key in any way. Nor is there any scenario in which we store the user's private key or make it accessible to them through their account.

i...@certly.io

unread,
Mar 29, 2017, 1:19:43 AM3/29/17
to mozilla-dev-s...@lists.mozilla.org
On Tuesday, March 28, 2017 at 3:46:05 PM UTC-4, Nick Lamb wrote:
> In order for Symantec to reveal anybody's private keys they'd first need to have those keys, which is already, IIRC forbidden in the BRs. So even proof that Symantec routinely had these keys is a big deal.

>From what I can tell, this may be correct in the context of retainment. Many CAs have provisions to generate the key on the behalf of the subscriber, though. The wording of the section you're probably thinking of (6.1.2) is tricky:

> Parties other than the Subscriber SHALL NOT archive the Subscriber Private Key without authorization by the Subscriber.

So I guess you would need to see if the subscribers here authorized it.

Peter Gutmann

unread,
Mar 29, 2017, 1:28:10 AM3/29/17
to mozilla-dev-s...@lists.mozilla.org, Nick Lamb
Nick Lamb via dev-security-policy <dev-secur...@lists.mozilla.org> writes:

>In order for Symantec to reveal anybody's private keys they'd first need to
>have those keys

That's standard practice for many CAs, they generate the key and certificate
for you and email it to you as a PKCS #12. It seems to be more common among
lesser-known CAs though, particularly ones with government-mandated monopolies
for some reason, so I'm not sure if Symantec does it.

Peter.

okaphone.e...@gmail.com

unread,
Mar 29, 2017, 3:22:56 AM3/29/17
to mozilla-dev-s...@lists.mozilla.org
Weird.

I expect there are no requirements for a CA to keep other people's private keys safe. After all handling those is definitely not part of being a CA. ;-)

CU Hans

Florian Weimer

unread,
Mar 29, 2017, 8:37:23 AM3/29/17
to mozilla-dev-s...@lists.mozilla.org
* Nick Lamb via dev-security-policy:

> In order for Symantec to reveal anybody's private keys they'd first
> need to have those keys, which is already, IIRC forbidden in the
> BRs.

I think this requirement was dropped because it makes it unnecessarily
difficult to report key compromises. There used to be a time when CAs
demanded zero-knowledge proofs of key compromise (which can be
surprisingly hard to do with existing tools). Fortunately, these
times are over, and CAs no longer categorically reject the submission
of compromised subscriber keys (although my sample is really small due
to my limited factorization capabilities).

Ryan Sleevi

unread,
Mar 29, 2017, 6:31:50 PM3/29/17
to okaphone.e...@gmail.com, mozilla-dev-security-policy
https://cabforum.org/wp-content/uploads/CA-Browser-Forum-BR-1.4.2.pdf

Section 6.1.2
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

okaphone.e...@gmail.com

unread,
Mar 30, 2017, 2:13:46 AM3/30/17
to mozilla-dev-s...@lists.mozilla.org
Right. It is then.

It says private keys can only be stored with permission of the subscriber and encryption must always be used to transfer them. And of course the certificate must be revoked if/when it becomes known that a private key has gotten to the wrong person.

Well... NOT my private keys. I'll create the CSR myself, thank you. ;-)

Anyway, would be nice if there was some evidence in this thread.

CU Hans

tarah.s...@gmail.com

unread,
Mar 31, 2017, 12:27:34 PM3/31/17
to mozilla-dev-s...@lists.mozilla.org
Hello, world.

I'm Tarah. I am the Principal Security Advocate and Senior Director of Engineering at Symantec Website Security (the certificate authority. We make the certs for all the people). Yes, that's a lot of capital letters. I have more, if you want them. I'm out of bubblegum and very seriously out of patience with terrible headlines on this issue.

There's wild irony in me saying this on a Google Group thread, but this has nothing whatsoever to do with Google's other issue with Symantec certificates that we're currently dealing with. Separate the two entirely in your mind. Chris's original claim resurfaced during a fertile media cycle and got rechurned along with whatever other dirty laundry sundry bloggers decided to respin for some hits. They might as well have titled their stories "You Won't Believe This Simple Trick To Irritate Symantec's Resident Curmudgeon Tarah...And Also A Kitten With A Hitler Mustache!".

I personally am the one who beat the everloving hell out of the QuickInvite system on Tuesday morning when checking on Chris Byrne's research and the 2015 report he made. Before I start in on this, let me say that I love Chris; he's part of the family, and none of this is a critique of him. It's actually a critique of the journalists who didn't understand him and his research. I think we can all agree that this is an ongoing issue with journalists who seize on headlines and don't translate well for people who aren't infosec experts.

So, here's what Symantec heard in 2015:

While I wasn't at Symantec then, I saw the original email chain that Chris had with one of our directors. That's a super smart guy we haul out from behind his desk when we're taking something seriously. Chris had two issues: 1) the QuickInvite URL (that's a URL that gets sent via email to a customer buying or renewing a cert) can be used during the process of ordering a certificate to intercept and substitute an arbitrary CSR, and 2) that the QuickInvite URL is cc’ed to the reseller if there is one.

> I found out about this problem... and others related to it... back in early 2015. I warned Symantec about it, and worked through some of the issues with them.
> At the end of it, they asked for a chance to verify how extensive the issue was, and to fix it.

Here's what happened. We did a deep dive into the system. You have to have access to the email account from the original order to get any QuickInvite URL in order to view or edit or issue or revoke certs. If your email account has been compromised, that's not a Symantec problem--that's an email problem. Some of our customers at the time didn't realize that they shouldn't be posting up their private QuickInvite URLs (hashed from order data and some other stuff and salted, but still one doesn't like to reveal specifics on that sort of thing to anyone from 4chan who's decided to poke about here for a while) to their websites or to our support forums. Those URLs had a validity period of 5 days. While we do our best to educate, we cannot control customers who publicly post the equivalent of their username and password. So, because Chris's 2015 complaint that the URL had a lifespan that was unreasonably long was absolutely valid—let me repeat that—Chris was absolutely correct that the URL validity period was too long, within one week we had decreased the lifespan of QuickInvite URLs to 2 hours. That didn't stop some customers from posting them, but it ameliorated the problem by making the URLs useless rapidly. On the second issue, that if you controlled the reseller email address to which a QIURL was cc’ed, you could do bad stuff, we agreed and did absolutely nothing about it, because email.

> I checked a few months later, and they HAD fixed several of the core problems... though not all of them... So I waited, and so did those who I had informed about the problem, and had verified it for themselves...

The core problem we saw was that customers shouldn’t be posting those URLS, and we implemented an amelioration within a week, as I said earlier. The other problem of having an email address I haven’t been able to solve in 25 years on the internet.

> If you purchased a Symantec certificate (or a cert from any of their associated subsidiaries and partners) through a third party, from at least as far back as early 2013 until recently; their third party certificate generation, management, and retrieval API allowed those certificates... including in some cases private keys generated by third parties... to be retrieved without proper authentication, or in some cases any authentication at all.
> Unless the third party added proper security around it, all you had to do was click a link sent in email, and you could retrieve a cert, revoke a cert, and re-issue a cert.

Again, only if someone else had access to the order email. If that is the case and your orders were compromised, please don’t just contact Symantec, but also contact your bank and other financial institutions and change all your passwords. I’m happy to rant about 2FA for a while if you want, too. I have some good sermons on that. One has lepers.

> To clarify... at no time were we able to retrieve private keys directly from Symantec, only from third parties that issued certificates from a web interface without requiring a server generated certificate request.

To the best of my knowledge, Symantec never received any proof of concept or any private keys in the original incident report. We received a report that a person could do so, if someone had control of that email. It behooves me to be humble here, so if a POC was submitted and I didn't see it, I'd be happy to publicly apologize. It doesn't change that a POC of how to intercept someone's email to snag a cert wouldn't change much here.

> It is possible that the certificate was never revoked at all, and another certificate was simply issued, leaving two valid certificates one controlled by the original requestor, and one controlled by the unauthorized party. This would be the worst case scenario.

I don’t think we requested a POC on this, as it’s just speculation, but I’d be personally interested if anyone has any more information. Cert revocation is a nontrivial problem.

> There was only so much testing we could do, without actually compromising the certificates of third parties, or without committing a felony, so we were not able to test just how far the exposure and potential for compromise went.

Committing felonies while at a keyboard. Who hasn’t been there? Everyone out there, please address your concerns to whatever precious darling thought the CFAA was a good idea. Chances are that I want to buy you a beer in person but I only get you the good draft beer if the story involves at least one Matthew Broderick reference.

> The third party that we tried to work with (before testing with other third party resellers to confirm it wasnt just the one) was entirely incompetent, to the point where they actually said they had no-one who could speak with us about the issue, and directed us to speak with Symantec, which we did.

Now that, I actually am interested in. Bad customer service is a problem for security. When Symantec is associated with bad customer service, normal humans that have no idea this thread or issue is happening get affected, and that I care deeply about. I’d want to know who that third party is, because I’ll make sure customers just trying to keep their site transactions safe get help when they need it. I'll reach out to Chris again to find out if we've already severed our relationship with them and follow up.

> Given Googles experience and actions here, it appears that Symantec did not fix these issues as they committed to, or at least not within a timely manner. They terminated some third parties participation in their reseller program, and revoked and reissued many thousands of certs, but nowhere near all of those that may have been impacted... Unless they had some way of proving those certs were not compromised, that we are not aware of.

AHA. And here is where Chris’s opinion got respun by bloggers after a separate Google issue with Symantec popped in the news. There’s no **here** here. No new information, no new vulnerabilities from Chris’s research, and nothing has changed with this issue since the original report he sent us, that a compromised email account could lead to rotten things happening. We’ve now moved from right/wrong to opinions. Let me reiterate: Chris was absolutely correct to talk to us about his original concerns and the URL validity period was fixed within a week. Chris has an opinion now based on Google’s assertions, and I disagree with his 2017 opinion while believing he was correct on his 2015 facts.

> Further, though the interfaces we knew about and had been able to demonstrate were compromised were fixed or shut down, they apparently did not fix their core API issues (they ended their RA program in November of 2016 but users may still have been able to retrieve, revoke, and reissue certificates using the original URI links and UID's) and thus as far as we are able to prove, this critical issue remained at least until November of 2016... and may still remain (I haven't attempted to verify again recently) and third party purchased symantec certs may still be subject to compromise in this manner.

We didn’t get a POC submitted via our vuln disclosure process, and I was told that no demonstration was given to Symantec, though it’s entirely possible that third parties know more about what he was talking about. Again, I’m not actually disagreeing in any way with Chris Byrne. The records at the time show that Symantec reached out twice more to Chris to try to verify what he was talking about and weren’t successful in reaching him to get a proof of concept of any vuln not associated with owning the email associated with the certificate.

Humans do security and humans are imperfect and humans are messy and I don’t **ever** want to stop people from reporting an issue based on whether they think they might be available on a perfect and convenient schedule for follow up with seven people from a Fortune 500 company. :-)

> I do know that they fixed the specific issues that I found in the specific interfacecs I was able to validate, within six months as they agreed to. That said... It would be safer to revoke and re-issue, given the problems that Google themselves identified.
> As to end users... I would be extremely wary of any site with a symantec cert issued before late 2016, and take some extra caution regarding any symantec cert period.

So, do again recognize that in blogger minds, this is being conflated with Google’s opinion, but if you want to have a Symantec cert reissued because you believe you were affected by what Chris was talking about in 2015, just reach out. We will make anything right that you feel needs to be fixed. Like I said, I care about customer service because humans need more security, and a bad experience with security professionals does the whole community harm.

Jakob Bohm

unread,
Mar 31, 2017, 12:51:03 PM3/31/17
to mozilla-dev-s...@lists.mozilla.org
Dear Tarah,

Below some friendly speculation as to what the parts that some bloggers
claimed was included (if those claims were somehow true) might have
been (i.e. where *you* might look for it in internal Symantec
systems/records).

On 31/03/2017 18:27, tarah.s...@gmail.com wrote:
> Hello, world.
>
> I'm Tarah. I am the Principal Security Advocate and Senior Director of Engineering at Symantec Website Security (the certificate authority. We make the certs for all the people). Yes, that's a lot of capital letters. I have more, if you want them. I'm out of bubblegum and very seriously out of patience with terrible headlines on this issue.
>
> There's wild irony in me saying this on a Google Group thread, but this has nothing whatsoever to do with Google's other issue with Symantec certificates that we're currently dealing with. Separate the two entirely in your mind. Chris's original claim resurfaced during a fertile media cycle and got rechurned along with whatever other dirty laundry sundry bloggers decided to respin for some hits. They might as well have titled their stories "You Won't Believe This Simple Trick To Irritate Symantec's Resident Curmudgeon Tarah...And Also A Kitten With A Hitler Mustache!".
>
> I personally am the one who beat the everloving hell out of the QuickInvite system on Tuesday morning when checking on Chris Byrne's research and the 2015 report he made. Before I start in on this, let me say that I love Chris; he's part of the family, and none of this is a critique of him. It's actually a critique of the journalists who didn't understand him and his research. I think we can all agree that this is an ongoing issue with journalists who seize on headlines and don't translate well for people who aren't infosec experts.
>
> So, here's what Symantec heard in 2015:
>
> While I wasn't at Symantec then, I saw the original email chain that Chris had with one of our directors. That's a super smart guy we haul out from behind his desk when we're taking something seriously. Chris had two issues: 1) the QuickInvite URL (that's a URL that gets sent via email to a customer buying or renewing a cert) can be used during the process of ordering a certificate to intercept and substitute an arbitrary CSR, and 2) that the QuickInvite URL is cc’ed to the reseller if there is one.
>
>> I found out about this problem... and others related to it... back in early 2015. I warned Symantec about it, and worked through some of the issues with them.
>> At the end of it, they asked for a chance to verify how extensive the issue was, and to fix it.
>
> Here's what happened. We did a deep dive into the system. You have to have access to the email account from the original order to get any QuickInvite URL in order to view or edit or issue or revoke certs. If your email account has been compromised, that's not a Symantec problem--that's an email problem. Some of our customers at the time didn't realize that they shouldn't be posting up their private QuickInvite URLs (hashed from order data and some other stuff and salted, but still one doesn't like to reveal specifics on that sort of thing to anyone from 4chan who's decided to poke about here for a while) to their websites or to our support forums. Those URLs had a validity period of 5 days. While we do our best to educate, we cannot control customers who publicly post the equivalent of their username and password. So, because Chris's 2015 complaint that the URL had a lifespan that was unreasonably long was absolutely valid—let me repeat that—Chris was absolutely correct that the URL validity period was too long, within one week we had decreased the lifespan of QuickInvite URLs to 2 hours. That didn't stop some customers from posting them, but it ameliorated the problem by making the URLs useless rapidly. On the second issue, that if you controlled the reseller email address to which a QIURL was cc’ed, you could do bad stuff, we agreed and did absolutely nothing about it, because email.
>
>> I checked a few months later, and they HAD fixed several of the core problems... though not all of them... So I waited, and so did those who I had informed about the problem, and had verified it for themselves...
>
> The core problem we saw was that customers shouldn’t be posting those URLS, and we implemented an amelioration within a week, as I said earlier. The other problem of having an email address I haven’t been able to solve in 25 years on the internet.
>
>> If you purchased a Symantec certificate (or a cert from any of their associated subsidiaries and partners) through a third party, from at least as far back as early 2013 until recently; their third party certificate generation, management, and retrieval API allowed those certificates... including in some cases private keys generated by third parties... to be retrieved without proper authentication, or in some cases any authentication at all.
>> Unless the third party added proper security around it, all you had to do was click a link sent in email, and you could retrieve a cert, revoke a cert, and re-issue a cert.
>
> Again, only if someone else had access to the order email. If that is the case and your orders were compromised, please don’t just contact Symantec, but also contact your bank and other financial institutions and change all your passwords. I’m happy to rant about 2FA for a while if you want, too. I have some good sermons on that. One has lepers.
>
>> To clarify... at no time were we able to retrieve private keys directly from Symantec, only from third parties that issued certificates from a web interface without requiring a server generated certificate request.
>
> To the best of my knowledge, Symantec never received any proof of concept or any private keys in the original incident report. We received a report that a person could do so, if someone had control of that email. It behooves me to be humble here, so if a POC was submitted and I didn't see it, I'd be happy to publicly apologize. It doesn't change that a POC of how to intercept someone's email to snag a cert wouldn't change much here.
>

Did Symantec ever offer Symantec-generated private keys (e.g. PKCS#12
files) via that or a similar mechanism? That would explain the
"private key" claim without an additional PoC.

>> It is possible that the certificate was never revoked at all, and another certificate was simply issued, leaving two valid certificates one controlled by the original requestor, and one controlled by the unauthorized party. This would be the worst case scenario.
>
> I don’t think we requested a POC on this, as it’s just speculation, but I’d be personally interested if anyone has any more information. Cert revocation is a nontrivial problem.
>
>> There was only so much testing we could do, without actually compromising the certificates of third parties, or without committing a felony, so we were not able to test just how far the exposure and potential for compromise went.
>
> Committing felonies while at a keyboard. Who hasn’t been there? Everyone out there, please address your concerns to whatever precious darling thought the CFAA was a good idea. Chances are that I want to buy you a beer in person but I only get you the good draft beer if the story involves at least one Matthew Broderick reference.
>
>> The third party that we tried to work with (before testing with other third party resellers to confirm it wasnt just the one) was entirely incompetent, to the point where they actually said they had no-one who could speak with us about the issue, and directed us to speak with Symantec, which we did.
>
> Now that, I actually am interested in. Bad customer service is a problem for security. When Symantec is associated with bad customer service, normal humans that have no idea this thread or issue is happening get affected, and that I care deeply about. I’d want to know who that third party is, because I’ll make sure customers just trying to keep their site transactions safe get help when they need it. I'll reach out to Chris again to find out if we've already severed our relationship with them and follow up.
>
>> Given Googles experience and actions here, it appears that Symantec did not fix these issues as they committed to, or at least not within a timely manner. They terminated some third parties participation in their reseller program, and revoked and reissued many thousands of certs, but nowhere near all of those that may have been impacted... Unless they had some way of proving those certs were not compromised, that we are not aware of.
>
> AHA. And here is where Chris’s opinion got respun by bloggers after a separate Google issue with Symantec popped in the news. There’s no **here** here. No new information, no new vulnerabilities from Chris’s research, and nothing has changed with this issue since the original report he sent us, that a compromised email account could lead to rotten things happening. We’ve now moved from right/wrong to opinions. Let me reiterate: Chris was absolutely correct to talk to us about his original concerns and the URL validity period was fixed within a week. Chris has an opinion now based on Google’s assertions, and I disagree with his 2017 opinion while believing he was correct on his 2015 facts.
>
>> Further, though the interfaces we knew about and had been able to demonstrate were compromised were fixed or shut down, they apparently did not fix their core API issues (they ended their RA program in November of 2016 but users may still have been able to retrieve, revoke, and reissue certificates using the original URI links and UID's) and thus as far as we are able to prove, this critical issue remained at least until November of 2016... and may still remain (I haven't attempted to verify again recently) and third party purchased symantec certs may still be subject to compromise in this manner.
>
> We didn’t get a POC submitted via our vuln disclosure process, and I was told that no demonstration was given to Symantec, though it’s entirely possible that third parties know more about what he was talking about. Again, I’m not actually disagreeing in any way with Chris Byrne. The records at the time show that Symantec reached out twice more to Chris to try to verify what he was talking about and weren’t successful in reaching him to get a proof of concept of any vuln not associated with owning the email associated with the certificate.
>
> Humans do security and humans are imperfect and humans are messy and I don’t **ever** want to stop people from reporting an issue based on whether they think they might be available on a perfect and convenient schedule for follow up with seven people from a Fortune 500 company. :-)
>

Could this be about some other URLs in the (now hopefully closed) RA
interface? Such as an interface for requesting the QuickInvite URL for
later forwarding to the subscriber?

Perhaps an interface that takes only "known/guessable" parameters, such
as subscriber e-mail or order number? This would explain the claim
that an URL sent to one subscriber by an incompetent RA could be
edited to retrieve the data for another subscriber.


>> I do know that they fixed the specific issues that I found in the specific interfacecs I was able to validate, within six months as they agreed to. That said... It would be safer to revoke and re-issue, given the problems that Google themselves identified.
>> As to end users... I would be extremely wary of any site with a symantec cert issued before late 2016, and take some extra caution regarding any symantec cert period.
>
> So, do again recognize that in blogger minds, this is being conflated with Google’s opinion, but if you want to have a Symantec cert reissued because you believe you were affected by what Chris was talking about in 2015, just reach out. We will make anything right that you feel needs to be fixed. Like I said, I care about customer service because humans need more security, and a bad experience with security professionals does the whole community harm.
>


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

tarah.s...@gmail.com

unread,
Mar 31, 2017, 1:31:38 PM3/31/17
to mozilla-dev-s...@lists.mozilla.org
On Friday, March 31, 2017 at 9:51:03 AM UTC-7, Jakob Bohm wrote:
> Dear Tarah,
>
> Below some friendly speculation as to what the parts that some bloggers
> claimed was included (if those claims were somehow true) might have
> been (i.e. where *you* might look for it in internal Symantec
> systems/records).
> >

Thank you.

>
> Did Symantec ever offer Symantec-generated private keys (e.g. PKCS#12
> files) via that or a similar mechanism? That would explain the
> "private key" claim without an additional PoC.

That was available for 1 or 2 customers, but it was not via the QuickInvite interface . It took a downloaded Java app (dear lord) to make that happen, and my belief is that this practice ended around the same time as this initial report but was not related to this issue. I don't think anyone can do this anymore, and I'll follow up on that too, but even in that case, we would never have seen their private keys. Resellers should never have had the ability to generate key pairs and CSRs and if they said they did to Chris, I'm furious.


> Could this be about some other URLs in the (now hopefully closed) RA
> interface? Such as an interface for requesting the QuickInvite URL for
> later forwarding to the subscriber?
>
> Perhaps an interface that takes only "known/guessable" parameters, such
> as subscriber e-mail or order number? This would explain the claim
> that an URL sent to one subscriber by an incompetent RA could be
> edited to retrieve the data for another subscriber.
>

Nope. That QuickInvite URL was/is hashed and salted. I have personally seen the code for generating it. Just editing stuff in that URL without being able to crack it wouldn't give any useful information or lead anywhere. Although now I'm kinda wanting to create a specific error page for people who try. Nah. It'd be funny to me and anyone trying to hack those but the customers wouldn't get my jokes, likely. Brute forcing the URL would take too long, which is why I agree with the original decision to decrease the URL validity timespan. It helps prevent brute force attacks too.

Also, just was chatting with Chris. I just found out that the reseller in question was dropped from our Symantec a while back. Details to follow, but suffice to say that they are no longer a problem for us.

Jakob Bohm

unread,
Mar 31, 2017, 1:49:40 PM3/31/17
to mozilla-dev-s...@lists.mozilla.org
On 31/03/2017 19:31, tarah.s...@gmail.com wrote:
> On Friday, March 31, 2017 at 9:51:03 AM UTC-7, Jakob Bohm wrote:
>> Dear Tarah,
>>
>> Below some friendly speculation as to what the parts that some bloggers
>> claimed was included (if those claims were somehow true) might have
>> been (i.e. where *you* might look for it in internal Symantec
>> systems/records).
>>>
>
> Thank you.
>
>>

>> Could this be about some other URLs in the (now hopefully closed) RA
>> interface? Such as an interface for requesting the QuickInvite URL for
>> later forwarding to the subscriber?
>>
>> Perhaps an interface that takes only "known/guessable" parameters, such
>> as subscriber e-mail or order number? This would explain the claim
>> that an URL sent to one subscriber by an incompetent RA could be
>> edited to retrieve the data for another subscriber.
>>
>
> Nope. That QuickInvite URL was/is hashed and salted. I have personally seen the code for generating it. Just editing stuff in that URL without being able to crack it wouldn't give any useful information or lead anywhere. Although now I'm kinda wanting to create a specific error page for people who try. Nah. It'd be funny to me and anyone trying to hack those but the customers wouldn't get my jokes, likely. Brute forcing the URL would take too long, which is why I agree with the original decision to decrease the URL validity timespan. It helps prevent brute force attacks too.
>

Yep, but there must have been an API (at some level) for generating or
processing the QuickInvite URL. That was what I was suggesting might
have been the issue.

> Also, just was chatting with Chris. I just found out that the reseller in question was dropped from our Symantec a while back. Details to follow, but suffice to say that they are no longer a problem for us.
>

Great!

tarah.s...@gmail.com

unread,
Mar 31, 2017, 2:09:48 PM3/31/17
to mozilla-dev-s...@lists.mozilla.org

> Yep, but there must have been an API (at some level) for generating or
> processing the QuickInvite URL. That was what I was suggesting might
> have been the issue.

So, it's hard for me to answer this question because I didn't see any POC, but 1) it's not physically possible for private keys to be revealed in the interface as described to me and in which I've spent the last few days completely submerged, and 2) there's still the validation process which isn't simply bypassed with this URL.

I have to be cautious here in a couple of my answers, and I hope you appreciate why: as soon as I found out Symantec wasn't affected any more by this reseller, I found out who was. I have to responsibly disclose that information first and get a confirmation from them. I've already notified everyone I had contact info for there, and I'll stay on them.

Daniel Baxter (Aractus)

unread,
Mar 31, 2017, 2:56:01 PM3/31/17
to mozilla-dev-s...@lists.mozilla.org
On Saturday, April 1, 2017 at 3:27:34 AM UTC+11, tarah.s...@gmail.com wrote:
> Hello, world.
>
> I'm Tarah. I am the Principal Security Advocate and Senior Director of Engineering at Symantec Website Security (the certificate authority. We make the certs for all the people).

With all due respect this reply is the most ridiculous load of nonsense I've ever read.

> There's wild irony in me saying this on a Google Group thread, but this has nothing whatsoever to do with Google's other issue with Symantec certificates that we're currently dealing with. Separate the two entirely in your mind. Chris's original claim resurfaced during a fertile media cycle and got rechurned along with whatever other dirty laundry sundry bloggers decided to respin for some hits. They might as well have titled their stories "You Won't Believe This Simple Trick To Irritate Symantec's Resident Curmudgeon Tarah...And Also A Kitten With A Hitler Mustache!".

Yeah OK, I got a few things wrong on my blog post, I can fix that shortly. It's no big deal. At least I'm informing people about security - claiming that we're just "looking for hits" is ridiculous. Most people pay no attention to security, I can't speak for others but I'm trying to inform.

> Here's what happened. We did a deep dive into the system. You have to have access to the email account from the original order to get any QuickInvite URL in order to view or edit or issue or revoke certs. If your email account has been compromised, that's not a Symantec problem--that's an email problem. Some of our customers at the time didn't realize that they shouldn't be posting up their private QuickInvite URLs (hashed from order data and some other stuff and salted, but still one doesn't like to reveal specifics on that sort of thing to anyone from 4chan who's decided to poke about here for a while) to their websites or to our support forums. Those URLs had a validity period of 5 days. While we do our best to educate, we cannot control customers who publicly post the equivalent of their username and password. So, because Chris's 2015 complaint that the URL had a lifespan that was unreasonably long was absolutely valid—let me repeat that—Chris was absolutely correct that the URL validity period was too long, within one week we had decreased the lifespan of QuickInvite URLs to 2 hours. That didn't stop some customers from posting them, but it ameliorated the problem by making the URLs useless rapidly. On the second issue, that if you controlled the reseller email address to which a QIURL was cc’ed, you could do bad stuff, we agreed and did absolutely nothing about it, because email.

> The core problem we saw was that customers shouldn’t be posting those URLS, and we implemented an amelioration within a week, as I said earlier. The other problem of having an email address I haven’t been able to solve in 25 years on the internet.

You realise that none of that fixed the security issue? OK so you salt and hash the URI, understood I got that wrong. Now let's get onto all the problems in your explanation... and there are lots.

Firstly you claim email accounts should be secure - um since WHEN? What universe do you live in?? How long does it take Yahoo! to tell their email clients their passwords were stolen? What about mandatory metadata retention here in Australia by ISPs - that requires decrypting your email (if it's even encrypted in the first place).

Next, you say that URLs in emails should be treated like a password. Um - SINCE WHEN? And furthermore, if it should be treated as a password, if that's your claim, WHY ARE THEY BEING SENT IN PLAINTEXT IN THE EMAIL? You can't have it both ways - if you want customers to treat that as they do a password, you need to treat it with the same care, and put it behind an authentication.

Finally, what have you actually done to address EV revocation? You clearly didn't bother to tell Commonwealth Bank:

https://www.commbank.com.au/

One of the largest banks in Australia that their EV status would evaporate in Chrome. So what did you do to inform your customers about this?

> Again, only if someone else had access to the order email.

Again, stop passing the buck. You need to assume that not every email account in the world is secure! Also, it's a breach of s.6.1.2:

https://cabforum.org/wp-content/uploads/CA-Browser-Forum-BR-1.4.2.pdf

No party other than subscriber shall archive the private key. I.e. it should be impossible to retrieve from an email in the first place.

> To the best of my knowledge, Symantec never received any proof of concept or any private keys in the original incident report.

How does that matter? Chris was able to do it, and if he was able to then your investigation should have uncovered the vulnerability. The fact that it didn't means that you didn't do your due diligence in conducting a thorough investigation. Stop blaming others for the sloppy work of Symantec.

Now we have a situation where hundreds or even thousands of private keys could have been stolen.

> Now that, I actually am interested in. Bad customer service is a problem for security.

Again, this is YOUR problem. Stop passing the buck. Yes the reseller or resellers are also to blame, but your API is what caused the problem. Without it sending clickable links to customer emails to get/revoke/etc. then even the world's most incompetent reseller couldn't have leaked private keys through that system.

> AHA. And here is where Chris’s opinion got respun by bloggers after a separate Google issue with Symantec popped in the news.

Right, well that makes it even worse. That means your admitting to two separate very serious violations of trust.

Oh and where's the transparency?

Jakob Bohm

unread,
Mar 31, 2017, 3:27:27 PM3/31/17
to mozilla-dev-s...@lists.mozilla.org
On 31/03/2017 20:37, Daniel Baxter (Aractus) wrote:
> On Saturday, April 1, 2017 at 3:27:34 AM UTC+11, tarah.s...@gmail.com wrote:
>> Hello, world.
>>
>> I'm Tarah. I am the Principal Security Advocate and Senior Director of Engineering at Symantec Website Security (the certificate authority. We make the certs for all the people).
>
> With all due respect this reply is the most ridiculous load of nonsense I've ever read.
>

Oh, come on, if that's her job title, that's her job title, and at any
CA, that is actually an important job that /someone/ should have.

>> There's wild irony in me saying this on a Google Group thread, but this has nothing whatsoever to do with Google's other issue with Symantec certificates that we're currently dealing with. Separate the two entirely in your mind. Chris's original claim resurfaced during a fertile media cycle and got rechurned along with whatever other dirty laundry sundry bloggers decided to respin for some hits. They might as well have titled their stories "You Won't Believe This Simple Trick To Irritate Symantec's Resident Curmudgeon Tarah...And Also A Kitten With A Hitler Mustache!".
>
> Yeah OK, I got a few things wrong on my blog post, I can fix that shortly. It's no big deal. At least I'm informing people about security - claiming that we're just "looking for hits" is ridiculous. Most people pay no attention to security, I can't speak for others but I'm trying to inform.
>
>> Here's what happened. We did a deep dive into the system. You have to have access to the email account from the original order to get any QuickInvite URL in order to view or edit or issue or revoke certs. If your email account has been compromised, that's not a Symantec problem--that's an email problem. Some of our customers at the time didn't realize that they shouldn't be posting up their private QuickInvite URLs (hashed from order data and some other stuff and salted, but still one doesn't like to reveal specifics on that sort of thing to anyone from 4chan who's decided to poke about here for a while) to their websites or to our support forums. Those URLs had a validity period of 5 days. While we do our best to educate, we cannot control customers who publicly post the equivalent of their username and password. So, because Chris's 2015 complaint that the URL had a lifespan that was unreasonably long was absolutely valid—let me repeat that—Chris was absolutely correct that the URL validity period was too long, within one week we had decreased the lifespan of QuickInvite URLs to 2 hours. That didn't stop some customers from posting them, but it ameliorated the problem by making the URLs useless rapidly. On the second issue, that if you controlled the reseller email address to which a QIURL was cc’ed, you could do bad stuff, we agreed and did absolutely nothing about it, because email.
>
>> The core problem we saw was that customers shouldn’t be posting those URLS, and we implemented an amelioration within a week, as I said earlier. The other problem of having an email address I haven’t been able to solve in 25 years on the internet.
>
> You realise that none of that fixed the security issue? OK so you salt and hash the URI, understood I got that wrong. Now let's get onto all the problems in your explanation... and there are lots.
>
> Firstly you claim email accounts should be secure - um since WHEN? What universe do you live in?? How long does it take Yahoo! to tell their email clients their passwords were stolen? What about mandatory metadata retention here in Australia by ISPs - that requires decrypting your email (if it's even encrypted in the first place).
>

Unfortunately, when initially establishing encryption certificates, one
generally has to start from some kind of unencrypted connection, such
as an unencrypted phone call or an unencrypted e-mail, or an
unencrypted paper mail.

> Next, you say that URLs in emails should be treated like a password. Um - SINCE WHEN? And furthermore, if it should be treated as a password, if that's your claim, WHY ARE THEY BEING SENT IN PLAINTEXT IN THE EMAIL? You can't have it both ways - if you want customers to treat that as they do a password, you need to treat it with the same care, and put it behind an authentication.
>

But a password or random secret string in a URL /is/ authentication.
And in an e-mail, it is behind whatever authentication you (not
Symantec) have set up for that e-mail (and Symantec obviously didn't
know about the Yahoo undisclosed breach at the time).

> Finally, what have you actually done to address EV revocation? You clearly didn't bother to tell Commonwealth Bank:
>
> https://www.commbank.com.au/
>
> One of the largest banks in Australia that their EV status would evaporate in Chrome. So what did you do to inform your customers about this?
>

This is the *unrelated* issue, now take that back to the thread that
Google is trying to hide on their own mailing list for browser engine
issues.


>> Again, only if someone else had access to the order email.
>
> Again, stop passing the buck. You need to assume that not every email account in the world is secure! Also, it's a breach of s.6.1.2:

They need to only assume that whatever e-mail account people provide
for signup is secure enough for the people choosing what e-mail to
provide /and only on the day or week where they provide that e-mail
address/.

>
> https://cabforum.org/wp-content/uploads/CA-Browser-Forum-BR-1.4.2.pdf
>
> No party other than subscriber shall archive the private key. I.e. it should be impossible to retrieve from an email in the first place.
>

And she has insisted they don't do that. Period. They may in the
past, via a completely different security system, have provided the
ability to generate and promptly delete the private key for people who
couldn't figure out how to do that themselves.


>> To the best of my knowledge, Symantec never received any proof of concept or any private keys in the original incident report.
>
> How does that matter? Chris was able to do it, and if he was able to then your investigation should have uncovered the vulnerability. The fact that it didn't means that you didn't do your due diligence in conducting a thorough investigation. Stop blaming others for the sloppy work of Symantec.
>

But was he, or did you simply exaggerate something in your blog?

> Now we have a situation where hundreds or even thousands of private keys could have been stolen.
>

According to what she said, there was no way that many private keys
(if any) was exposed.


>> Now that, I actually am interested in. Bad customer service is a problem for security.
>
> Again, this is YOUR problem. Stop passing the buck. Yes the reseller or resellers are also to blame, but your API is what caused the problem. Without it sending clickable links to customer emails to get/revoke/etc. then even the world's most incompetent reseller couldn't have leaked private keys through that system.
>

And that's what she wrote. Bad customer service is *their* problem and
she wanted to know who to kick out. In a later post she found out
(from Chris) that they already *kicked out the people providing the bad
customer service* .


>> AHA. And here is where Chris’s opinion got respun by bloggers after a separate Google issue with Symantec popped in the news.
>
> Right, well that makes it even worse. That means your admitting to two separate very serious violations of trust.

The Google issue is *microscopic* except Google is using it as a lame
excuse to threaten Symantec big time.

>
> Oh and where's the transparency?
>

You should ask Google that.

Vincent Lynch

unread,
Mar 31, 2017, 3:51:30 PM3/31/17
to Daniel Baxter (Aractus), mozilla-dev-s...@lists.mozilla.org
>
> Finally, what have you actually done to address EV revocation? You clearly
> didn't bother to tell Commonwealth Bank:
>
> https://www.commbank.com.au/
>
> One of the largest banks in Australia that their EV status would evaporate
> in Chrome. So what did you do to inform your customers about this?



As Jacob Bohm has said in his reply, Google's proposal to remove EV status
for Symantec certificates has *nothing* to do with this report.

Furthermore, the issue you have brought up with Commonwealth Bank has
*nothing* to do with *either* of the two separate issues Symantec is
currently dealing with. It also has nothing to do with Commbank.com.au
being "hacked," as you have written on your website
<https://blog.aractus.com/the-commonwealth-bank-loses-its-green-bar/>.

It is simply a bug, related to an OID included in the certificate. This has
been documented by Chrome
<https://bugs.chromium.org/p/chromium/issues/detail?id=705285>.

You can test this by visiting https://www.commbank.com.au/ in Chrome
Canary, which has fixed the bug, and returned the green address bar to the
site.

tarah.s...@gmail.com

unread,
Mar 31, 2017, 6:11:00 PM3/31/17
to mozilla-dev-s...@lists.mozilla.org

>
> Yeah OK, I got a few things wrong on my blog post, I can fix that shortly. It's no big deal. At least I'm informing people about security - claiming that we're just "looking for hits" is ridiculous. Most people pay no attention to security, I can't speak for others but I'm trying to inform.
>

So sorry, but I don't know what you're referring to. Did you tweet me a link to a blog post? Blame Jack if so; all of us are dealing with hugely problematic threading today due to the Twitter @ changes. If you reply here with what you are talking about, I can take a look, though unfortunately I might not be able to get to it today. I always like hearing opposing viewpoints.

I feel like Jakob basically covered every other point.

mono...@gmail.com

unread,
Mar 31, 2017, 7:03:45 PM3/31/17
to mozilla-dev-s...@lists.mozilla.org
Maybe I'm alone in this but, while entertaining, I'm taken aback a bit if this is official Symantec communication in a forum like m.d.s.p.

tarah.s...@gmail.com

unread,
Mar 31, 2017, 8:28:06 PM3/31/17
to mozilla-dev-s...@lists.mozilla.org
On Friday, March 31, 2017 at 4:03:45 PM UTC-7, mono...@gmail.com wrote:
> Maybe I'm alone in this but, while entertaining, I'm taken aback a bit if this is official Symantec communication in a forum like m.d.s.p.

I don't do the official Symantec public relations responses. There's a channel for that; they give the statement you can give corporate folks and customers. I'm in infosec and I'm here answering developer questions. I actually don't even know if those are collected somewhere. I'll go find out if you need something for your clients. LMK and I'll see what I can do.

Daniel Baxter

unread,
Mar 31, 2017, 9:01:13 PM3/31/17
to mozilla-dev-s...@lists.mozilla.org
On Saturday, April 1, 2017 at 6:27:27 AM UTC+11, Jakob Bohm wrote:
> Oh, come on, if that's her job title, that's her job title, and at any
> CA, that is actually an important job that /someone/ should have.

I meant the content of her reply, not her job title.

> Unfortunately, when initially establishing encryption certificates, one
> generally has to start from some kind of unencrypted connection, such
> as an unencrypted phone call or an unencrypted e-mail, or an
> unencrypted paper mail.

Not the private key though. And besides, it doesn't matter if your CSR and Certificate are duplicated, they're not a secret only the private key is.

> But a password or random secret string in a URL /is/ authentication.
> And in an e-mail, it is behind whatever authentication you (not
> Symantec) have set up for that e-mail (and Symantec obviously didn't
> know about the Yahoo undisclosed breach at the time).

I don't know how to say this any other way - but that's unbelievably sloppy. That's like saying you'll let your customers email their credit card numbers to you. You don't put anything that sensitive into an email address, ever. Any time I get a password emailed to me I change it immediately.

Besides this is passing the blame to the customers. And there is a big difference of how informed each party was. Symantec knew about this exploit, customers didn't. They knew to take extra precautions, customers didn't have that knowledge.

> They need to only assume that whatever e-mail account people provide
> for signup is secure enough for the people choosing what e-mail to
> provide /and only on the day or week where they provide that e-mail
> address/.

So since December 2016 they banned all Yahoo! emails, right?

> But was he, or did you simply exaggerate something in your blog?

He says it on his facebook post: "for some third party vendors, depending on the type of certificate, and how the certificate was issued, you could also retrieve private keys."

> The Google issue is *microscopic* except Google is using it as a lame
> excuse to threaten Symantec big time.

I believe the 30,000 number they talked about relates to the same security issue that Chris Byrne identified. If it didn't then here's my question: how many certificates were affected by the Chris Byrne issue?

On Saturday, April 1, 2017 at 9:11:00 AM UTC+11, tarah.s...@gmail.com wrote:
>
> So sorry, but I don't know what you're referring to. Did you tweet me a link to a blog post? Blame Jack if so; all of us are dealing with hugely problematic threading today due to the Twitter @ changes. If you reply here with what you are talking about, I can take a look, though unfortunately I might not be able to get to it today. I always like hearing opposing viewpoints.

Sure, it's no secret: https://blog.aractus.com/the-symantec-ssl-shitstorm/

That page I'm pretty sure wasn't even indexed in google when I posted. Even it it was, my viewership is only small. I was here reading these threads, and gathering more information so that I can fix up all the errors I've made - that's how we all learn, by making mistakes and fixing them. Anyway, I only used primary sourced material (ie, the google groups discussions, Chris's facebook post, and the Symantec website), I haven't read anything on Twitter.

Again, I obviously can't speak for others, but any confusion over the facts here could have been easily avoided had Symantec made a full public statement about the Chris Byrne vulnerability the moment that it no longer posed a threat to customers.

Where I will agree with you is that from the description, it would have involved a fairly sophisticated attack to steal private keys from the "incompetent resellers", and that they were equally culpable.

aractusp...@gmail.com

unread,
Mar 31, 2017, 9:02:32 PM3/31/17
to mozilla-dev-s...@lists.mozilla.org
On Saturday, April 1, 2017 at 6:51:30 AM UTC+11, Vincent Lynch wrote:
>
> It is simply a bug, related to an OID included in the certificate. This has
> been documented by Chrome
> <https://bugs.chromium.org/p/chromium/issues/detail?id=705285>.

OK, I'll update that, thanks.

Vincent Lynch

unread,
Mar 31, 2017, 10:22:47 PM3/31/17
to Daniel Baxter, mozilla-dev-s...@lists.mozilla.org
>
> I don't know how to say this any other way - but that's unbelievably
> sloppy. That's like saying you'll let your customers email their credit
> card numbers to you. You don't put anything that sensitive into an email
> address, ever. Any time I get a password emailed to me I change it
> immediately.
> Besides this is passing the blame to the customers. And there is a big
> difference of how informed each party was. Symantec knew about this
> exploit, customers didn't. They knew to take extra precautions, customers
> didn't have that knowledge.



Email validation is a *standard* method for issuing DV certificates used by
*most* CAs. Hundreds of certificates a day are issued with this method. It
is detailed in the CA/B Forum's Baseline Requirements which all trusted CAs
are bound to. If you have an issue with that, well, it is off topic to this
incident.


> He says it on his facebook post: "for some third party vendors, depending
> on the type of certificate, and how the certificate was issued, you could
> also retrieve private keys."


As you have quoted: "for some third party vendors." Symantec themselves had
nothing to do with storing private keys. If a reseller chose to do this,
the vulnerability lies with them. It would be inaccurate to say that a
Symantec reseller storing private keys is the same thing as Symantec
storing private keys.

According to Tarah's response, Chris' report involved three different types
of problems: legitimate problems with Symantec's API (which have been fixed
a while ago), unfortunate but uncontrollable general security issues (such
as the ability to control a user's account if you have access to their
email), and security issues of a third party (resellers storing private
keys).


I believe the 30,000 number they talked about relates to the same security
> issue that Chris Byrne identified. If it didn't then here's my question:
> how many certificates were affected by the Chris Byrne issue?


No, it does not. The issue Google has been dealing with in the blinkdev
thread - which I will refer to as the "RA Incident" - involves companies
which Symantec contracted with and had "independent authority to issue
certificates under Symantec intermediates
<https://wiki.mozilla.org/CA:Symantec_Issues>". This is separate from a
reseller, which is purely a business/marketing relationship that does not
grant the reseller any ability to issue certificates. The RA Incident is
ENTIRELY separate from what Chris Byrne has reported.

Ryan Sleevi (Google engineer) posted the proposal about how to handle the
RA Incident and you can see it makes no mention of any API issues:
https://groups.google.com/a/chromium.org/d/msg/blink-dev/eUAKwjihhBs/rpxMXjZHCQAJ

As far as I am aware, there have been no figures shared about the
certificates affected by the Chris Byrne issue.


On Fri, Mar 31, 2017 at 9:01 PM, Daniel Baxter via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On Saturday, April 1, 2017 at 6:27:27 AM UTC+11, Jakob Bohm wrote:
> > Oh, come on, if that's her job title, that's her job title, and at any
> > CA, that is actually an important job that /someone/ should have.
>
> I meant the content of her reply, not her job title.
>
> > Unfortunately, when initially establishing encryption certificates, one
> > generally has to start from some kind of unencrypted connection, such
> > as an unencrypted phone call or an unencrypted e-mail, or an
> > unencrypted paper mail.
>
> Not the private key though. And besides, it doesn't matter if your CSR and
> Certificate are duplicated, they're not a secret only the private key is.
>
> > But a password or random secret string in a URL /is/ authentication.
> > And in an e-mail, it is behind whatever authentication you (not
> > Symantec) have set up for that e-mail (and Symantec obviously didn't
> > know about the Yahoo undisclosed breach at the time).
>
> I don't know how to say this any other way - but that's unbelievably
> sloppy. That's like saying you'll let your customers email their credit
> card numbers to you. You don't put anything that sensitive into an email
> address, ever. Any time I get a password emailed to me I change it
> immediately.
>
> Besides this is passing the blame to the customers. And there is a big
> difference of how informed each party was. Symantec knew about this
> exploit, customers didn't. They knew to take extra precautions, customers
> didn't have that knowledge.
>
> > They need to only assume that whatever e-mail account people provide
> > for signup is secure enough for the people choosing what e-mail to
> > provide /and only on the day or week where they provide that e-mail
> > address/.
>
> So since December 2016 they banned all Yahoo! emails, right?
>
> > But was he, or did you simply exaggerate something in your blog?
>
> He says it on his facebook post: "for some third party vendors, depending
> on the type of certificate, and how the certificate was issued, you could
> also retrieve private keys."
>
> > The Google issue is *microscopic* except Google is using it as a lame
> > excuse to threaten Symantec big time.
>
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>



--
Vincent Lynch

Peter Bowen

unread,
Mar 31, 2017, 11:07:32 PM3/31/17
to Daniel Baxter, mozilla-dev-s...@lists.mozilla.org

> On Mar 31, 2017, at 6:01 PM, Daniel Baxter via dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
>
> On Saturday, April 1, 2017 at 6:27:27 AM UTC+11, Jakob Bohm wrote:
>> Oh, come on, if that's her job title, that's her job title, and at any
>> CA, that is actually an important job that /someone/ should have.
>
> I meant the content of her reply, not her job title.

Hi there,

I’m Peter. I am a Principal Security Engineer at Amazon and Vice President of Amazon Trust Services (the certification authority). I’ve been participating in this group for several years, mostly in an individual capacity. One of the part of the Mozilla CA program I value the most is open and transparent communication. To that end, I appreciate the direct and clear email from Tarah to the group.

As the mozilla.dev.security.policy name indicates, this is a netnews (i.e. Usenet) group which is gatewayed to an email list and Google Group. It is part of the Mozilla Forums, which are primarily a place for technical discussions. I would hope that anyone looking for a formal statement from any organization whose employees participate in this group would reach out to the appropriate PR team.

I'm glad to see posts that help keep a high signal-to-noise ratio in this fourm, as long as they fall within the etiquette rules (https://www.mozilla.org/en-US/about/forums/etiquette/).

Thanks,
Peter

uri...@gmail.com

unread,
Mar 31, 2017, 11:37:50 PM3/31/17
to mozilla-dev-s...@lists.mozilla.org
Well, just to bring everything full circle, here's CrossCert's public Trello board (???) where they discuss implementing QuickInvite.

https://trello.com/c/tSgQDta3/8-symantec-ssl-reseller-api

Gervase Markham

unread,
Apr 1, 2017, 4:29:06 AM4/1/17
to mozilla-dev-s...@lists.mozilla.org
Hi Daniel,

We appreciate your additional input into determining the exact scope of
this problem.

On 31/03/17 19:37, Daniel Baxter (Aractus) wrote:
> With all due respect this reply is the most ridiculous load of
> nonsense I've ever read.

However, please keep the tone civil. If it's nonsense, demonstrate why
that's so rather than asserting it.

> Yeah OK, I got a few things wrong on my blog post, I can fix that
> shortly.

We would appreciate it if you would let us know what the updates are.

> Firstly you claim email accounts should be secure - um since WHEN?

Regardless of the wisdom of this assertion, it is true that many CAs
rely on the (relative) security of email when doing domain validation.
Symantec is not alone in this respect. It's probably not productive to
have an argument at this point over whether email-based domain
validation is a good idea or not.

> Next, you say that URLs in emails should be treated like a password.
> Um - SINCE WHEN? And furthermore, if it should be treated as a
> password, if that's your claim, WHY ARE THEY BEING SENT IN PLAINTEXT
> IN THE EMAIL? You can't have it both ways - if you want customers to
> treat that as they do a password, you need to treat it with the same
> care, and put it behind an authentication.

This leads to a chicken-and-egg problem. To use email for domain
validation, you need to send something in the email which the domain
owner does not already know, and then use that to validate that the
person wanting the certificate is able to receive the email. It doesn't
matter whether it's a token or the username and password to a web interface.

> Again, stop passing the buck. You need to assume that not every email
> account in the world is secure! Also, it's a breach of s.6.1.2:
>
> https://cabforum.org/wp-content/uploads/CA-Browser-Forum-BR-1.4.2.pdf
>
> No party other than subscriber shall archive the private key. I.e.
> it should be impossible to retrieve from an email in the first
> place.

Do you have evidence that private keys were retrievable? Can you provide
steps to reproduce?

> How does that matter? Chris was able to do it, and if he was able to
> then your investigation should have uncovered the vulnerability. The

It would be great if Chris were available to drop in and corroborate
this. I may reach out to him.

Gerv

Nick Lamb

unread,
Apr 1, 2017, 6:01:23 AM4/1/17
to mozilla-dev-s...@lists.mozilla.org
On Friday, 31 March 2017 17:27:34 UTC+1, tarah.s...@gmail.com wrote:
> I'm Tarah. I am the Principal Security Advocate and Senior Director of Engineering at Symantec Website Security (the certificate authority.

Hello Tarah,

Regular readers of m.d.s.policy will not be surprised that the news media doesn't do a great job of explaining problems with the Web PKI.

As so often I have questions, none of which involve kittens or Ferris Bueller but instead today focus on QuickInvite URLs.

1. Symantec's own web site describes "Quick Invite" in several places, I presume this is the same QuickInvite system you're talking about. It explains that "The Quick Invite Duration default expiration time is 30 days, but can be set during the sending of the invite" with a maximum of one year. Presumably this is simply obsolete documentation, or else it refers to some other feature under a similar name? If the former, I am happy to provide the URLs where I found this, are you able to ensure they get updated or removed ?

2. What was the designed purpose of the QuickInvite URL / the QuickInvite system itself ? I appreciate that for you its purpose is very obvious as you've spent time up to your neck in it, but to me it's still rather opaque. Let me set out some possible actors in our play, and hopefully you can tell me which actors arrange for the URL to be sent out, which actors receive it, and what they can do with it. That last is quite important. If the list I provide is inadequate feel free to invent more people, just explain what they do.

Exam PLE is a small business with a web site, www.example.com
Andrea is the sysadmin at Exam PLE
Betty is Alice's boss, her details are listed in WHOIS for example.com
Catherine is an employee at the CA, Symantec
Jo is an SSL reseller, she offers cheap Symantec certs
Valorie is a seemingly helpful person who answers Andrea's queries on Q&A sites
Wendy runs a web hosting business, she runs the servers www.example.com uses

uri...@gmail.com

unread,
Apr 1, 2017, 5:11:53 PM4/1/17
to mozilla-dev-s...@lists.mozilla.org
I think page 8 of their manual at least partially explains how and what "QuickInvite" is. The whole document is rather interesting...

https://www.geotrust.com/geocenter/resources/partnercenter-user-guide.pdf
Reply all
Reply to author
Forward
0 new messages