| How do you handle mass revocation requests? | Jeremy Rowley | 28/02/18 09:38 | Hi everyone,
I wanted to share an incident report regarding the revocation of certain certificates ordered through a reseller. On February 2nd, 2018, we received a request from Trustico to mass revoke all certificates that had been ordered by end users through Trustico. Unfortunately, the email was not sent to the appropriate certificate problem reporting channels and did not surface immediately so we're delayed in sharing the concerns and information. Once we were alerted, the team kicked off a debate that I wanted to bring to the CAB Forum. Basically, our position is that resellers do not constitute subscribers under the Baseline Requirement's definitions (Section 1.6.1). As such, we needed to confirm that either the key was compromised or that they revocation was authorized by the domain holder (the subscriber) prior to revoking the certificate. The certificates were not alleged as compromised at that time. Later, the company shared with us that they held the private keys and the certificates were compromised, trying to trigger the BR's 24-hour revocation requirement. However, we insisted that the subscriber must confirm the revocation request or there must be evidence of the private key compromise. Normally, we permit partners to revoke and manage the certificates freely on behalf of their customer, with DigiCert providing all validation and issuance services. However, the sheer number of revocation requests (50k) and allegation of compromise triggered concerns around the impact to the web and browser users. Therefore, this was categorized as high risk, especially considering the relationship between Trustico and DigiCert is terminating. On 2/27/2018, at my request for proof of compromise, we received a file with 23k private keys matched to specific Trustico customers. This definitely triggered our 24-hour revocation processing requirement under 4.9.1.1.3. Once we received the keys, we confirmed that these were indeed the matching private keys for the reported certificates. We will be revoking these certificates today (February 28th, 2018). At this time, Trustico has not provided any information about how these certificates were compromised or how they acquired the private keys. As is standard practice for a Certificate Authority, DigiCert never had possession of these private keys. Currently, we are only revoking the certificates if we received the private keys. There are additional certificates the reseller requested to have revoked, but DigiCert has decided to disregard that request until we receive proof of compromise or more information about the cause of this incident. DigiCert sent out emails to the affected customers in order to notify them that their certificate(s) are being revoked. This revocation only affects those customers and there is no additional exposure that we are aware of at this time, nor any reason to believe there is. This raises a question about the MDSP policy and CAB Forum requirements. Who is the subscriber in the reseller relation? We believe this to be the key holder. However, the language is unclear. I think we followed the letter and spirit of the BRs here, but I'd like feedback, perhaps leading to a ballot that clarifies the subscriber in a reseller relationship. This also brings up a ballot about the level of due diligence required for cert revocation. We generally believe that the private key or demonstration of domain control is sufficient to request revocation. Others are at the CAs discretion. Should we clarify what the due diligence looks like? Are there other things we should have done or been doing? What kind of transparency would the Mozilla community like around this issue? There aren't many more facts than I shared above, but there is a lot of speculation. Let me know what I can share to help alleviate confusion and answer questions. Jeremy |
| Re: How do you handle mass revocation requests? | Tom | 28/02/18 09:54 | On 28 February 2018 at 11:37, Jeremy Rowley via dev-security-policy
<dev-secur...@lists.mozilla.org> wrote:Have you contacted the customers whose certificates you have not revoked; but which were in the original batch? It seems likely they're going to wind up revoked too. Is there any way to identify these certificates through crt.sh or through a manual cert search? (Some special Intermediate/CRL/OID/string...?) Has Trustico said anything about whether or not they will provide more information in the future? -tom |
| Re: How do you handle mass revocation requests? | Nick Lamb | 28/02/18 10:46 | On Wed, 28 Feb 2018 17:37:25 +0000
Jeremy Rowley via dev-security-policy> Hi everyone, > On February 2nd, 2018, we received a request from Trustico to massIs this date (2 February, so almost four weeks ago) correct? In any case, my first thought here was to check whether Trustico understand what revocation would and would not achieve, and still want to go ahead. I know from dealing with Let's Encrypt users that people can get some ideas about what's going on with the Web PKI that bear no resemblance to reality, and although we have no formal duty to teach them better, there's no benefit to the world from pretending they're right. For example at least once a week we'll see Let's Encrypt users who want to revoke a certificate they've issued and are asking questions about how - almost always it turns out they don't have a sensible reason to revoke, e.g. they hoped revoking will reset their API rate limits, or the files are taking up too much disk space for them and of course revocation won't help with either of these things. I'm not sure that resellers in particular make this more murky than many other common scenarios, but if we come up with better and more clearly defined terminology that never hurts anybody. In this particular case I have a concern raised above (did Trustico _really_ want revocation?) that I think could and perhaps should have occurred to you at DigiCert and if it didn't I hope a lesson can be learned there. It's not clear yet to me that it's a problem in this case, but where you're provided with proof of control over keys and there's dispute about whether you should have revoked the certs it seems like proof of control (not the keys themselves) for a random sample of affected certs is a good thing to show m.d.s.policy In most settings I think that this proof would not necessarily be helpful because it's going to need a neutral third party expert to explain what it means, but m.d.s.policy is if nothing else brimming with third party experts so it's welcome here. |
| How do you handle mass revocation requests? | google....@trustico.com | 28/02/18 10:56 | Jeremy,
Today many of our customers experienced lengthy delays when attempting to contact us via phone, e-mail and live chat. The reason for the delays were due to an unexpected e-mail that DigiCert sent to our customers containing some inaccurate information. We were not informed that the e-mail would be sent and were caught by surprise. We had disabled this e-mail within your control panel and opted to send our own notices, though you took it upon yourselves to send my personal details to 20,000 customers. We didn't authorise DigiCert to contact our customers and we didn't approve the content of their e-mail. At no time had any private keys been compromised, nor had we ever informed to you that any private keys had been compromised. During our many discussions over the past week we put it to you that we believe Symantec to have operated our account in a manner whereby it had been compromised. Your usage of the word compromise has been twisted by you to your benefit and is absolutely defamatory. We believe the orders placed via our Symantec account were at risk and were poorly managed. We have been questioning Symantec without response as to concerning items for about a year. Symantec simply ignored our concerns and appeared to bury them under the next issue that arose. There are a range of issues that Trustico intends to investigate via legal means. In good conscience we decided it wasn't ideal to have any active SSL Certificates on the Symantec systems, nor any that didn't meet our stringent security requirements. Our concerns also relate to the upcoming distrust of all Symantec SSL Certificate brands within Google Chrome and the reasoning behind it. The same management team responsible for that situation is duly employed at DigiCert and are fully managing our account, causing grave concern on our part as it appears to be business as usual with a new name. We were also a victim whereby Symantec mis-issued SSL Certificates owned by us, subsequently we were asked to keep the matter quiet, under a confidentially notice. We had implemented a system to ensure that all customers would receive a replacement SSL Certificate, though today it had failed to perform this function. In our view it is absolutely critical that an SSL Certificate performs its intended function. Symantec's issue with Google appeared to seal that deal, whereby they will all eventually fail due to distrust. In accordance with CAB Forum guidelines we acted to immediately revoke active SSL Certificates whereby trust was questionable. Trustico absolutely distrusts the Symantec brand due to the issues that forced Symantec into having to hand over its entire authentication business to an alternate CA and a range of issues beforehand. Though, Symantec was ultimately acquired by DigiCert - though now DigiCert appears to be influenced by the Symantec management team - that to date still is managing our account. Trustico stopped offering the Symantec brands early February after a meeting with your Symantec management team, whereby they had disclosed to us that various reckless issues had occurred (recording available). We realize that this mass revocation is bothersome and time consuming for all that have been affected. We're working to contact all customers to get orders replaced as priority and working through a backlog of enquiries. Unfortunately, things didn't go very well for us today and we are extremely sorry for all the confusion and inconvenience that has been caused. We were relying on systems that would easily replace and issue SSL Certificates automatically, though that didn't occur and we ended up in quite a mess. DigiCert didn't work with us to understand the issues and resolve them, we felt we were at a dead end. We'll be following up again shortly with an update surrounding what occurred and more information about where we experienced failures. In the meantime, our staff are concentrating on getting SSL Certificates issued as quickly as possible from a reputable and trusted CA. As for the question of who is the subscriber, well ultimately that came down to the agreement that we had made with you and the agreement on the website. The conditions of the subscriber agreement were not honoured by you when it came to our requests anyway – so it’s hard to comment on who you refer to be a subscriber – even though your subscriber agreement clearly states that the reseller is the subscriber in our case. We instigated revocation of SSL Certificates as per your subscriber agreement. I shall like to further note that DigiCert terminated our contract on Sunday 25th February immediately after we put to you that we intended to seek a legal opinion in respect to the operation of our account and security concerns. Jeremy, the evidence that we have at hand as per the issues surrounding the revocation is somewhat different to what you are disclosing here today. As one of Symantec's former largest partners - my personal opinion and personal experience is that Symantec is a company that thrives on recklessness and one that I wouldn't trust nor deal with. Zane Lucas |
| Re: How do you handle mass revocation requests? | Ryan Sleevi | 28/02/18 10:58 | On Wed, Feb 28, 2018 at 12:37 PM, Jeremy Rowley via dev-security-policy < > Unfortunately, the email was not sent to the appropriate certificateI think there's a little nuance to this in the general case (e.g. consider "Resllers" who are also the hosting provider, and thus constitute the Applicant/Subscriber even when they're not the domain holder, but authorized by them), but based on this specific case, I think this sounds like a correct determination. I think the biggest question is who agreed to the terms - Trustico or the end-user - and that mostly impacts the rest of the decision here. Further, did Trustico warrant on behalf of the user that the user agreed to the terms (in which case, I would think it's a bit of a copout, and it's really Trustico agreeing, thus the Subscriber), or did DigiCert have direct communication with the user on the basis of Trustico's introduction (in which case, yeah, the Subscriber is the user) Anyway, just highlighting it here as perhaps not a uniform consensus, but perhaps not as material as what follows. I think all of this sounds reasonable, no different than a security researcher (or random member of the public) who were to claim compromise with no evidence of that. For the same reason that "Jane Random User" should not be able to cause revocation merely by assertion, I think that sounds reasonable. I think the question here is less about who is the Applicant, but who is the Applicant Representative. If the Reseller is signing/agreeing the request on behalf of the applicant, or the Subscriber Agreement, they're the Applicant Representative and ostensibly should be taken as authorized. I think the gap here is we have this notion of Applicant/Applicant Representative prior to the issuance - in which some 3P may agree to a Subscriber Agreement (or warrant agreement), yet claim the Subscriber is somehow a different entity or that Representative is no longer bound in scope. That seems pretty troubling - both in terms of how the Applicant Representative is verified as authorized (which seems fundamentally what a Reseller who agrees to a ToS is) - and as to how revocation works. I think Wayne's still looking at the revocation space (and I'm much belated in my offering feedback), but I think one of the gaps is one we've seen come up with Google domains and Let's Encrypt. And while I share these stories, to be clear, I don't think LE has done anything wrong in issuance or handling of it - it just is a good demonstration of the nuance that any such clarification should consider. Consider https://crt.sh/?id=245397170 , which was a google.tg certificate obtained from LE following an apparent Registry compromise. Prior to the compromise, Google operated google.tg, during the compromise, an unknown third-party did, and following the compromise, Google again operated/controlled google.tg. When it came to requesting revocation, however, this highlighted a challenge. Let's Encrypt has several defined methods for validation - the HTTP-01 method, TLS-SNI-01 (at the time), and DNS-01. Google services - particularly those hosting/serving google.tg - do not support any of those methods (by somewhat intentional design). Google also did not control the private key - naturally. Based on the evidence Google provided - and, importantly, through consultation with 3P - LE was able to determine the .tg compromise and revoke the certificate. A mandate of 'due diligence' that suggested demonstration of private key, or of using LE's preferred methods, would have prevented that revocation. A model that offers CA significant flexibility and discretion does have the downside of the CA choosing to *refuse* to revoke, and them claim that they were operating within the bounds of the Baseline Requirements. Thus, any form of requiring due diligence has to consider an adversarial model of CAs (sorry Jeremy!), in which the CA may have incentives, whether real or imagined, not to revoke and/or to impose hardship on the domain operator. The current structure favors the 'victims' in the example I just gave, but I think also presents risks - as you raise - of 3P requesting revocation without authorization. I don't know if or how we'll be able to square those somewhat competing interests, but I think it's worth keeping in the consideration. If we stretch the idea out further, one could imagine that CAs must support 'all' validation methods to authenticate a revocation request. But now that shifts the burden from the victim (in having to prove control) onto the CA - which is its own set of tradeoffs. It also introduces new risk - in which adversaries may use a weaker method of authentication maliciously (for example, consider 3.2.2.4.8, in which anyone sharing an IP could cause the revocation of a cert of anyone else sharing that IP) In any event, I'm hugely appreciative of the details you've taken to share and be transparent with the community regarding this. This is an incredibly valuable discussion to have, and more importantly, and valuable level of transparency regarding the incident, given the amount of apparent misinformation and confusion regarding this flying about. |
| Re: How do you handle mass revocation requests? | Peter Bowen | 28/02/18 11:14 | On Wed, Feb 28, 2018 at 9:37 AM, Jeremy Rowley via dev-security-policy > This raises a question about the MDSP policy and CAB Forum requirements. WhoFor certs with subject identity information (commonly called IV, OV, and EV certs), there is no question about the subscriber. The Subscriber is the entity identified in the subject: "The Subject is either the Subscriber or a device under the control and operation of the Subscriber." For certificates without subject identity information (DV certificates), the certificate does not list the subscriber. However the CA clearly knows the subscriber, as the subscriber is the "natural person or Legal Entity to whom a Certificate is issued and who is legally bound by a Subscriber Agreement or Terms of Use" In some cases the "reseller" might be the subscriber if the reseller is a hosting company and is the one that accepts the subscriber agreement but in the traditional reseller model their customer is the subscriber as the reseller's customer is the one accepting the subscriber agreement. Given that DigiCert appears to have contact information for the Trustico customers, that suggests that the Trustico customer is likely the subscriber, but looking at IV/OV/EV certificates (if any) should tell for sure. Thanks, Peter |
| Re: How do you handle mass revocation requests? | timx...@gmail.com | 28/02/18 11:14 | We have purchased thousands of certificates using Trustico as a reseller within the last years.
Back in these days Trustico created CSR / Private Key pair within their online platform (Yes, you read it right - you can create CSR/Private Key on their webpage !!!) which was the default at this time and it is still possible to do so in their web interface. Regarding to our investigation they were only able to send the private keys for those certificates where the CSR / private key pair were generated within their online private key generating tool. This has to be the 23k amount of keys which Jeremy received. I am not aware of guidelines of the CA/B forum but keeping 23.000 (!) private keys at your online platform seems more than alarming and is careless and the public should be made aware of this fact. We do not know all aspects of parties involved but we suspect that this could be for economic reasons, because Trustico is offering a voucher for Comodo certificates in order to replace the revoked RapidSSL/GeoTrust/DigiCerts certificates now. Furthermore they are spreading FUD regarding the upcoming distrust in Google Chrome release and trying to force their customers to get a Comodo or Trustico-branded certificate. Very important: Do not revoke any certificate where Trustico is not able to provide the private key!!! |
| RE: How do you handle mass revocation requests? | Jeremy Rowley | 28/02/18 11:21 | I believe transparency is the best policy. I think it'd be helpful to the community if we could post the email exchange about the revocation. We can redact the agreement termination portions if you'd like, but that'd give a lot more clarity around what's going on. Do I have your permission to post those emails?
_______________________________________________ dev-security-policy mailing list dev-secur...@lists.mozilla.org https://clicktime.symantec.com/a/1/agKxz5YN2oS2j3aBKGdI3rd5u57u8Xu5UsUNv9qPbXg=?d=KSM3YiopOvVUIKUCYWim2psjXx0oJUTN3GJYVvv86cH2hgAOuX0A4yUCYWJCrf8DxvICOyaAgSOSsQuBhgT-fwrAhvYJcuiMS5v21RcS10hnpz5YP_t7wqIWFHDiFcsXTRaUzlEsq157OU9akewIgfvDdgoovCCDhFXMsSO0Qz9-IsjYuLcY2r5XBg_U0za-RAQVdms513WgdS689UdoNpphZVMIsrO9rGs_qyRWVeAdA8xwkhsRgfAw3SP_TS7sCdki2kvz6t5uiO6j-BwoRUjAoVFRQtmxKYxWnvnGTjkZ-wHx9WGaqLsiP86sx_Dn1ve-_wY3O6scnyKHEz-UebwmKoH_wtXjc6f3fXxsTzV6Q7aQSiD21hpz98AC9_ZI7lf3ZXhg--cecs5fBcwAQf0XYkYpeAQqfp1i3UtBpVvxuFCa6LzeG0ONfbF_i08rNP8986U0v0nJ54W_m0eI&u=https%3A%2F%2Flists.mozilla.org%2Flistinfo%2Fdev-security-policy |
| RE: How do you handle mass revocation requests? | John Merrill | 28/02/18 11:22 | Let's talk it through with Mike J as this will end up in court
|
| RE: How do you handle mass revocation requests? | Jeremy Rowley | 28/02/18 11:28 | I agree with the OV, EV, and IV. Admittedly, DV certs, which constitute almost
all the certs, are relatively new to DigiCert so that's partly where the question arises. We identified it as the key holder or the domain holder. Hence, we'd revoke with confirmation of a domain validation. The reseller could be the subscriber, but I'm not sure how we tell with DV certs. This is especially with legacy Symantec customers where we are still trying to establish the personal relationship and understand their use cases, communication expectations, etc. |
| Re: How do you handle mass revocation requests? | Wayne Thayer | 28/02/18 11:30 | On Wed, Feb 28, 2018 at 12:13 PM, timx84039--- via dev-security-policy <> I agree with this sentiment, but I also think it creates another policy question with respect to DigiCert's decision to revoke due to key compromise: were these 23,000 keys really compromised? The BR definition of Key Compromise is: A Private Key is said to be compromised if its value has been disclosed to an unauthorized person, an unauthorized person has had access to it, or there exists a practical technique by which an unauthorized person may discover its value. A Private Key is also considered compromised if methods have been developed that can easily calculate it based on the Public Key (such as a Debian weak key, see http://wiki.debian.org/SSLkeys) or if there is clear evidence that the specific method used to generate the Private Key was flawed. In this case it might be reasonable to argue that Trustico was unauthorized (unless their customers agreed to key escrow when using the online key generation tool). However, in the case of a hosting provider reselling certificates for use on their platform, it's required that they hold the private key and we don't consider that a Key Compromise. - Wayne |
| Re: How do you handle mass revocation requests? | Alex Gaynor | 28/02/18 11:32 | I would say that at the point that Trustico emailed them to DigiCert they
necessarily became compromised -- while Trustico may (or may not) have been authorized to escrowing the keys by the subscriber, the subscriber did not authorize them to be emailed around, presumably. Alex > _______________________________________________> https://lists.mozilla.org/listinfo/dev-security-policy > |
| Re: How do you handle mass revocation requests? | Peter Bowen | 28/02/18 11:37 | On Wed, Feb 28, 2018 at 11:29 AM, Wayne Thayer via dev-security-policy
Jeremy's email suggests that the keys were emailed to him. If this is accurate, then it is reasonable that they have been "disclosed to an unauthorized person". The only other alternative, again assuming Jeremy did receive the keys, is to determine that he was authorized by the subscriber to access the keys. Thanks, Peter |
| RE: How do you handle mass revocation requests? | Jeremy Rowley | 28/02/18 11:41 | The end user agreed to the subscriber agreement, not Trustico. Our analysis follows what Peter B. posted – the subscriber is the “natural person or Legal Entity to whom a Certificate is issued and who is legally bound by a Subscriber Agreement or Terms of Use"—which in this case was Trustico’s customers. In addition, we felt that given (1) the number of certificates Trustico was asking us to mass-revoke and (2) the lack of any substantiation of why Trustico thought the certs were “compromised,” we needed more information before revoking. At the minimum, it warranted alerting the contact for each certificate that revocation was imminent.
I also agree that there’s no problem with the way or that the keys were provided to DigiCert for cert revocation. I certainly don’t want to discourage revocation of compromised certs! My main question is how do you handle these things? The process at scale should not be different if a reseller requests revocation compared to any other security researcher. The question is how you handle the this volume of revocations when its happen? I’ve never received a revocation request of this magnitude before so I’m seeking the wisdom of the community in what we do. I’m happy to share any of the details I can. Jeremy From: Ryan Sleevi <ry...@sleevi.com> Sent: Wednesday, February 28, 2018 11:58 AM To: Jeremy Rowley <jeremy...@digicert.com> On Wed, Feb 28, 2018 at 12:37 PM, Jeremy Rowley via dev-security-policy <dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org> > wrote:sharing the concerns and information. Once we were alerted, the team kicked off a debate that I wanted to bring to the CAB Forum. Basically, our I think there's a little nuance to this in the general case (e.g. consider "Resllers" who are also the hosting provider, and thus constitute the Applicant/Subscriber even when they're not the domain holder, but authorized by them), but based on this specific case, I think this sounds like a correct determination. This raises a question about the MDSP policy and CAB Forum requirements. Who I think the question here is less about who is the Applicant, but who is the Applicant Representative. If the Reseller is signing/agreeing the request on behalf of the applicant, or the Subscriber Agreement, they're the Applicant Representative and ostensibly should be taken as authorized.Consider https://crt.sh/?id=245397170 , which was a google.tg <http://google.tg> certificate obtained from LE following an apparent Registry compromise. Prior to the compromise, Google operated google.tg <http://google.tg> , during the compromise, an unknown third-party did, and following the compromise, Google again operated/controlled google.tg <http://google.tg> . When it came to requesting revocation, however, this highlighted a challenge. Let's Encrypt has several defined methods for validation - the HTTP-01 method, TLS-SNI-01 (at the time), and DNS-01. Google services - particularly those hosting/serving google.tg <http://google.tg> - do not support any of those methods (by somewhat intentional design). Google also did not control the private key - naturally. |
| Re: How do you handle mass revocation requests? | Ryan Sleevi | 28/02/18 11:56 | On Wed, Feb 28, 2018 at 2:40 PM, Jeremy Rowley via dev-security-policy <Assuming Trustico sent the keys to DigiCert, it definitely sounds like even if Trustico was authorized to hold the keys (which is a troubling argument, given all things), they themselves compromised the keys of their customers, and revocation is both correct and necessary. That is, whether or not Trustico believed they were compromised before, they compromised their customers keys by sending them, and it's both correct and accurate to notify the Subscribers that their keys have been compromised by their Reseller. If your Reseller compromises your keys capriciously, it sounds like it might be time to find a new Reseller. For the remaining certificates, it sounds like, based on the evidence, that Trustico has no authority to request revocation for those remaining, and it would be monumentally hostile/unwise of them to subject that their customers to do that. To the more general case - how to handle mass revocations - this seems similar to when Heartbleed hit, and the lessons from then apply now: - CAs should design their systems and be prepared for mass revocations on sufficient demonstration - Sharding out CRLs, for example, to ensure no CRL becomes unreasonably large (say, >64KB, which is many clients' limits), using critical issuingDistributionPoint extensions on the CRLs and hosting at unique URLs - Designing the OCSP serving infrastructure to be able to handle a large influx of revocations - Reducing the overall lifetime of certificates to minimize the total number of ongoing (OCSP, CRL) signatures that would need to be produced - Site Operators should design their systems to be prepared for sudden revocation - Don't introduce unnecessary third parties (such as Resellers), when possible - Invest in automation such that certificate replacement is automated/automatic - Corollary: Have multiple CAs support the automation mechanisms used (such as standard automation methods) in the event it is the CA compromised - Corollary: Support multiple automation validation methods (such as DNS + HTTP + TLS) in the event it is the automation method that is compromised |
| RE: How do you handle mass revocation requests? | Jeremy Rowley | 28/02/18 12:04 | The keys were emailed to me. I'm trying to get a project together where we
self-sign a cert with each of the keys and publish them. That way there's evidence to the community of the compromise without simply listing 23k private keys. Someone on Reddit suggested that, which I really appreciated. I think the main hold up in this plan is, which websites do we want to call out?
On Behalf Of Peter Bowen via dev-security-policy |
| Re: How do you handle mass revocation requests? | Ryan Hurst | 28/02/18 12:24 | On Wednesday, February 28, 2018 at 11:56:04 AM UTC-8, Ryan Sleevi wrote:That seems to be the case to me as well. It also seems that this situation should result in the UAs and/or CABFORUM re0visit section 6.1.2 (https://github.com/cabforum/documents/blob/master/docs/BR.md) in the BRs. Specifically, this section states: ``` Parties other than the Subscriber SHALL NOT archive the Subscriber Private Key without authorization by the Subscriber. If the CA or any of its designated RAs generated the Private Key on behalf of the Subscriber, then the CA SHALL encrypt the Private Key for transport to the Subscriber. ``` In this case, TrustIco is not the subscriber, and there is no indication in their terms and conditions (https://www.trustico.com/terms/terms-and-conditions.php) that they are authorized to archive the private key. Yet clearly if they were able to provide 20k+ private keys to DigiCert they are archiving them. This text seems to cover this case clearly but as worded I do not see how audits would catch this behavior. I think it may make sense for the CAs to be responsible for demonstrating how they and other non-subscribers in the lifecycle flow handle this case. Additionally, it seems if the private keys were provided to DigiCert in a way they were verifiable by them they may have been stored in a non-encrypted fashion, at a minimum they were likley not generated and protected on an HSM. The BRs should probably be revised to specify some minimum level of security to be provided in these cases of for these cases to be simply disallowed altogether. Finally, the associated text speaks to RAs but not to the non-subscriber (reseller) case, this gap should be addressed minimally. |
| Re: How do you handle mass revocation requests? | Matthew Hardeman | 28/02/18 12:29 | I would echo Mr. Gaynor's point.
While it's perhaps a pedantic distinction, the private keys are definitely compromised now and were the moment that Trustico provided the keys to Digicert, even if Trustico is defined to be the original authorized recipient. The CA is explicitly not to be in possession of these private keys and if Digicert's assertions are correct, Digicert now holds the private keys. It would seem that revocation is unquestionably the appropriate remedy at this time, despite whether or not Trustico understands what they requested. > On Wed, Feb 28, 2018 at 2:29 PM, Wayne Thayer via dev-security-policy < > > - Wayne |
| Re: How do you handle mass revocation requests? | Matthew Hardeman | 28/02/18 13:08 | Did this whole thing start because someone at Trustico wanted to accelerate the process of getting their resold Symantec certificates reissued under a DigiCert trust path?
And somehow some misinformed soul imagined creating a revocation crisis would somehow help achieve that goal without significant consequences? If so, that surfaces a significant potential problem with the reseller ecosystem, which as I understand it has largely gone unaudited as yet (providing the CA actually provides the validation functions). It shows what havoc a reseller can cause from poor judgement. Then there's the issue of a reseller holding onto the private keys indefinitely... |
| RE: How do you handle mass revocation requests? | kevin.b...@gmail.com | 28/02/18 13:22 | It’s absolutely incredible that Trustico has 23k private keys, and just attached them to an email. This suggests serious flaws in the CA/reseller relationship.
|
| Re: How do you handle mass revocation requests? | uri...@gmail.com | 28/02/18 13:23 | Is Trustico's storage of private keys related to this security report from a few months back (which did not appear to ever have been fully investigated...)?
https://groups.google.com/d/msg/mozilla.dev.security.policy/CEww8w9q2zE/F_bzX1guCQAJ Does Digicert have (or will it have) some sort of process in place to prevent resellers from storing private keys so casually? |
| Re: How do you handle mass revocation requests? | Nick Lamb | 28/02/18 13:37 | On Wed, 28 Feb 2018 20:03:51 +0000
Jeremy Rowley via dev-security-policy > The keys were emailed to me. I'm trying to get a project togetherThat's probably me (tialaramex). Anyway, if it is me you're referring to, I suggested using the private keys to issue a bogus CSR. CSRs are signed, proving that whoever made them had the corresponding private key but they avoid the confusion that comes from DigiCert (or its employees) issuing bogus certs. Everybody reading m.d.s.policy can still see that a self-signed cert is harmless and not an attack, but it may be harder to explain in a soundbite. Maybe more technically able contributors disagree ? |
| Re: How do you handle mass revocation requests? | Ryan Sleevi | 28/02/18 13:41 | On Wed, Feb 28, 2018 at 4:23 PM, urijah--- via dev-security-policy <It was fully investigated. There's no evidence to suggest relation, other than there's no way to effectively make rules on that (other than the threat of revocation for breach of contract, which you can only demonstrate when you can prove it, which you'd revoke anyways) > > https://groups.google.com/d/msg/mozilla.dev.security.policy/CEww8w9q2zE/F_ > bzX1guCQAJWhile well-intentioned, I don't think this is really a reasonable thing to ask/expect. If they had contractual requirements on resellers, it just means you'd have new entities spring up that are equivalent to resellers without the requirements. Would we similarly require that hosting providers don't store the private keys? It's the same conceptual issue. |
| RE: How do you handle mass revocation requests? | Jeremy Rowley | 28/02/18 13:43 | We don't have a process to prevent third parties from storing private keys.
I'm not sure how that would even work considering the approved third-party use cases vs. non-approved use cases. In fact, I'd postulate there's nothing wrong with Trustico holding the private keys if they were hosting the site or providing CDN services for all of these sites. The issue is Trustico alleged compromise of the certificates and sent us the private keys believed compromised in support. There were a lot of them. This is a mass revocation without any explanation of what went wrong or why. The reseller mentioned and proved compromise, but there's no way to see if what happened, whether the issue was mitigated, or how it's going to be prevented from happening again. Basically, we're revoking 50k people without being able to explain why (well, other than the key was compromised).
On Behalf Of urijah--- via dev-security-policy Is Trustico's storage of private keys related to this security report from ahttps://clicktime.symantec.com/a/1/gUJtIwxyRazHoxd9FIcS40cx3EykeGjrwv4urOpHj u0=?d=-1kl6pTbi8aU0HfRh9ZNP86TCSEsjm5TvumKZsLlfO6tgKV5N4_MitDAK64FGwutmiXu3D eTelMEfv3ep7X8kj6GIwMY8tUVM_WKGkqW_uhm0fHWgY9ZwlUytCHVYb_1WWceLMVi9w9Ec7h3sH G1wIJHWNk2L8RcVycGdrIGRFvV76QXqOck9szIT1MqOPD6gSaSZRnxp-ItDtBU5xSit2RwQjv5sB Ln2qFtnD9T_wuM6-KAeTUDm1O51uIY3NpT-q5hyIKG5YG-Ds6a9ceM0JgMrLhM3evdd-BmO3e6eP 1fXZXAQgJCZ_A7SnTbSagjzHzUVC65PN3AnrdJ6ANhZAwjjCdHdbjOeIqwInT-oFJ8lufSzaBeui OECQuf-IqhCnfsam6KnTGT5k-aQlG_BjOW8fny0TS21Upa4k4aX2Xm7es0oRfOUS-OxAQLVKCGg- cER_Pj3TmyKGPFFw_SzUky&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsg%2Fmozilla .dev.security.policy%2FCEww8w9q2zE%2FF_bzX1guCQAJ
_______________________________________________https://clicktime.symantec.com/a/1/HzhUTcKY59cs-68Bx4Zcxt9mxYoEtI12E4LdF4LDI mk=?d=-1kl6pTbi8aU0HfRh9ZNP86TCSEsjm5TvumKZsLlfO6tgKV5N4_MitDAK64FGwutmiXu3D eTelMEfv3ep7X8kj6GIwMY8tUVM_WKGkqW_uhm0fHWgY9ZwlUytCHVYb_1WWceLMVi9w9Ec7h3sH G1wIJHWNk2L8RcVycGdrIGRFvV76QXqOck9szIT1MqOPD6gSaSZRnxp-ItDtBU5xSit2RwQjv5sB Ln2qFtnD9T_wuM6-KAeTUDm1O51uIY3NpT-q5hyIKG5YG-Ds6a9ceM0JgMrLhM3evdd-BmO3e6eP 1fXZXAQgJCZ_A7SnTbSagjzHzUVC65PN3AnrdJ6ANhZAwjjCdHdbjOeIqwInT-oFJ8lufSzaBeui OECQuf-IqhCnfsam6KnTGT5k-aQlG_BjOW8fny0TS21Upa4k4aX2Xm7es0oRfOUS-OxAQLVKCGg- cER_Pj3TmyKGPFFw_SzUky&u=https%3A%2F%2Flists.mozilla.org%2Flistinfo%2Fdev-se curity-policy |
| RE: How do you handle mass revocation requests? | Jeremy Rowley | 28/02/18 13:44 | Yep - that was you. Thanks a ton. We posted 10 CSRs so far. Is this what you
were thinking?
|
| Fwd: [cabfpub] How do you handle mass revocation requests? | Jeremy Rowley | 28/02/18 13:50 | Posted to cab forum accidentally instead of Mozilla dev
Begin forwarded message: From: Jeremy Rowley <jeremy...@digicert.com<mailto:jeremy...@digicert.com>> Date: February 28, 2018 at 2:33:41 PM MST To: Ryan Sleevi <sle...@google.com<mailto:sleevi@google.com>>, Geoff Keating <geo...@apple.com<mailto:geoffk@apple.com>> Cc: CA/Browser Forum Public Discussion List <pub...@cabforum.org<mailto:public@cabforum.org>> Subject: RE: [cabfpub] How do you handle mass revocation requests? Here’s 10 CSRs that people can correlate with the CT logs. I’ll create another 100 or so to dispel any doubt. -----BEGIN CERTIFICATE REQUEST----- MIICwDCCAagCAQAwezELMAkGA1UEBhMCVVMxDTALBgNVBAgTBFV0YWgxEDAOBgNV BAcTB05ld2J1cnkxDzANBgNVBAoTBkplcmVteTEgMB4GA1UECxMXUHJvb2Ygb2Yg a2V5IGNvbXByb21pc2UxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBAMamagziY67jAV1XWBT2UBudz8leqPeZ nMGCP9Sct2qc5tDDLz34QIMFO9mv4eDRduMOTG7rwQygKPvI0mAKzKDJCxUvKLYJ Th/IALgkHfSvv7UzXUxF2kxviuvTCoP0Oee3DUJl4V614R2BnEtEpjEC4WNZglIU ytQiX36u4WQEbU1LGxp16+Rqz55TOnqRaUNlCCVjB99A3dvxYxpa+6qUHt2aeEyW WBguBq4sDzOeeLCcnfiCywbDKD4YeqQxvn1EJGBrCQnOf5UdHidJB3ZKKngYWKaZ 48nYK2CqM1dq44vIH6EezGq+0Xs8EFJfi2mjDghfziiX1UtpUt2/vUcCAwEAAaAA MA0GCSqGSIb3DQEBCwUAA4IBAQA/HM907Hc/rV+olpHW8n1N0UkhfbVHjSegFhQZ 2Wn4jFKAargvOCGDChThcnUbquptKpTFVaKJap0JB2T+fCI3WAMPG8CJKakcOjG/ ZKheN3fNGUiaNk/Wyh/f6XnhWbchIK2OpcsSUAA5ju14bqs2epWl17c0MBPVY5sJ wFIBrNVjqji3Zkf7aE9GSMx36d8swfhqwomvFvO5SGKspPm2eRpBPwi8lRORDxz3 cAoG4TU3/7xs2XAyTE27UQwdrUo7jkVUlCFoWf6DySHEy7CKTz3Vwu9ABtNdBtG9 oEnWxhIhBPcYOwCrGkiJGuRImFAfLifHWvLUNqGKpz/nEr+c -----END CERTIFICATE REQUEST----- -----BEGIN CERTIFICATE REQUEST----- MIICvDCCAaQCAQAwdzELMAkGA1UEBhMCVVMxDTALBgNVBAgTBFV0YWgxEDAOBgNV BAcTB05ld2J1cnkxDzANBgNVBAoTBkplcmVteTEcMBoGA1UECxMTUHJvb2Ygb2Yg Y29tcHJvbWlzZTEYMBYGA1UEAxMPd3d3LmV4YW1wbGUuY29tMIIBIjANBgkqhkiG 9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqS0TSQbsGiJdAYfhuNrGzvXX4XvwToLBq+Hx trxKq8zoWIRtimRuH66uRmVy0I/lR78u4FEewAjblaS+v1jTLNopik+taBiFHudn /RGliOcKFohp4BYZuSZRRt3uLN5z8Jr5VbRMzZy6SNp3wX3f+Ie/XsAV04TXmbSv +V8ZUjxzj1448DsCFL1NNDUcoic/MUcW+fsu9VuxhdETOwrf7CgJ91FgzwHHSMKL Mq6DwY6xF90KQ5vInhYhRQ447zoSW1ABnmF+gPxDjXXb5pCh4aua8wOv3AmiZbbn Nnkv9y1YXIeBJ1o1zbTt51v62Qu4LeRYVfWjhI1sn8k96DH8mQIDAQABoAAwDQYJ KoZIhvcNAQELBQADggEBADgA4tGSyIpAA9uXooQivo9NH7lZH+M4bXpn+nOmNxn/ aRzmbg9NksRKrQoN0/CkWpiu6vwp31gAG2eIpnvNX9ltzPD2/yHAQCLUZmiGZnUP fUdV1t7Z1EZ9Mj7YmlAN5NuQPAu7SL5fZ8UJSzzY1H7AuECU29j81dK2jLxRR1p6 PaajUGPAvraVTZND4JGQJpIazrF+mVDADdt1aOntr6lj+CC92E5oQxCWtU8uUX7Y k5OJdewmNlVIk7wtcuVA2ju3jFlNtHP66DE/UDlcx5X5vE2qFq3aZFAqUAf0XXWW bagqqjxfHSQaNVGlWBkJb0eCdD+DW8IK0nuw5GP2rzY= -----END CERTIFICATE REQUEST----- -----BEGIN CERTIFICATE REQUEST----- MIICwDCCAagCAQAwezELMAkGA1UEBhMCVVMxDTALBgNVBAgTBFV0YWgxEDAOBgNV BAcTB05ld2J1cnkxDzANBgNVBAoTBkplcmVteTEgMB4GA1UECxMXUHJvb2Ygb2Yg a2V5IGNvbXByb21pc2UxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBAMfZnCmYERBmZPEMdK5yvvC0QXjB1FQo bu4IC9W1ThdFySkJL6t2MeoxbaTW8hwdATf8JHlRnrpevXStUsYDZShxpsQePZk8 LUW5OHQ6W/XDbGo4tC69Q3XTYvLeX+3q971mbiyFpBondTKZaKZYAR5omV7e/Olt ZzN1yqyX551NsSwDGTrgBqCU2XQzYu9Tl9Nibjl4yCKCG9/JbgGa+gy8j1eKWxIt lW7jmJID8s0N2v0ed1lvxr7A3oCiYef4TSi2RjLyZThGw1a7j3QqlBeGIqo89XtG 8GxA2nNSGx7gnIceMGGUd0q9iGQl1gc+FOI5oBbyjLcMCty2YInMQucCAwEAAaAA MA0GCSqGSIb3DQEBCwUAA4IBAQBQyDCiBYXRMOryHhVVW52kY6wvsgWsPXLR0piQ FJKP/drPdqgs/uxY6Nx163sNaiWPEI0tUWSmnuPe43qAyIqJXmnd/C6EqGy+ZI1Y lf0XAfqVzJ+tc9639pSkGfeGxU75qdPwWqbwzEEdNZCDR/4QqzhGgMLyvH7icoc0 7ikxxwyUiKpP4h3nAV7Fg4EMKeEn/3m+vM0aGnZNx16WHpQF/VnyBM8NimADmO/u vywC2TbLNBYYYG7dlLUk7fYfoFY5okel4z2fjmaNhuQQEpffJ366DC41A3fWDgp8 j1Ok8PfhiVySEwdSCNgpZbHnSwxodk4E3aZ22kCa2f7nOzDd -----END CERTIFICATE REQUEST----- -----BEGIN CERTIFICATE REQUEST----- MIICwDCCAagCAQAwezELMAkGA1UEBhMCVVMxDTALBgNVBAgTBFV0YWgxEDAOBgNV BAcTB05ld2J1cnkxDzANBgNVBAoTBkplcmVteTEgMB4GA1UECxMXUHJvb2Ygb2Yg a2V5IGNvbXByb21pc2UxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBALjEWXaHdMifi0LbL0GrnYV6uoltTicU ywDOkSz/cReCI6gzxb1jpuQu1iVAkmiZUmPk4sPjc4e/OvAo0IyXgVEqBVcB4cmB JTdWFj4fde8G9/NWoKIHIRVS4envx4jjRFEh6uDI9o4pDcpfvjh59s1RCjqU9EqX DueUoKCk7eynjBxePNYZtZL5xBbBRIeqkjtBALtdnQdkbMTBAHJT4WvK2y9ExtWY aJAoAf0ZYmQOeqzXZC4w1FKs7/GEa1qaPjyxe596LvZvOMZbB49gUYou6lhmG+ga PaJIGm53/A4PgLnCisEjrVL46YB/k/EzTQtwq5jRg5usIL0/YCJlvCkCAwEAAaAA MA0GCSqGSIb3DQEBCwUAA4IBAQCgCziprbbD+aS22QHBMOnjy5r7iYiteKO1uB1o zdaKpNBg32+tNyYsNazijAb0rcyFLGAeTfjbWQ5YDbK44qGmKxhO5nyeUkb9/ulI scT94Trwu8j77DTxaFNTziETbw5KIBfiC7M5cD+vQ2UexJ8giv9s7ZchXY/hK8TA IPY1jfEzioEgjap2bXhZ7GGHhjNg/3DIHCy2dD+eDeTsHhZQ+4ndfMeydg9fn6se 20A73X5rFGYKtp3z19stTDjXFDVyf/ngXtyti830ebQgmxRLJRAKV+MSHrdxW4Jj qkuW6fmTj2s3x8iTKecd5Q/NOKt2XjOMldc6eKA4VSi3QNuc -----END CERTIFICATE REQUEST----- -----BEGIN CERTIFICATE REQUEST----- MIICwDCCAagCAQAwezELMAkGA1UEBhMCVVMxDTALBgNVBAgTBFV0YWgxEDAOBgNV BAcTB05ld2J1cnkxDzANBgNVBAoTBkplcmVteTEgMB4GA1UECxMXUHJvb2Ygb2Yg a2V5IGNvbXByb21pc2UxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBALlY3wwEZcO3U4HGuYE6atvaG3vOiOGq y1W1Nwv4xVCiVkTECgbumYZyBjq1XxVKC1dNJ2nzxDSIhPNxnJZHA7v5SvSf60+Z 1W+QmvznlRfqptKNt9L26LCRAFppjfT5Z0F0fw300e7NawkWKxNyujDsltpFrkNP 72SvHWizvMpySx5aclAb/TD6iAY1NQh1PUVdeCJZMtZreD+v3UOKPsnztdRgYe/f FbPRYQaxAaKYm0oYUZ0x0kurTjDdGQHtm/0H253KPDFHiC9bWCqljFTqUFT2v2TG 2m04pv+wMLFIt8FpQwzi7M7v0O1TD4hBLswUGSAmCxu6fOaMJFUAM1ECAwEAAaAA MA0GCSqGSIb3DQEBCwUAA4IBAQCnl8UUGLPd76FgaBtwZo9cktY1reJtImP8/613 JWwvxCWy5r46LqD9BDPZh4yqO48FLeoaep0+CuCBKjGHQf/xZzpb4USnHcyAZKpR Ey4SVlNpHczszoNZoUYdiL2kvWo0NvD7+oF+O9lE3rvxkNk4tRrfe8/xMq6rhhlC NzrM6vdR4tIJlVtqQ7j60bMsRQJLZ8rz4Lb36R2JItbXeckWiGwXjv3Bi2r/MxaT x7Pvj3oB+amqPg2Muk+HuKvL/s5o7mIBeayt/TRev6a9YeamxhcGxINEjH52uIII NKQS007WuE//OAJcWYpCphJ3pPDu2gUbBd2X7vX+JiYO8nhO -----END CERTIFICATE REQUEST----- -----BEGIN CERTIFICATE REQUEST----- MIICwDCCAagCAQAwezELMAkGA1UEBhMCVVMxDTALBgNVBAgTBFV0YWgxEDAOBgNV BAcTB05ld2J1cnkxDzANBgNVBAoTBkplcmVteTEgMB4GA1UECxMXUHJvb2Ygb2Yg a2V5IGNvbXByb21pc2UxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBAL3v00AFGsfyCQWVL2K/EHPLS8hh5vqv hC+WJm8E0m1uVmc08tEwPGfW5+nxHTw4Fav8hlhObfOt/KVx2Z1TsBxzPzM89amw o7jzx9dpll8v80ueoSE7UUzhYZ5OirRc2q19d1aC9S4Ji3PyGtOiG+kxvabEe/Gf YkrJGpOaTB02wa3M3mBGdd3oCCwHLfKB7ylou5W3m6t2GEQjCYUJnL9gUF9rgIo6 oGxVpEu0fGKTpfjqaM1yX71SWEemvCfbahq8F1L+xwONCl9PQF8JvOxD3L7Mtj3N 04cIQIMdIhqthJ88ciaxj5mEN6BONf2oIExw+qJv0pusP0S/DI+/TSECAwEAAaAA MA0GCSqGSIb3DQEBCwUAA4IBAQBE0TbWwA8nuVkuwgvOJMTFosp8/ufoAlKDSNlN hEKf1sJSlPRmTLpq9Rv9fwpnCbFFE7UN6UuDAYGKOIuwaqad2iTj/t75IYC0jYCd l5fLf7hWhnk9iRYufT57iM2Qdmh4zHZxNjuZb+qYAejXoehPmbQgpVVXBB4Vf4I7 iqY7vMF3AxhNxBmGaCvrWEChjw6DPoYWca+tcUUi/O1P5NPOCjbNrRa5c4AT18nD WOhFafjmGq9OeuWKfXjZDeRe53fys0nVeXfpJ20QbGRyW43/6Oj9sVwgxaTu2OXV 9AqboNdzQ1vP63VVT+X2KoVES8YheQ+AhOi1grhi9m9J4Fer -----END CERTIFICATE REQUEST----- -----BEGIN CERTIFICATE REQUEST----- MIICwDCCAagCAQAwezELMAkGA1UEBhMCVVMxDTALBgNVBAgTBFV0YWgxEDAOBgNV BAcTB05ld2J1cnkxDzANBgNVBAoTBkplcmVteTEgMB4GA1UECxMXUHJvb2Ygb2Yg a2V5IGNvbXByb21pc2UxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBAK9NrWXf1zy+R6rkObfXJtS5F+VFtGA1 SmDakjg4pNqaBUAo9fQReRmjA1SeO6qJQLJLAFtSE7a9oTiSlPlTiq/SdYe6FNfg uXRR34kAaCPdXMyBfRq7c5oooRXRHQL4ZCNrPU1Kpcf+XsLIKUX1WQGQBjnPHJhX QwTdDDB93FXO/d+vyAmdIycrqDtUYRQPaGR0nCyHCsdioNhZPjH2QuANTFWtEtXY O6KKO2AlZrNahflRQcO2YKvH0VDHK8FmibKPDkvoOAo/f/LDn15Uc5ecrLQXl0Lu RJMnSGH0B1A13ls6RU5TsCcqVvbUseSSOQJPOxdbGFGTS7jWXSxEdD8CAwEAAaAA MA0GCSqGSIb3DQEBCwUAA4IBAQBVAGhyg1C+kcEWKc1aQEU+iOwtLYWrLJIxaywX Z9GDMRd0Sy0E/UkciOLWvU/TZybJDZRyoXPmsTZsXMZ1fAzkB8ZPb5rGtin95jud 4YePcamJGf3N5AgHD2UmGobn3jbvoC2pGVivGTxTdqcCQJWWdSVXxzw6nH5KcQ/O UuOiSmVVvJOD8+oxt4U3XVal6VtumtuhGQSl4wVpC+xAoz/o7UbToBp/WhS1Xlhn DjJWuFIRdA8TgieTaYMaLdLNWQkYVY3EYMUPwkO2mPpqNuRGlFbALcRMKoqv7O+A VBKcstS41uwsjU1v/5zgC2Po5w0ocbLaXRe34In+Jr3HCMPc -----END CERTIFICATE REQUEST----- -----BEGIN CERTIFICATE REQUEST----- MIICwDCCAagCAQAwezELMAkGA1UEBhMCVVMxDTALBgNVBAgTBFV0YWgxEDAOBgNV BAcTB05ld2J1cnkxDzANBgNVBAoTBkplcmVteTEgMB4GA1UECxMXUHJvb2Ygb2Yg a2V5IGNvbXByb21pc2UxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBALHvGriP3eFIhYXPP9LB2h17Rn9gaXh/ zG2gsiPjklF2JSTRoAAKQDyrmREsBP2qp8pkdbaSudPNfGZb1beBB9gw4fToiWYc PnAhi0aH2WrEE0U2wGQcOTMpzLhP1y5xyckiBybkoJnuUWxxyyO5DQbMDGEOYRdG pA9BHIg1oOKnTKFGgq1lvZrFqNYhZI4rhBSs1PofYr1tfSTlAEC1pWCKPWb4crB8 8z1MDwG7c/UEe8qMPGxnTKBtpYszDVaMzH7AV4PZm0M/PZqpZOUA3IoBysi/st59 ISUGhC3OTjz76/92Ki7a5EiGgxqO24A0+cPpC/u538OfZuJp269pXdECAwEAAaAA MA0GCSqGSIb3DQEBCwUAA4IBAQAOIBi1VCs9voFdKDWfB2algBka2GHEL+uCTUl4 r5ooCJUafP69S0YFu9y1XQfIwSSioxHer8st1P1qGXc62LHtRRWk6GHKQ+/8pPiL U9gXsGFovV2K5tMz5laTLrwWCpZ9OgCk3EemZiFSVjCvOnaOC2PKTK5VS018joTJ xzbnHuzhS4yNwPtUz3VR6zULk2+tRHCYGROt6YuKwDJmSLO/qQTveBEh5QklQ3zs 26/1j9mHirOQZIo0kgBfere4lkCzeweiO8rPJXJNhhMjCRbDP0lKscXzQH/8OJLf Y5W5uOfsoGp9Tnc36mcHIHhxfwLb+SDyB3FKsX6kFCp62vuQ -----END CERTIFICATE REQUEST----- -----BEGIN CERTIFICATE REQUEST----- MIICwDCCAagCAQAwezELMAkGA1UEBhMCVVMxDTALBgNVBAgTBFV0YWgxEDAOBgNV BAcTB05ld2J1cnkxDzANBgNVBAoTBkplcmVteTEgMB4GA1UECxMXUHJvb2Ygb2Yg a2V5IGNvbXByb21pc2UxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBAMNFOlVBa9P2Vvb4CCsgJyvUZxYaRlK4 SEPH7z4cvIIu68p4yi8aaMkFs4VaPg/2PtIGFHNQjnj4x2URg+7ocf3U5qko9W0W ZMeM5QU9i7lHfZnGQ+SQ4q+AFwzjSbwO4wdTzPpzjJdUKmtOguILT8j/0uTn/Q/W PylzfmdgzPU0BUUTBuz9uGzHY3480ctKib81ZvALSwSX++Qeh+QCrueJZgS3hSYf dgZCXUN85+tN+xCLXNg/YwsHa6eJoBfBaUW7aokjtRhl0A+SLsyfc7agbnvhJzHO NywHa4SD9pRwhbdaO5/zKcgCDUlrJQnDWojc8qiAcEvmyrhAJhzdxjMCAwEAAaAA MA0GCSqGSIb3DQEBCwUAA4IBAQBtN41DLhpzP82LZOXE0Dk1gNzUWHAAt85GQoOU lcN3jrycJFarFmliz4R46uQzvqfhPpaBb5z9VyJP0UzQVoHhy37fkOLarnVkHx/n JPBMZ4cCoDHQLsVYs7bt+hjzGbaWPY5G/NkYiGaOL6azVCMj3wyS1lIbrywHGPqO JLjBMNUUyiXpVqmkBgPv1j7dwgdew9uxwvGO7OMuZuF5vv2gCRZoz7sBiyd6TL6e JvvnK3Rd5/PlYy2dLHpMdRNtg2YZzQXhyJkTxdSrnbgoc0SVaMebuI9HIOmKkF0A 1ftX4BfaPeO4nuQRq7jZ9sfMZUVp8nOAOdb13U4p43s0Cvqy -----END CERTIFICATE REQUEST----- -----BEGIN CERTIFICATE REQUEST----- MIICwDCCAagCAQAwezELMAkGA1UEBhMCVVMxDTALBgNVBAgTBFV0YWgxEDAOBgNV BAcTB05ld2J1cnkxDzANBgNVBAoTBkplcmVteTEgMB4GA1UECxMXUHJvb2Ygb2Yg a2V5IGNvbXByb21pc2UxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBAMR5uqfSJeKarT1ZgGT+/2pwYWnmllEU Yy5Tc33IoIV4FL9nr45rtBnyY0pXHUc1zqjQZspoAynFJBOY3NB2zBf6CQb3/2R7 mry8PxFlJDXc9M2iQLHpbSQq+P8ZOZ8mXdFRThtW5Xl9orLyGnFBXak49MF4HuyR ff492p+otlFrpzXIAIV4dRnkKxeKDCvPIHLeUHG3slgY/Gs6vdj88Akp09EY7pHn AGiEAuJPTzINJ4yLXJJLRH18VXIlg/35KeUE/x2vPCwEDXmVYOEhZnm3ZpF6L3dg 7wlmNWIJMCyBbO9E0rFp2JfSZir3YD2aouOiX+JF83AIXdRJwoNei4sCAwEAAaAA MA0GCSqGSIb3DQEBCwUAA4IBAQBl2D2FccAcixqfbOTCeE01bh6jXInxWp3gX0tt LgrB9DuWxNp+KStg2rptM060cjJZxwZ5x6Fzqpt8/jCJkjSPLJSOKeXuPAxKGENO l+32xhc1tosW/pjD5wF0VllZ6h4C7gzwiHWvXSAHvOVEI/0flChzCc+Afm1CXUdf vAJakNPu5TgiR1VM6w1cQbzgMumZ0eFv2UOZDn51WJ3E0hGzBxQWrF2LUBuBMNLu bx8uVGudoiJiAZg8e7Wq+oe15vhsSKvAFMpbnleyjKUw8eZ7zU9OH6VtmTPsmrDP /YFWxeNbJeJTJ8BTuBwXIVZ2seYp5Bjz5+ZDC6K1uQsNdlvm -----END CERTIFICATE REQUEST----- |
| Re: How do you handle mass revocation requests? | jomo | 28/02/18 13:53 | > Basically, we're revoking 50k people without
> being able to explain why (well, other than the key was compromised). Unless I misunderstood, you originally said you received 23k compromised keys and are revoking those. > Currently, we are only revoking the certificates if we received the private > keys. There are additional certificates the reseller requested to have > revoked, but DigiCert has decided to disregard that request until we receive > proof of compromise or more information about the cause of this incident. Has DigiCert received proof of compromise of all 50k in the meantime? On 28.2.18 22:42, Jeremy Rowley via dev-security-policy wrote: > We don't have a process to prevent third parties from storing private keys. > I'm not sure how that would even work considering the approved third-party > use cases vs. non-approved use cases. In fact, I'd postulate there's > nothing wrong with Trustico holding the private keys if they were hosting > the site or providing CDN services for all of these sites. The issue is > Trustico alleged compromise of the certificates and sent us the private keys > believed compromised in support. There were a lot of them. > > This is a mass revocation without any explanation of what went wrong or why. > The reseller mentioned and proved compromise, but there's no way to see if > what happened, whether the issue was mitigated, or how it's going to be > prevented from happening again. Basically, we're revoking 50k people without > being able to explain why (well, other than the key was compromised). > > -----Original Message----- > From: dev-security-policy > <dev-security-policy-bounces+jeremy.rowley=digice...@lists.mozilla.org> > On Behalf Of urijah--- via dev-security-policy > Sent: Wednesday, February 28, 2018 2:24 PM > To: mozilla-dev-s...@lists.mozilla.org > Subject: Re: How do you handle mass revocation requests? > > Is Trustico's storage of private keys related to this security report from a > few months back (which did not appear to ever have been fully > investigated...)? > > https://clicktime.symantec.com/a/1/gUJtIwxyRazHoxd9FIcS40cx3EykeGjrwv4urOpHj > u0=?d=-1kl6pTbi8aU0HfRh9ZNP86TCSEsjm5TvumKZsLlfO6tgKV5N4_MitDAK64FGwutmiXu3D > eTelMEfv3ep7X8kj6GIwMY8tUVM_WKGkqW_uhm0fHWgY9ZwlUytCHVYb_1WWceLMVi9w9Ec7h3sH > G1wIJHWNk2L8RcVycGdrIGRFvV76QXqOck9szIT1MqOPD6gSaSZRnxp-ItDtBU5xSit2RwQjv5sB > Ln2qFtnD9T_wuM6-KAeTUDm1O51uIY3NpT-q5hyIKG5YG-Ds6a9ceM0JgMrLhM3evdd-BmO3e6eP > 1fXZXAQgJCZ_A7SnTbSagjzHzUVC65PN3AnrdJ6ANhZAwjjCdHdbjOeIqwInT-oFJ8lufSzaBeui > OECQuf-IqhCnfsam6KnTGT5k-aQlG_BjOW8fny0TS21Upa4k4aX2Xm7es0oRfOUS-OxAQLVKCGg- > cER_Pj3TmyKGPFFw_SzUky&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsg%2Fmozilla > .dev.security.policy%2FCEww8w9q2zE%2FF_bzX1guCQAJ > > Does Digicert have (or will it have) some sort of process in place to > prevent resellers from storing private keys so casually? > > On Wednesday, February 28, 2018 at 12:38:16 PM UTC-5, Jeremy Rowley wrote: >> Hi everyone, >> >> >> >> I wanted to share an incident report regarding the revocation of >> certain certificates ordered through a reseller. >> >> >> >> On February 2nd, 2018, we received a request from Trustico to mass >> revoke all certificates that had been ordered by end users through > Trustico. >> Unfortunately, the email was not sent to the appropriate certificate >> problem reporting channels and did not surface immediately so we're >> delayed in sharing the concerns and information. Once we were alerted, >> the team kicked off a debate that I wanted to bring to the CAB Forum. >> Basically, our position is that resellers do not constitute >> subscribers under the Baseline Requirement's definitions (Section >> 1.6.1). As such, we needed to confirm that either the key was >> compromised or that they revocation was authorized by the domain >> holder (the subscriber) prior to revoking the certificate. The > certificates were not alleged as compromised at that time. >> >> >> Later, the company shared with us that they held the private keys and >> the certificates were compromised, trying to trigger the BR's 24-hour >> revocation requirement. However, we insisted that the subscriber must >> confirm the revocation request or there must be evidence of the private > key compromise. >> >> >> Normally, we permit partners to revoke and manage the certificates >> freely on behalf of their customer, with DigiCert providing all >> validation and issuance services. However, the sheer number of >> revocation requests (50k) and allegation of compromise triggered >> concerns around the impact to the web and browser users. Therefore, >> this was categorized as high risk, especially considering the relationship > between Trustico and DigiCert is terminating. >> >> >> On 2/27/2018, at my request for proof of compromise, we received a >> file with 23k private keys matched to specific Trustico customers. >> This definitely triggered our 24-hour revocation processing requirement > under 4.9.1.1.3. >> Once we received the keys, we confirmed that these were indeed the >> matching private keys for the reported certificates. We will be >> revoking these certificates today (February 28th, 2018). >> >> >> >> At this time, Trustico has not provided any information about how >> these certificates were compromised or how they acquired the private >> keys. As is standard practice for a Certificate Authority, DigiCert >> never had possession of these private keys. >> >> >> >> Currently, we are only revoking the certificates if we received the >> private keys. There are additional certificates the reseller requested >> to have revoked, but DigiCert has decided to disregard that request >> until we receive proof of compromise or more information about the cause > of this incident. >> >> >> DigiCert sent out emails to the affected customers in order to notify >> them that their certificate(s) are being revoked. This revocation only >> affects those customers and there is no additional exposure that we >> are aware of at this time, nor any reason to believe there is. >> >> >> >> This raises a question about the MDSP policy and CAB Forum >> requirements. Who is the subscriber in the reseller relation? We >> believe this to be the key holder. However, the language is unclear. I >> think we followed the letter and spirit of the BRs here, but I'd like >> feedback, perhaps leading to a ballot that clarifies the subscriber in a > reseller relationship. >> >> >> This also brings up a ballot about the level of due diligence required >> for cert revocation. We generally believe that the private key or >> demonstration of domain control is sufficient to request revocation. >> Others are at the CAs discretion. Should we clarify what the due >> diligence looks like? Are there other things we should have done or been > doing? >> >> >> What kind of transparency would the Mozilla community like around this >> issue? There aren't many more facts than I shared above, but there is >> a lot of speculation. Let me know what I can share to help alleviate >> confusion and answer questions. >> >> >> >> Jeremy > _______________________________________________ > dev-security-policy mailing list > dev-secur...@lists.mozilla.org > https://clicktime.symantec.com/a/1/HzhUTcKY59cs-68Bx4Zcxt9mxYoEtI12E4LdF4LDI > mk=?d=-1kl6pTbi8aU0HfRh9ZNP86TCSEsjm5TvumKZsLlfO6tgKV5N4_MitDAK64FGwutmiXu3D > eTelMEfv3ep7X8kj6GIwMY8tUVM_WKGkqW_uhm0fHWgY9ZwlUytCHVYb_1WWceLMVi9w9Ec7h3sH > G1wIJHWNk2L8RcVycGdrIGRFvV76QXqOck9szIT1MqOPD6gSaSZRnxp-ItDtBU5xSit2RwQjv5sB > Ln2qFtnD9T_wuM6-KAeTUDm1O51uIY3NpT-q5hyIKG5YG-Ds6a9ceM0JgMrLhM3evdd-BmO3e6eP > 1fXZXAQgJCZ_A7SnTbSagjzHzUVC65PN3AnrdJ6ANhZAwjjCdHdbjOeIqwInT-oFJ8lufSzaBeui > OECQuf-IqhCnfsam6KnTGT5k-aQlG_BjOW8fny0TS21Upa4k4aX2Xm7es0oRfOUS-OxAQLVKCGg- > cER_Pj3TmyKGPFFw_SzUky&u=https%3A%2F%2Flists.mozilla.org%2Flistinfo%2Fdev-se > curity-policy > > _______________________________________________ > dev-security-policy mailing list > dev-secur...@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy |
| Re: How do you handle mass revocation requests? | Ryan Duff | 28/02/18 13:55 | >From what I've read, it appears the situation here is that Trustico wanted to revoke all their customer certs from Digicert so they could do a mass migration to another CA (which is not a proper reason to revoke). When asked for proof by Digicert that the certificates were compromised and needed to be revoked, Trustico sent Digicert 23,000(!) private keys that *they had stored* due to the fact that they were generated by their web-based system in order to effectively *make them* compromised.
Am I missing anything? |
| Re: How do you handle mass revocation requests? | Jeremy Rowley | 28/02/18 13:58 | No. Just the 23k. Part of the reason I posted to the Mozilla list are open questions about whether Trusticos request is sufficient to trigger the br requirements. My gut is no, and sounds like the browsers agree. We’ll only revoke the remaining 27k if we receive evidence of compromise
<dev-security-policy-bounces+jeremy.rowley=digice...@lists.mozilla.org><mailto:dev-security-policy-bounces+jeremy.rowley=digice...@lists.mozilla.org> To: mozilla-dev-s...@lists.mozilla.org<mailto:mozilla-dev-security-policy@lists.mozilla.org>cER_Pj3TmyKGPFFw_SzUky&u=https%3A%2F%2Fgroups.google.com<http://2Fgroups.google.com>%2Fd%2Fmsg%2Fmozilla dev-secur...@lists.mozilla.org<mailto:dev-security-policy@lists.mozilla.org> https://clicktime.symantec.com/a/1/HzhUTcKY59cs-68Bx4Zcxt9mxYoEtI12E4LdF4LDIcER_Pj3TmyKGPFFw_SzUky&u=https%3A%2F%2Flists.mozilla.org<http://2Flists.mozilla.org>%2Flistinfo%2Fdev-se curity-policydev-secur...@lists.mozilla.org<mailto:dev-security-policy@lists.mozilla.org> https://lists.mozilla.org/listinfo/dev-security-policy<https://clicktime.symantec.com/a/1/3wjI0lE22JgYGnrIgtAQv7FW6pnTRdrR8fiCDSlaje0=?d=qnG23jeRzsakHnmv0m3tUVVNhTHCDp_XhG50BufJUY1ASHMt_9WbK21Ax7NgWSAA2xIM-RvMri-USsX9RhFJjmsPle-xObmX_pXxmdHIaaoZKpCslmIbvdctjhlZvrQalkvVEfm1zX0iu1zYXyDIibSV07BzfyniUgevaRZAqRNjw1Jx8MmcLciydYLHSPiCNOMb1MBcDYKtNOFp5SUKVh3TAAGLgtMS3cF3U45pLXWRezAm2ZE8YI3wxz9CImhm2Hx7tyCZ0IxQ0WaYMm1DHzaHoC_BuLeBMZfYs4ANAavnNt3j55dnS1EKIAfSMY__JjDbvyzGhCQ6IuH69wElLd1adi_CwkGlPMI8C7rWq7xl6e_4i9FU0b5QJmHdqHuR9XLIkW-TRfI%3D&u=https%3A%2F%2Flists.mozilla.org%2Flistinfo%2Fdev-security-policy> |
| Re: [cabfpub] How do you handle mass revocation requests? | Ryan Sleevi | 28/02/18 14:07 | On Wed, Feb 28, 2018 at 4:50 PM, Jeremy Rowley via dev-security-policy <> bl...@cabforum.org>> Not the most efficient way, perhaps, but it works. For example, the last CSR here corresponds to the SHA256(SPKI) = bd753fb218dc69c8a9cafa99b4a99bfe15d0706d2800830058c8e890526e2c65, which is https://crt.sh/?id=29463720 , which presently is unrevoked :) To verify a CSR: openssl req -in "${csr}" -noout -verify (you get "Verify OK") To output the Subject (to demonstrate DigiCert is in control) openssl req -in "${csr}" -noout -subject (you get "subject=/C=US/ST=Utah/L=Newbury/O=Jeremy/OU=Proof of key compromise/CN=www.example.com") To export the SPKI hash: openssl req -in "${csr}" -noout -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -hex (you get "bd753fb218dc69c8a9cafa99b4a99bfe15d0706d2800830058c8e890526e2c65", which you can search for with https://crt.sh/?spkisha256= - e.g. https://crt.sh/?spkisha256=bd753fb218dc69c8a9cafa99b4a99bfe15d0706d2800830058c8e890526e2c65 ) |
| Re: How do you handle mass revocation requests? | Matthew Hardeman | 28/02/18 14:23 | On Wednesday, February 28, 2018 at 3:55:37 PM UTC-6, Ryan Duff wrote:That's kind of what I was thinking happened also. Though a couple points to correct: The original issuing CA hierarchy is a Symantec trust path. This suggests that what they really wanted to occur was to trigger a 24 hour reissue of all of these certificates under a DigiCert trusted path -- since presumably any issuance at this point would fall under a DigiCert path. Thus, within 24 hours, getting new certificates for all their customers under the new trust path. I'm going to guess someone at Trustico was getting annoyed at support calls regarding the migration and somehow assumed there'd be no consequences for pushing the issue by way of getting all those certificates revoked on "security" grounds. As grounds for this belief, I submit the strangely worded statement of Mr. Rowley at the start of the thread "Later, the company shared with us that they held the private keys and the certificates were compromised, trying to trigger the BR's 24-hour revocationThat language seems to imply that there's a sense that the security / web PKI integrity aspect is less the matter at stake and more that the keys were located and sent over to create an impossible to ignore security issue forcing the 24 hour window. My guess is that the person at Trustico wanted immediate reissuance of all of the Symantec certificates under the DigiCert trust paths and assumed: 1. That revoking the certs for security reasons would result in ASAP reissue (probably true in one-offs). 2. That the reissuance would happen in the DigiCert trust path (almost certainly true). 3. That they'd have a spike of support issues related to the reissuances, but that Trustico would have more control over the period over which they had to help customers migrate certificates and then the "bleeding" would stop. |
| Re: How do you handle mass revocation requests? | Ryan Sleevi | 28/02/18 14:29 | Note: The evidence on this thread does not support those assumptions. Among
other things, Trustico no longer is a reseller of Symantec (now DigiCert) certificates, and has transitioned new issuance to Comodo. This was already covered in https://groups.google.com/d/msg/mozilla.dev.security.policy/wxX4Yv0E3Mk/jx6r9jlPAwAJ |
| RE: How do you handle mass revocation requests? | Jeremy Rowley | 28/02/18 14:44 | 1) Not all of the certificates being revoked use the Symantec hierarchy.
There are some certs that use the DigiCert replacement hierarchy. Not many though. 2) Sorry my wording was strange. It almost always is. What I meant, is Trustico specifically asked for the certs to be revoked within 24 hours as required by the BRs. They said in the email they were triggering the 24 hour requirement. 3) I really feel like a third point would be important, but I can't think of one.
On Behalf Of Matthew Hardeman via dev-security-policy _______________________________________________https://clicktime.symantec.com/a/1/1SicdYM5NOSV8S29ZlemtDgUftWWe-mEnCeFfDCfA lw=?d=paMnJZlER7IHb626K-31V2u-Kf4DBs2hoO7Bro78ZuywH3EZ9N1dve7JXTCFPZFHyQrvw7 cDEqjPm1CCO0iQqqCbe-J7q2_uVOXTQzSslJpmeaUe9RmpgB81xaobKJOyb_YpAY8IOdkE832w7N tu7J94BmAMUFyIN-LINYJhcdKrmRggiP3dwENTjoH8GFBgeJgbAAJ7AXEY8EsaOLt-dqVymUiml9 GCqK8Pz2bZYq21i2TjrlH5i5ctTnHGcaJLrBukwohzJ1XA0B0Ma6sfh6NdDO9yi3psZCXUIDcZyp 5FtLBoYtTo6DUQVC8VhfOJY6KGhSpursgUnQdy6yQRtirjLO9kmIOKJSuJvlbaKD3gYKkxIRzMLA 6jPbVCxU77Z6q6OAnzLtAIRiCe7-MVzty2jFdcG8R9uiVI7Vf241-3ANcBFbr288JrDaZYHwhYJu rWo3wUC3HNJXFV_nf74MJE&u=https%3A%2F%2Flists.mozilla.org%2Flistinfo%2Fdev-se curity-policy |
| Re: How do you handle mass revocation requests? | Matthew Hardeman | 28/02/18 15:24 | On Wednesday, February 28, 2018 at 4:44:50 PM UTC-6, Jeremy Rowley wrote:So, I suspect what I and others in the community, like Mr. Duff, are wondering is: Why did Trustico go to such lengths to forcibly cause revocation of all of these certificates? Because their initial request was for the revocation to occur and Digicert's initial position was that they are not the subscriber and so didn't have to revoke at their request they finally had someone put together the keys and re-request with the formal 24 hour deadline. That sounds like they went to significant effort to ensure the revocation would happen. If the keys were truly leaked or otherwise compromised, it makes sense. If they were not and Trustico just pulled them out to trigger the 24 hour revocation window, why did they do that? It seems like the only people they're (directly) hurting are their own customers. That's quite irrational behavior and generally businesses do not so overtly act against their own interest. That sort of thing will scare a customer away from a reseller in a hurry. So the question on my mind is, why would Trustico do that to their own customers? It can't just be about migration to a new CA, right? If so, wouldn't the certificates just be left valid and new certificates be issued by the new CA? Forcing the 24 hour revocation window just creates a support nightmare for Trustico and Trustico's customers with no apparent upside, unless there was a real compromise. Trustico's customers would seem to be entitled to know whether Trustico actually suffered a breach which compromises those customers. |
| Re: How do you handle mass revocation requests? | Eric Mill | 28/02/18 16:04 | Trustico doesn't seem to provide any hosting or CDN services that would
make use of the private key, nor do they appear to explicitly inform users about the storage of this private key. In their statement, they say they keep the private keys explicitly to perform revocation as necessary: https://www.trustico.com/news/2018/symantec-revocation/certificate-replacement.php (archived: https://archive.is/0AnyR ) > These Private Keys are stored in cold storage, for the purpose of revocation. Their CSR/key generation form is here: https://www.trustico.com/ssltools/create/csr-pem/create-a-new-csr-instantly.php (archived: https://archive.is/hJV42 ) The storage of private keys appears to be done without the user's knowledge or consent. And of course, only the keys they create through their form are stored, so it is clearly not a necessary business function for most of their certificate business. Finally -- the CSR/key generation form page incorporates JavaScript from at least 5 or 6 different companies (including ad servers), which would allow any of those third parties (intentionally or through compromise of their own) to capture generated keys. This is a reckless amount of exposure, to the point that even if the keys were generated completely inside the browser and never exposed to the server (which does not appear to be the case), I would consider them compromised at the time of generation. Given everything that's known, then regardless of who emailed whose customers when and why, I think it's clear that Trustico compromised those keys at _least_ at the time they were stored, if not at the time of generation, and has been routinely compromising customer keys for years. Emailing them to DigiCert only widened their exposure to more unauthorized parties. And given that there's no evidence that Trustico has acknowledged this fact, or indicated any intent to change their business practices, then I believe it's appropriate for all CAs to immediately suspend or terminate their relationship with Trustico -- as any CA who continued doing business with Trustico would now be knowingly allowing Trustico to compromise the keys of the certificates issued under their hierarchy. -- Eric On Wed, Feb 28, 2018 at 3:24 PM, Ryan Hurst via dev-security-policy < dev-secur...@lists.mozilla.org> wrote: > On Wednesday, February 28, 2018 at 11:56:04 AM UTC-8, Ryan Sleevi wrote: > > Assuming Trustico sent the keys to DigiCert, it definitely sounds like > even > > if Trustico was authorized to hold the keys (which is a troubling > argument, > > given all things), they themselves compromised the keys of their > customers, > > and revocation is both correct and necessary. That is, whether or not > > Trustico believed they were compromised before, they compromised their > > customers keys by sending them, and it's both correct and accurate to > > notify the Subscribers that their keys have been compromised by their > > Reseller. > > That seems to be the case to me as well. > > It also seems that this situation should result in the UAs and/or CABFORUM > re0visit section 6.1.2 (https://github.com/cabforum/ > documents/blob/master/docs/BR.md) in the BRs. > > Specifically, this section states: > > ``` > Parties other than the Subscriber SHALL NOT archive the Subscriber Private > Key without authorization by the Subscriber. > > If the CA or any of its designated RAs generated the Private Key on behalf > of the Subscriber, then the CA SHALL encrypt the Private Key for transport > to the Subscriber. > ``` > > In this case, TrustIco is not the subscriber, and there is no indication > in their terms and conditions (https://www.trustico.com/ > terms/terms-and-conditions.php) that they are authorized to archive the > private key. Yet clearly if they were able to provide 20k+ private keys to > DigiCert they are archiving them. This text seems to cover this case > clearly but as worded I do not see how audits would catch this behavior. I > think it may make sense for the CAs to be responsible for demonstrating how > they and other non-subscribers in the lifecycle flow handle this case. > > Additionally, it seems if the private keys were provided to DigiCert in a > way they were verifiable by them they may have been stored in a > non-encrypted fashion, at a minimum they were likley not generated and > protected on an HSM. The BRs should probably be revised to specify some > minimum level of security to be provided in these cases of for these cases > to be simply disallowed altogether. > > Finally, the associated text speaks to RAs but not to the non-subscriber > (reseller) case, this gap should be addressed minimally. > _______________________________________________> https://lists.mozilla.org/listinfo/dev-security-policy > -- konklone.com | @konklone <https://twitter.com/konklone> |
| Re: How do you handle mass revocation requests? | Ben Laurie | 01/03/18 02:48 | On 28 February 2018 at 19:40, Jeremy Rowley via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote: > The end user agreed to the subscriber agreement, not Trustico. Our > analysis follows what Peter B. posted – the subscriber is the “natural > person or Legal Entity to whom a Certificate is issued and who is legally > bound by a Subscriber Agreement or Terms of Use"—which in this case was > Trustico’s customers. In addition, we felt that given (1) the number of > certificates Trustico was asking us to mass-revoke and (2) the lack of any > substantiation of why Trustico thought the certs were “compromised,” we > needed more information before revoking. At the minimum, it warranted > alerting the contact for each certificate that revocation was imminent. > > > > I also agree that there’s no problem with the way or that the keys were > provided to DigiCert for cert revocation. Agree with who? Both asking for the keys and providing them seems weird to me. The more secure thing to do would be to ask for proof of possession of the keys, e.g. by signing a random string with them... > I certainly don’t want to discourage revocation of compromised certs! My > main question is how do you handle these things? The process at scale > should not be different if a reseller requests revocation compared to any > other security researcher. The question is how you handle the this volume > of revocations when its happen? I’ve never received a revocation request of > this magnitude before so I’m seeking the wisdom of the community in what we > do. > > > > I’m happy to share any of the details I can. > > > > Jeremy > > > > > > From: Ryan Sleevi <ry...@sleevi.com> > Sent: Wednesday, February 28, 2018 11:58 AM > To: Jeremy Rowley <jeremy...@digicert.com> > Cc: mozilla-dev-s...@lists.mozilla.org > Subject: Re: How do you handle mass revocation requests?> On Wed, Feb 28, 2018 at 12:37 PM, Jeremy Rowley via dev-security-policy < > dev-secur...@lists.mozilla.org <mailto:dev-secur...@ > lists.mozilla.org> > wrote:> I think there's a little nuance to this in the general case (e.g. consider > "Resllers" who are also the hosting provider, and thus constitute the > Applicant/Subscriber even when they're not the domain holder, but > authorized by them), but based on this specific case, I think this sounds > like a correct determination. > > > > I think the biggest question is who agreed to the terms - Trustico or the > end-user - and that mostly impacts the rest of the decision here. Further, > did Trustico warrant on behalf of the user that the user agreed to the > terms (in which case, I would think it's a bit of a copout, and it's really > Trustico agreeing, thus the Subscriber), or did DigiCert have direct > communication with the user on the basis of Trustico's introduction (in > which case, yeah, the Subscriber is the user) > > > > Anyway, just highlighting it here as perhaps not a uniform consensus, but > perhaps not as material as what follows. >> I think all of this sounds reasonable, no different than a security > researcher (or random member of the public) who were to claim compromise > with no evidence of that. >> For the same reason that "Jane Random User" should not be able to cause > revocation merely by assertion, I think that sounds reasonable. >> I think the question here is less about who is the Applicant, but who is > the Applicant Representative. If the Reseller is signing/agreeing the > request on behalf of the applicant, or the Subscriber Agreement, they're > the Applicant Representative and ostensibly should be taken as authorized. > > > > I think the gap here is we have this notion of Applicant/Applicant > Representative prior to the issuance - in which some 3P may agree to a > Subscriber Agreement (or warrant agreement), yet claim the Subscriber is > somehow a different entity or that Representative is no longer bound in > scope. That seems pretty troubling - both in terms of how the Applicant > Representative is verified as authorized (which seems fundamentally what a > Reseller who agrees to a ToS is) - and as to how revocation works. >> I think Wayne's still looking at the revocation space (and I'm much > belated in my offering feedback), but I think one of the gaps is one we've > seen come up with Google domains and Let's Encrypt. And while I share these > stories, to be clear, I don't think LE has done anything wrong in issuance > or handling of it - it just is a good demonstration of the nuance that any > such clarification should consider. > > > > Consider https://crt.sh/?id=245397170 , which was a google.tg < > http://google.tg> certificate obtained from LE following an apparent > Registry compromise. Prior to the compromise, Google operated google.tg < > http://google.tg> , during the compromise, an unknown third-party did, > and following the compromise, Google again operated/controlled google.tg < > http://google.tg> . > > > > When it came to requesting revocation, however, this highlighted a > challenge. Let's Encrypt has several defined methods for validation - the > HTTP-01 method, TLS-SNI-01 (at the time), and DNS-01. Google services - > particularly those hosting/serving google.tg <http://google.tg> - do not > support any of those methods (by somewhat intentional design). Google also > did not control the private key - naturally. > > > > Based on the evidence Google provided - and, importantly, through > consultation with 3P - LE was able to determine the .tg compromise and > revoke the certificate. > > > > A mandate of 'due diligence' that suggested demonstration of private key, > or of using LE's preferred methods, would have prevented that revocation. A > model that offers CA significant flexibility and discretion does have the > downside of the CA choosing to *refuse* to revoke, and them claim that they > were operating within the bounds of the Baseline Requirements. > > > > Thus, any form of requiring due diligence has to consider an adversarial > model of CAs (sorry Jeremy!), in which the CA may have incentives, whether > real or imagined, not to revoke and/or to impose hardship on the domain > operator. The current structure favors the 'victims' in the example I just > gave, but I think also presents risks - as you raise - of 3P requesting > revocation without authorization. I don't know if or how we'll be able to > square those somewhat competing interests, but I think it's worth keeping > in the consideration. > > > > If we stretch the idea out further, one could imagine that CAs must > support 'all' validation methods to authenticate a revocation request. But > now that shifts the burden from the victim (in having to prove control) > onto the CA - which is its own set of tradeoffs. It also introduces new > risk - in which adversaries may use a weaker method of authentication > maliciously (for example, consider 3.2.2.4.8, in which anyone sharing an IP > could cause the revocation of a cert of anyone else sharing that IP) > > > > In any event, I'm hugely appreciative of the details you've taken to share > and be transparent with the community regarding this. This is an incredibly > valuable discussion to have, and more importantly, and valuable level of > transparency regarding the incident, given the amount of apparent > misinformation and confusion regarding this flying about. |
| Re: How do you handle mass revocation requests? | Ben Laurie | 01/03/18 02:51 | On 28 February 2018 at 21:37, Nick Lamb via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote: > On Wed, 28 Feb 2018 20:03:51 +0000 > Jeremy Rowley via dev-security-policy> > The keys were emailed to me. I'm trying to get a project together > > where we self-sign a cert with each of the keys and publish them. > > That way there's evidence to the community of the compromise without > > simply listing 23k private keys. Someone on Reddit suggested that, > > which I really appreciated. > > That's probably me (tialaramex). > > Anyway, if it is me you're referring to, I suggested using the private > keys to issue a bogus CSR. CSRs are signed, proving that whoever made > them had the corresponding private key but they avoid the confusion > that comes from DigiCert (or its employees) issuing bogus certs. > Everybody reading m.d.s.policy can still see that a self-signed cert is > harmless and not an attack, but it may be harder to explain in a > soundbite. Maybe more technically able contributors disagree ? > Seems to me that signing something that has nothing to do with certs is a safer option - e.g. sign random string+Subject DN. |
| Re: How do you handle mass revocation requests? | Rob Stradling | 01/03/18 04:20 | And also throw in some transparency...
https://mailarchive.ietf.org/arch/msg/trans/WLFmIyaH4BJo77ZJDinKJcylOcg -- Rob Stradling Senior Research & Development Scientist Email: Rob.Stradling@ComodoCA.com Bradford, UK Office: +441274730505 ComodoCA.com |
| RE: How do you handle mass revocation requests? | Michel Gre | 01/03/18 05:51 | > I'd postulate there'sI manage one of the affected domains. I can tell that in no way does Trustico hosts the site, nor provide us any CDN service. We just purchased them a certificate 4 years ago and renewed it for 3 years in april 2015. Since we are usually quite busy we simply used their form to generate the key, the CSR, and get the certificate... So, Trustico should be actually Dontrustico. The worst is that the CEO himself publicly said (here!) that they HELD OUR PRIVATE KEYS!!! Come on. M. Zane Lucas, your staff sent me (after I asked them from an explanation regarding the Digicert's first email) a coupon for a "Trustico(r) Single Site" certificate, would you expect me to trust it after what YOU disclosed here? Looks like you just cut the branch your company was sitting on. |
| Re: How do you handle mass revocation requests? | nic....@gmail.com | 01/03/18 07:08 | I agree with Eric, I would call storing the customers private keys (without their knowledge!!) as an immediate compromise and a clear breach of trust.
|
| Re: How do you handle mass revocation requests? | Nick Lamb | 01/03/18 07:32 | On Thu, 1 Mar 2018 10:51:04 +0000
Ben Laurie via dev-security-policy <dev-secur...@lists.mozilla.org> wrote:That does sounds sane, I confess I have not spent much time playing with easily available tools to check what is or is not easily possible on each platform in terms of producing and checking such proofs. I knew that you can make a CSR on popular platforms, and I knew how to check a CSR is valid and a bogus CSR seemed obviously harmless to me. I feel sure I saw someone's carefully thought through procedure for proving control over a private key written up properly for close to this sort of situation but I have tried and failed to find it again since the incident was first reported, and apparently Jeremy didn't know it either. |
| Re: How do you handle mass revocation requests? | RS Tyler Schroder | 01/03/18 08:08 | In relevant news, Trustico's site is down due to an apparent flaw, apparently allowing users to run commands as root on their production webserver.
My question is, assuming this was discovered previously by an attacker, is there possibility of exploiting that to fetch these cold-storage keys? https://twitter.com/Manawyrm/status/969230542578348033 in reply to https://twitter.com/svblxyz/status/969220402768736258 |
| Re: How do you handle mass revocation requests? | Ryan Duff | 01/03/18 08:33 | Given that they were able to readily produce all of these keys, I would suspect they were never really in cold storage. At least not exclusively.
|
| Re: How do you handle mass revocation requests? | Ryan Sleevi | 01/03/18 09:52 | Perhaps you were thinking about the ROBOT attack, which covered
https://robotattack.org/ You can produce such messages, if you have the key, through something like (note: haven't tested, but you get the idea) associated_spki_hash=`openssl pkey -inform PEM -in foo.key -pubout | openssl dgst -sha256 -binary | openssl enc -base64` associated_crt_sh_url="http://crt.sh/?spkisha256=${associated_spki_hash}" echo "${associated_crt_sh_url} is compromised - 2018-03-01" > message.txt openssl dgst -sha256 -sign foo.key -out foo.key -out ${associated_spki_hash}.signature message.txt to verify openssl dgst -sha256 -verify <(openssl x509 -in "cert file goes here" -pubkey -noout) -signature ${associated_spki_hash}.signature message.txt |