Gen Kanai wrote: > Are we going to receive information from Comodo regarding how many other > Comodo resellers may be in a similar position to Certstar?
> Are we going to receive information from Certstar as to how many other > certs may have been issued in error?
I've asked Robin Alden of Comodo to make an accounting regarding these two issues. I don't expect to see that immediately (i.e., in the next day or two), though I also don't expect to wait a month for it (as Kyle is concerned about).
> How do we verify the claims from Comodo or Certstar?
Some things by their nature we can directly verify, and some we can't. For example, we can certainly verify how Certstar is operating currently (as Eddy and I did), and if certs get revoked we can verify that by downloading the CRLs that get issued.
Frank Hecker <hec...@mozillafoundation.org> writes: > My intent is to balance the disruption that would be caused by pulling > a root vs. the actual security threat to users. Right now we have no > real idea as to the extent of the problem (e.g., how many certs might > have been issued without proper validation, how many of those were > issued to malicious actors, etc.).
Isn't that, by itself, a very good reason to take immediate action? Security should be default-fail rather than default-pass.
<hec...@mozillafoundation.org> wrote: > I've asked Robin Alden of Comodo to make an accounting regarding these two > issues. I don't expect to see that immediately (i.e., in the next day or > two), though I also don't expect to wait a month for it (as Kyle is > concerned about).
Actually, I think it's very important that the accounting include this:
for each name (not just certificate, but name in subjectAlternativeNames) that has been certified, a connection to the TLS ports should be made, and the certificate presented by the site compared against the certificate that Comodo issued. This obviously won't be a complete verification, but it should give a start to see how widespread the problem is.
A script to do this could probably be written fairly easily, but depending on the number of certificates Comodo has issued that are currently valid (and I'd like to see some hard numbers on that, as well) it could take a while to run.
From the script, the numbers I'd like to see are: the number of unreachable/not-answering names/hosts, the number of matching certificates, and the number of mismatched certificates. From that output plus Comodo's records, I would also like to see how many resellers there are and how many of them have sold mismatched certificates.
I hate to say this, but this IS The Worst-Case Scenario. A CA has gone rogue and issued certificates that violate its standards, and the standards of the root programs that it's a part of -- it is true that Comodo didn't /intend/ to go rogue, but it has, and we can't afford to let it damage the greater PKI. Since every CA in the root store is treated the same, there is no differentiation between them -- and this means that Verisign and Comodo and Thawte and *every* CA share the same reputation. If one goes rogue, it's exactly the same as if all of them have gone rogue, in the eye of the end-user.
To put it another way, it's exactly the same problem as putting the public keys of webservers in DNSSEC. Since the end-system can't know if a response came from DNSSEC or just unauthenticated DNS, the end-system can't trust anything (including RFC2538-style CERT records) that comes from DNS.
THIS is why I want to see greater differentiation in the browser chrome between CAs, so that one bad apple doesn't spoil the whole root barrel. THIS is why the argument against changing the chrome (user convenience) fails.
I'd rather deal with disruption caused thereby (and, yes, the user complaints generated thereby -- at least then the end-user would KNOW that there's a problem that's being dealt with rather than having a FALSE SENSE OF SECURITY) than let those potential security breaches continue to wreak their quiet little havoc.
On Tue, Dec 23, 2008 at 11:15 AM, Hendrik Weimer <hend...@enyo.de> wrote: > Frank Hecker <hec...@mozillafoundation.org> writes:
>> My intent is to balance the disruption that would be caused by pulling >> a root vs. the actual security threat to users. Right now we have no >> real idea as to the extent of the problem (e.g., how many certs might >> have been issued without proper validation, how many of those were >> issued to malicious actors, etc.).
> Isn't that, by itself, a very good reason to take immediate action? > Security should be default-fail rather than default-pass.
> Frank Hecker<hec...@mozillafoundation.org> writes:
>> My intent is to balance the disruption that would be caused by pulling >> a root vs. the actual security threat to users. Right now we have no >> real idea as to the extent of the problem (e.g., how many certs might >> have been issued without proper validation, how many of those were >> issued to malicious actors, etc.).
> Isn't that, by itself, a very good reason to take immediate action? > Security should be default-fail rather than default-pass.
It should be. I realized that there are more points which will have to be addressed in due time at Mozilla, which however can wait for now.
Concerning the disruption, Comodo has many roots and the resetting of this specific root would affect low-assurance sites as far as I know. The higher validated sites would not be affected. Having said that, the roots of their low-assurance products have been previously of concern it must be looked at in the broader context of Comodo's operations. It's not surprising in itself.
> I'd rather deal with disruption caused thereby (and, yes, the user > complaints generated thereby -- at least then the end-user would KNOW > that there's a problem that's being dealt with rather than having a > FALSE SENSE OF SECURITY)
> On 12/23/08 11:27 AM, Kyle Hamilton wrote: >> I'd rather deal with disruption caused thereby (and, yes, the user >> complaints generated thereby -- at least then the end-user would KNOW >> that there's a problem that's being dealt with rather than having a >> FALSE SENSE OF SECURITY)
> Hmm, would they?
*sigh* I somehow managed to click "Send" mid-message.
Anyway, the rest of my incomplete thought:
...I think there's some risk that if a Firefox update suddenly breaks a large swath of legitimate SSL sites, that could end up training users to ignore the problem. I'm not even sure how you'd present this condition to Joe The User in a comprehensible way.
That said, the Comodo/Certstar is hugely sucky and I would hope there's something we can do about it that helps users.
Frank Hecker wrote: > Eddy Nigg wrote: >> Disabling the trust bits of "AddTrust External CA Root" could be a >> temporary measure to prevent damage to relying parties
> Also note that any "suspension" of a root would last at last 1-3 months, > since that the typical interval between security updates for Firefox and > other Mozilla-based products.
And we don't have a magic switch we can flip in the office. We'd have to make the change, test the change, make the builds, ship the builds, users would have to update (about a week from ship until most users have the update).
If the sole purpose of the update was to break lots of sites (from the user's POV) then some number of them disable updates, making them less secure in the future.
If Comodo is acting in good faith then anything they can do would be lightyears faster than a client update. If they're not fulfilling their responsibilities then a permanent removal would make sense, but given the time scales it's hard to see how a "temporary" month-or-so removal helps.
Maybe we need to build in something like a CRL that pings back to Mozilla that would let us revoke roots without having to ship a client update.
> Maybe we need to build in something like a CRL that pings back to > Mozilla that would let us revoke roots without having to ship a client > update.
Of course we (@ mozilla) also take our lessons from this event, I'm sure. Indeed it was previously suggested here and I think we'll have to look at such options in due time.
Anything other than taking down their site is insufficient (as an immediate action to prevent further damage), specially in light of apparently more flaws in their system. Not having done that as I requested already two days ago isn't exactly comforting. Doesn't show really cooperation and good faith here...
...they'd do that better now before any more damage happens. I've warned them and requested to shut them down. My warning was ignored, their bad...
> Frank Hecker wrote: > > Eddy Nigg wrote: > >> Disabling the trust bits of "AddTrust External CA Root" could be a > >> temporary measure to prevent damage to relying parties
> > Also note that any "suspension" of a root would last at last 1-3 months, > > since that the typical interval between security updates for Firefox and > > other Mozilla-based products.
> And we don't have a magic switch we can flip in the office. We'd have to > make the change, test the change, make the builds, ship the builds, > users would have to update (about a week from ship until most users have > the update).
> If the sole purpose of the update was to break lots of sites (from the > user's POV) then some number of them disable updates, making them less > secure in the future.
> If Comodo is acting in good faith then anything they can do would be > lightyears faster than a client update. If they're not fulfilling their > responsibilities then a permanent removal would make sense, but given > the time scales it's hard to see how a "temporary" month-or-so removal > helps.
> Maybe we need to build in something like a CRL that pings back to > Mozilla that would let us revoke roots without having to ship a client > update.
I, for example, have a ssl cert from comodo reseller, and they DO have made all the validation steps.
My site, a legitimate one, would be in trouble with this. Are you all sure that it is a good measure to just knock off the root cert or security bit?
> On 23 dez, 18:23, Daniel Veditz <dved...@mozilla.com> wrote:
> > Frank Hecker wrote: > > > Eddy Nigg wrote: > > >> Disabling the trust bits of "AddTrust External CA Root" could be a > > >> temporary measure to prevent damage to relying parties
> > > Also note that any "suspension" of a root would last at last 1-3 months, > > > since that the typical interval between security updates for Firefox and > > > other Mozilla-based products.
> > And we don't have a magic switch we can flip in the office. We'd have to > > make the change, test the change, make the builds, ship the builds, > > users would have to update (about a week from ship until most users have > > the update).
> > If the sole purpose of the update was to break lots of sites (from the > > user's POV) then some number of them disable updates, making them less > > secure in the future.
> > If Comodo is acting in good faith then anything they can do would be > > lightyears faster than a client update. If they're not fulfilling their > > responsibilities then a permanent removal would make sense, but given > > the time scales it's hard to see how a "temporary" month-or-so removal > > helps.
> > Maybe we need to build in something like a CRL that pings back to > > Mozilla that would let us revoke roots without having to ship a client > > update.
> I, for example, have a ssl cert from comodo reseller, and they DO have > made all the validation steps.
> My site, a legitimate one, would be in trouble with this. Are you all > sure that it is a good measure to just knock off the root cert or > security bit?
> please, think twice
I puzzled by the security approach on this issue. There was a severe security problem, and at least two, maybe many certs are compromised, who knows by this point. Do i still have trust in the Certs of Comodo or their signing resellers? No, until the whole issue is cleared up completely. Now it's the users duty to check every Cert on the issuer to evaluate the trustlevel, while his browser states, everything is fine? This is not about punishing Comodo, their resellers or their customers, this is about trusting my browser software.
> On Tue, Dec 23, 2008 at 10:43 AM, Frank Hecker > <hec...@mozillafoundation.org> wrote: >> I've asked Robin Alden of Comodo to make an accounting regarding these two >> issues. I don't expect to see that immediately (i.e., in the next day or >> two), though I also don't expect to wait a month for it (as Kyle is >> concerned about).
> Actually, I think it's very important that the accounting include this:
> for each name (not just certificate, but name in > subjectAlternativeNames) that has been certified, a connection to the > TLS ports should be made, and the certificate presented by the site > compared against the certificate that Comodo issued. This obviously > won't be a complete verification, but it should give a start to see > how widespread the problem is.
Um. Some questions... On what basis are we asking for this "accounting" ?
Is there a criteria that calls for public breach accounting? Perhaps Mozilla needs to add it? Most of the criteria call for revocation when a false cert has been issued, and that seems to have been met right now.
Also, some of the process might instead rest with Comodo's external auditor or its internal audit process or its internal security process. Perhaps in the next audit, there will be an examination of the remedial steps taken, to see if they conform to the security manual, etc.
There are some slippery slope aspects here. If the framework for such "accounting" is outside Mozilla's area, then care might be needed. There could be confusion in role and liabilities.
It's nice if Mozilla were minded to do all the audit and security activities for all, but I'm not sure Frank has time ;) OTOH, if we feel the need to usurp the CA's processes and the conventional audit relationship with an "accounting" then perhaps we need to also simplify some other processes?
Another aspect here is that the goal of any security or governance process is not to eliminate or stop all breaches. The process is more about how you keep dangers to a minimum, deal with the breaches that do happen, and improve.
> I hate to say this, but this IS The Worst-Case Scenario.
Noooo.... With respect, I'd say worst case is that a root has been breached, and the Carders have got hold of it and are MITMing the major retail sites. Damages!
Earlier, Frank used the language of "clear and present danger."
* clear: we can measure the costs of it, and cost of defences. * present: it is happening today, provably. * danger: it can be shown capable of doing damage, at least in theory
Only the last one is shown. It is a danger in that anyone relying on a cert wrongly issued could be harmed. But it hasn't actually caused anyone damages, and nobody has shown it to exist.
This situation is about as harmful as the average exploit demo. In this particular case it was caught by an insider rather than by an outsider. However, beyond that, the situation is already sealed in that Comodo have already taken the urgent triage steps to make sure nobody else gets one.
> A CA has > gone rogue and issued certificates that violate its standards, and the > standards of the root programs that it's a part of -- it is true that > Comodo didn't /intend/ to go rogue, but it has, and we can't afford to > let it damage the greater PKI.
This is no different to when Verisign got schnikered into issuing a couple of Microsoft certs. No damages ever resulted. The system worked as it was expected to, it seems. Panic is not called for.
> Since every CA in the root store is > treated the same, there is no differentiation between them -- and this > means that Verisign and Comodo and Thawte and *every* CA share the > same reputation. If one goes rogue, it's exactly the same as if all > of them have gone rogue, in the eye of the end-user.
...
> THIS is why I want to see greater differentiation in the browser > chrome between CAs, so that one bad apple doesn't spoil the whole root > barrel. THIS is why the argument against changing the chrome (user > convenience) fails.
Oh, on that, yes, strongly agree. There's no trust without brand :)
> Earlier, Frank used the language of "clear and present danger."
> * clear: we can measure the costs of it, and cost of defences. > * present: it is happening today, provably. > * danger: it can be shown capable of doing damage, at least in theory
> Only the last one is shown. It is a danger in that anyone relying on a > cert wrongly issued could be harmed. But it hasn't actually caused > anyone damages, and nobody has shown it to exist.
The first and second are valid as well. We can measure the costs, can't we? The second item shows that it can be happening today, you don't have to prove it, because I did already. The fact that I disclosed publicly about it doesn't mean that there aren't scores of others out there. Considering the aggressive and illegal mailing campaign they operated, one can expect that there are many certificates issued in this way.
> This situation is about as harmful as the average exploit demo. In this > particular case it was caught by an insider rather than by an outsider. > However, beyond that, the situation is already sealed in that Comodo > have already taken the urgent triage steps to make sure nobody else gets > one.
Well, that's exactly the point! I haven't heard back from Robin anymore, their site is still online as well. Clear statements about how many certificates are affected (all of the ones issued through them) and revocations (or at least urgent re-validation within 48 hours and thereafter revocation) would be satisfactory maybe.
The handling of the situation is insufficient, response and measures are a joke, communication doesn't exist. Apparently this resellers business was too good for Comodo.
>I'd rather deal with disruption caused thereby (and, yes, the user >complaints generated thereby -- at least then the end-user would KNOW >that there's a problem that's being dealt with rather than having a >FALSE SENSE OF SECURITY) than let those potential security breaches >continue to wreak their quiet little havoc.
And I would not.
Just because a few people loudly proclaim their preferences on either side, it does not mean that their preferences should be acted on in a way that affects millions of Firefox users.
> Just because a few people loudly proclaim their preferences on either side, it does not mean that their preferences should be acted on in a way that affects millions of Firefox users.
It was Comodo that affected millions of Firefox users; it's up to Mozilla to protect those users by failing safe.
Presumably it was Comodo that underwent an audit to be added to Mozilla's roots, and Comodo should not be allowed to delegate trust to their resellers for domain validation. If, today, trust is delegated to their resellers, then we can't trust Comodo, period.
Although disruptive, their trust bits should be suspended. The explanation to users: "The CA purporting to provide assurance about the site you are trying to visit cannot be trusted. Please contact the site operator and advise them to find a trustworthy certification authority."
Yes, perception is that Mozilla releases code expressly to "break" access to legitimate sites, but this is because a trusted CA has gone rogue. Users can still jump through hoops to expressly include the site's certificate and keep going.
The trust model for browsers should be fail-safe, even if this inconveniences users. Better that than me and countless others inadvertently exposing my credentials to a site pretending to be my bank, investment house, government revenue agency, etc.
If Mozilla doesn't pull the trust bits, what's it's accountability for any breaches that occur due to keeping the bits? With assurance must come liability, whether from the certification authority, or those who are implicitly trusted with vetting them.
The only effective and appropriate response to a root that does not have sufficient internal controls to maintain its own security is to remove the trust in it. If you've purchased a certificate from them because it's trusted, and then they lose that trust, I would think that you should be clamoring for your money back and looking for an alternate certificate issuer rather than trying to maintain the problem.
On Tue, Dec 23, 2008 at 12:44 PM, <doug...@theros.info> wrote: > On 23 dez, 18:23, Daniel Veditz <dved...@mozilla.com> wrote: >> Frank Hecker wrote: >> > Eddy Nigg wrote: >> >> Disabling the trust bits of "AddTrust External CA Root" could be a >> >> temporary measure to prevent damage to relying parties
>> > Also note that any "suspension" of a root would last at last 1-3 months, >> > since that the typical interval between security updates for Firefox and >> > other Mozilla-based products.
>> And we don't have a magic switch we can flip in the office. We'd have to >> make the change, test the change, make the builds, ship the builds, >> users would have to update (about a week from ship until most users have >> the update).
>> If the sole purpose of the update was to break lots of sites (from the >> user's POV) then some number of them disable updates, making them less >> secure in the future.
>> If Comodo is acting in good faith then anything they can do would be >> lightyears faster than a client update. If they're not fulfilling their >> responsibilities then a permanent removal would make sense, but given >> the time scales it's hard to see how a "temporary" month-or-so removal >> helps.
>> Maybe we need to build in something like a CRL that pings back to >> Mozilla that would let us revoke roots without having to ship a client >> update.
> I, for example, have a ssl cert from comodo reseller, and they DO have > made all the validation steps.
> My site, a legitimate one, would be in trouble with this. Are you all > sure that it is a good measure to just knock off the root cert or > security bit?
Eddy Nigg wrote: > Concerning the disruption, Comodo has many roots and the resetting of > this specific root would affect low-assurance sites as far as I know.
I don't think that's necessarily true. I don't think it would affect EV sites (because of the way validation for those sites is special-cased), but it could affect non-EV sites under other Comodo brands (I mean, other than the PositiveSSL brand) and it could also affect other CAs whose CA certs are cross-signed by the UTN-UserFirst-Hardware root.
> Eddy Nigg wrote: >> Concerning the disruption, Comodo has many roots and the resetting of >> this specific root would affect low-assurance sites as far as I know.
> I don't think that's necessarily true. I don't think it would affect EV > sites (because of the way validation for those sites is special-cased), > but it could affect non-EV sites under other Comodo brands (I mean, > other than the PositiveSSL brand) and it could also affect other CAs > whose CA certs are cross-signed by the UTN-UserFirst-Hardware root.
How about ADDING the chained issuing CA certificate to NSS and mark it deliberately untrusted (no bits turned on)?
But our dilemma here shows clearly that we must have tools to act to such threats. It's probably one of the lessons to learn from this incident.
> Presumably it was Comodo that underwent an audit to be added to > Mozilla's roots, and Comodo should not be allowed to delegate trust to > their resellers for domain validation. If, today, trust is delegated > to their resellers, then we can't trust Comodo, period.
I would second that and in light of many other problematic practices which were discovered during their inclusion/update of EV, it's simply too much. More than 24 hours into this, I've come to the conclusion that this is a sever incident which requires action. If Robin can assure us of reasonable actions from their side (as suggested previously by me) it would serve all participants the best. Inaction and non-cooperation will leave Mozilla with not much choice I think. Ignorance by Mozilla itself will hunt it for a long time too. But it must happen now, either way!
(I don't think we have the time to discuss each and every aspect of RA and reseller responsibilities and what we deem as save, I'm certain we'll take this issue up (which apparently has about the same implications as intermediate externally operated CAs))
I believe that Startcom (and other certification authorites in Mozilla's root program) would likely have cause to bring an action for the tort of negligence against Mozilla. I feel that this is something that Mozilla should likely ask its general counsel very quickly.
0) Comodo is plainly found to have derelicted its duty to uphold the Mozilla CA agreement. As a result, damage to the trust in the PKI occurs, causing people to look outside of the PKI for solutions to the problems that encryption and authentication can solve.
1) As a result of this Startcom and other CAs in the PKI have a reduced market for their products. (damage)
2) Mozilla has the authority and ability to remove CAs from its trust list if and when they are found to be negligent in their duties. It, as an organization which attempts to vet the CAs that it includes in its PSM and which has policies which allow for it to take action should the policies be breached, has accepted the duty to remain diligent. (duty, and proximity of duty, since in order to get into the program in the first place all CAs in the PKI must accede to Mozilla's demands of trustworthiness.)
3) Mozilla fails to remain diligent, and fails to remove trust bits upon notice of dereliction. (causation, which leads to the trust in the PKI being eroded further.)
I am not a lawyer, this is not legal advice, etc. I'm trying to prevent Mozilla from having problems.
-Kyle H
On Tue, Dec 23, 2008 at 2:05 PM, Paul C. Bryan <em...@pbryan.net> wrote:
> Presumably it was Comodo that underwent an audit to be added to > Mozilla's roots, and Comodo should not be allowed to delegate trust to > their resellers for domain validation. If, today, trust is delegated > to their resellers, then we can't trust Comodo, period.
> Although disruptive, their trust bits should be suspended. The > explanation to users: "The CA purporting to provide assurance about > the site you are trying to visit cannot be trusted. Please contact the > site operator and advise them to find a trustworthy certification > authority."
> Yes, perception is that Mozilla releases code expressly to "break" > access to legitimate sites, but this is because a trusted CA has gone > rogue. Users can still jump through hoops to expressly include the > site's certificate and keep going.
> The trust model for browsers should be fail-safe, even if this > inconveniences users. Better that than me and countless others > inadvertently exposing my credentials to a site pretending to be my > bank, investment house, government revenue agency, etc.
> If Mozilla doesn't pull the trust bits, what's it's accountability for > any breaches that occur due to keeping the bits? With assurance must > come liability, whether from the certification authority, or those who > are implicitly trusted with vetting them. > _______________________________________________ > dev-tech-crypto mailing list > dev-tech-cry...@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-tech-crypto
On Dec 23, 5:51 pm, "Kyle Hamilton" <aerow...@gmail.com> wrote:
> I believe that Startcom (and other certification authorites in > Mozilla's root program) would likely have cause to bring an action for > the tort of negligence against Mozilla. I feel that this is something > that Mozilla should likely ask its general counsel very quickly.
Apart from the legal and procedural ramifications of all this, can you guys publish a manual procedure for those of us who want to remove Comodo's certificates from our browser installations? Assuming one is available?
> On Dec 23, 5:51 pm, "Kyle Hamilton"<aerow...@gmail.com> wrote: >> I believe that Startcom (and other certification authorites in >> Mozilla's root program) would likely have cause to bring an action for >> the tort of negligence against Mozilla. I feel that this is something >> that Mozilla should likely ask its general counsel very quickly.
> Apart from the legal and procedural ramifications of all this, can you > guys publish a manual procedure for those of us who want to remove > Comodo's certificates from our browser installations? Assuming one is > available?
Select Preferences -> Advanced -> View Certificates -> Authorities. Search for AddTrust AB -> AddTrust External CA Root and click "Edit". Remove all Flags.
This would remove the trust from the potentially affected sites and their certificates. Comodo has many more roots if you are interested, such as all AddTrust, Comodo CA Limited, The Usertrust Network.
>If they don't shut that site, we can perhaps just publish the private key for the mozilla.com certificate as well so everybody can enjoy it.
It is indeed unbelievable to hear the COO of a CA company making threats like this. I'm sure that making such threats is not covered by Mozilla's inclusion policies, but maybe it should be.
And, yes, I'm serious. Given that Startcom has the ability to issue bogus certificates like the kind that Eddy is threatening, I would think that a public statement like the above is relevant to Mozilla or Microsoft deciding whether or not the organization is trustworthy.
