Unbelievable!

1092 views
Skip to first unread message

Eddy Nigg

unread,
Dec 22, 2008, 5:25:23 PM12/22/08
to
https://blog.startcom.org/?p=145

--
Regards

Signer: Eddy Nigg, StartCom Ltd.
Jabber: star...@startcom.org
Blog: https://blog.startcom.org

Robin Alden

unread,
Dec 22, 2008, 8:15:35 PM12/22/08
to mozilla's crypto code discussion list
Eddy,
That reseller's ability to sell Comodo certificates has been
suspended while we investigate why they are apparently not fulfilling their
contractual obligations to us.
We revoked your certificate for mozilla.com.

Regards
Robin Alden
Comodo

> _______________________________________________
> dev-tech-crypto mailing list
> dev-tec...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tech-crypto

Eddy Nigg

unread,
Dec 22, 2008, 8:33:46 PM12/22/08
to
On 12/23/2008 03:15 AM, Robin Alden:

> Eddy,
> That reseller's ability to sell Comodo certificates has been
> suspended while we investigate why they are apparently not fulfilling their
> contractual obligations to us.

How can you outsource such a critical part as domain control validation
to a reseller is a complete mystery to me! Your controls (if any) have
completly failed.

And apparently if the fish stinks at the tail, chances that the head is
rotten too are pretty high.

> We revoked your certificate for mozilla.com.
>

I suggest to revoke ALL certificates from this reseller and perform an
urgent review of your policies and implementations in relation to
resellers at all. Other steps might be needed as well, you know better
than me.

Robin Alden

unread,
Dec 22, 2008, 8:38:14 PM12/22/08
to mozilla's crypto code discussion list
Eddy,

As I noted in my prior correspondence, Comodo has undertaken an internal
review of the Certstar reseller account. We have informed CertStar that
their email violates their contractual obligation to refrain from sending
unsolicited emails and that their email could be interpreted as misleading
and confusing to the customer. During our review, we discovered that
Certstar had apparently issued a certificate to mozilla.com without
validating control of the domain. We immediately revoked the certificate
(prior to your posting) and have suspended Certstar's reseller activities
until our investigation has been completed. Please let me know if you have
any further problems.

Regards
Robin Alden
Comodo

> -----Original Message-----
> From: dev-tech-crypto-bounces+robin=comod...@lists.mozilla.org
> [mailto:dev-tech-crypto-bounces+robin=comod...@lists.mozilla.org] On
> Behalf Of Eddy Nigg

Eddy Nigg

unread,
Dec 22, 2008, 8:47:16 PM12/22/08
to
On 12/23/2008 03:38 AM, Robin Alden:

> Eddy,
>
> As I noted in my prior correspondence, Comodo has undertaken an internal
> review of the Certstar reseller account. We have informed CertStar that
> their email violates their contractual obligation to refrain from sending
> unsolicited emails and that their email could be interpreted as misleading
> and confusing to the customer. During our review, we discovered that
> Certstar had apparently issued a certificate to mozilla.com without
> validating control of the domain. We immediately revoked the certificate
> (prior to your posting) and have suspended Certstar's reseller activities
> until our investigation has been completed.

A.) The certificate was revoked after posting to this list.
B.) Your CA also issued a certificate to startcom.org without validating
anything.
C.) Yes, I think we have a problem of a wider scale here. This is not
about me personally.

Kyle Hamilton

unread,
Dec 22, 2008, 8:53:07 PM12/22/08
to mozilla's crypto code discussion list
I believe that this is a symptom of faulty internal control, and that
this is direct evidence of not living up to the Mozilla CA policy.

I advocate at least temporarily removing the trust bits from Comodo
until a new external audit can be completed, with an eye toward
ensuring that Comodo, not the reseller, perform the domain
validations.

-Kyle H

Frank Hecker

unread,
Dec 23, 2008, 12:09:50 AM12/23/08
to
Kyle Hamilton wrote:
> I advocate at least temporarily removing the trust bits from Comodo
> until a new external audit can be completed, with an eye toward
> ensuring that Comodo, not the reseller, perform the domain
> validations.

There are two general reasons for pulling a root, to address a clear and
present danger to Mozilla users, and to punish a CA and deter others. My
concern right now is with the former. I see at least three issues in
relation to that:

1. Issuance of further non-validated certs by this reseller. Comodo
seems to have addressed this by suspending the reseller's ability to get
certs issued. (I can testify that this is the case, as I tried to
duplicate Eddy's feat earlier today and got my uploaded CSR rejected.)

2. Potential problems with certs already sold through this reseller.
Comodo should investigate this and take action if needed. (This need not
necessarily require revoking all certificates associated with the
reseller; for example, the existing certs and their associated domains
could be re-validated, the registered domain owners could be notified of
the potential for bogus certs floating around, etc.)

3. Potential problems with other Comodo resellers. I'm not going to tell
Comodo how to operate its reseller network, but they certainly should
take a look at whether and where this might be a problem with other
resellers, and how they could revamp their systems to reduce potential
problems with resellers.

Pulling a Comodo root will knock out Firefox, etc., access to thousands
of SSL sites, maybe tens of thousands. Given the disruption that would
cause, the final decision on this IMO should be made in conjunction with
the Firefox security folks. From my point of view I'd wait on more
information regarding items 2 and 3 above before making a recommendation.

Frank

--
Frank Hecker
hec...@mozillafoundation.org

Kyle Hamilton

unread,
Dec 23, 2008, 2:09:35 AM12/23/08
to mozilla's crypto code discussion list, mozilla-...@mozilla.org
I agree with all three of these issues -- it's not *just* this
reseller, it's potentially the entire reseller network, and it's
entirely possible that the problem exists across multiples.
(Especially if Comodo delegates full Registration Authority capability
without verification, which seems to be the case -- though they could
have simply issued a sub-CA certificate.)

It occurs to me that there is no facility in Firefox or other Mozilla
products to provide an explanatory dialog that there's an issue, and
such a facility would be extremely useful at this point. Being able
to print a message to the user like "The Mozilla Foundation has
identified issues with the trusted root that issued this certificate
which prevent Firefox from being able to guarantee that this is truly
the site to which you intended to go. While it is unlikely that this
is a widespread problem, and an attack would rely on more technical
intrusions into the network, the nature of these issues requires that
you be warned of this circumstance so that you can exercise
appropriate levels of caution. The Mozilla Foundation is working with
the trusted root to resolve these issues." would help a lot.

(I word it like that because in order for an attacker to succeed he
would need to also hijack DNS, or place a entry in the user's hosts
file.)

Of course, this would be an NSS change (the addition of a 'trust
suspended' bit, in addition to the trust flags), a Firefox/Thunderbird
code change (to look for that bit), and a chrome change (to explain
what's going on). Or even a check for the name in the hosts file to
see if it's overridden from DNS. Placing an entry in a user's hosts
file is easier than hijacking DNS, or at least it's supposed to be.
I'm pretty sure we're all aware of the fairly recent cache poisoning
attack, but the people who write BIND pushed out a change to protect
against it very rapidly.

The addition of a 'trust suspended' bit is primarily unlikely to
happen, though, because there's too much inertia in the way things are
to do what needs to be done to fix the authentication system to allow
a root to stay in while the user is potentially at risk.

-Kyle H

patr...@certstar.com

unread,
Dec 23, 2008, 3:48:12 AM12/23/08
to
Hi all,

A glitch in our validation system has today caused a certificate to be
issued to a person who successfully abused our system.

We have now strengthened our domain validation system so that such
abuse cannot happen again. Comodo has handled this issue in a
professional way by invoking the certificate immediately after issuing
and contacting Certstar.

Again, I cannot stress enough how seriously we take this issue and I
would like to apologize to the Mozilla organization for the mis-issue.

--
kind regards,
Patricia, Certstar ApS

Thorsten Becker

unread,
Dec 23, 2008, 3:54:07 AM12/23/08
to
Hi Patricia,

patr...@certstar.com schrieb:

> We have now strengthened our domain validation system so that such
> abuse cannot happen again.

just curious: How do you normally validate domain ownership?

TIA,

Thorsten

Eddy Nigg

unread,
Dec 23, 2008, 4:42:49 AM12/23/08
to
On 12/23/2008 10:48 AM, patr...@certstar.com:

> Hi all,
>
> A glitch in our validation system has today caused a certificate to be
> issued to a person who successfully abused our system.

It's not me who abused your system, it's your company which sent out
illegal, misleading emails to our customers. Since I couldn't find out
who stands behind your site I had to get a certificate. There was no
abuse, there was only use-and-get.

- No affiliation published, the fact that you are a reseller for Comodo
is nowhere noted.
- No subscriber agreement or notice presented to subscribers.
- No control validation of the domain name. Just pay-and-go.

I didn't had to do anything to overcome any validation whatsoever. There
was no abuse!

> We have now strengthened our domain validation system so that such
> abuse cannot happen again. Comodo has handled this issue in a
> professional way by invoking the certificate immediately after issuing
> and contacting Certstar.

Yes, after I posted my article here at the mailing list.

> Again, I cannot stress enough how seriously we take this issue and I
> would like to apologize to the Mozilla organization for the mis-issue.
>


--

Eddy Nigg

unread,
Dec 23, 2008, 5:03:18 AM12/23/08
to
For those interested, Frank opened a bug to investigate this incident:

https://bugzilla.mozilla.org/show_bug.cgi?id=470897

Eddy Nigg

unread,
Dec 23, 2008, 5:39:03 AM12/23/08
to
On 12/23/2008 07:09 AM, Frank Hecker:

> There are two general reasons for pulling a root, to address a clear and
> present danger to Mozilla users, and to punish a CA and deter others. My
> concern right now is with the former. I see at least three issues in
> relation to that:
>
> 1. Issuance of further non-validated certs by this reseller. Comodo
> seems to have addressed this by suspending the reseller's ability to get
> certs issued. (I can testify that this is the case, as I tried to
> duplicate Eddy's feat earlier today and got my uploaded CSR rejected.)

As long as this site keeps operating, our customers are still being let
to believe that they have to renew their certificate with them. This is
only a reminder about how it started at all. CAs and their customers are
still taking damage from the previously sent messages.

>
> 2. Potential problems with certs already sold through this reseller.
> Comodo should investigate this and take action if needed. (This need not
> necessarily require revoking all certificates associated with the
> reseller; for example, the existing certs and their associated domains
> could be re-validated, the registered domain owners could be notified of
> the potential for bogus certs floating around, etc.)

You shouldn't notify the subscribers or domain name owners, but the
relying parties. How to do that is up to you and Comodo I guess.

Comodo not only shouldn't just investigate and take action, Comodo needs
to report publicly about their findings and full report about the
actions taken. This isn't a suggestion of resolution about this
incident, it's the transparency I expect from them at this hour.

> Pulling a Comodo root will knock out Firefox, etc., access to thousands
> of SSL sites, maybe tens of thousands.

I'm not advocating removing their root, however we must assess the risk
which is potentially caused to the relying parties. There may be
thousands of sites which received certificates without validating them.

> Given the disruption that would
> cause, the final decision on this IMO should be made in conjunction with
> the Firefox security folks.

Disabling the trust bits of "AddTrust External CA Root" could be a
temporary measure to prevent damage to relying parties until Mozilla
receives full report and disclosure from Comodo about its resellers and
conclusion of their investigation.

Additionally instead of just yanking a root as a deterrent and
punishment, as you mentioned above, I'd prefer to receive a commitment
from Comodo to address other issues noted during the last discussion -
mainly those listed in the "Problematic Practices" document. Considering
that OCSP in Firefox is set to soft-fail, an issue with their OCSP
server could still circumvent bogus sites for potentially the next five
years. A full review of their current status in NSS and their practices
might be advocated too.

Eddy Nigg

unread,
Dec 23, 2008, 6:36:47 AM12/23/08
to
On 12/23/2008 09:09 AM, Kyle Hamilton:

> (I word it like that because in order for an attacker to succeed he
> would need to also hijack DNS, or place a entry in the user's hosts
> file.)

Or be a WiFi operator. This was the attack vector of
https://bugzilla.mozilla.org/show_bug.cgi?id=460374

>
> Of course, this would be an NSS change (the addition of a 'trust
> suspended' bit,

I think this to be an interesting idea and should be considered.

Frank Hecker

unread,
Dec 23, 2008, 8:05:35 AM12/23/08
to
Eddy Nigg wrote:
> For those interested, Frank opened a bug to investigate this incident:
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=470897

Actually Nelson opened this bug.

Kyle Hamilton

unread,
Dec 23, 2008, 8:09:42 AM12/23/08
to mozilla's crypto code discussion list
Patricia, I believe it's important to realize a couple of things:

1) An unsolicited commercial email (UCE) message was sent from your
company to the party in question suggesting that there already existed
a relationship between your company and the party in question. This
is obvious from the verb 'renew' in the original message -- a
non-party to the original certificate can't renew "on behalf of" the
original certificate issuer. If they can, there's a major problem.

1a) The message from your company, and in fact the entire process up
to and including paying for the certificate which your company issued,
did not expressly claim an affiliation, and the party in question went
through the process of "renewal" with the intent of figuring out which
CA's services were being advertised via UCE.

2) The party in question (Eddy Nigg) is the founder of another
Certifying Authority, with a root which participates in the Mozilla
root program and which is also included in the root store.

3) It falls within the concept of "due diligence" for a
security-conscious warden of a public trust to recognize the lack of
actual domain control verification in a trusted certificate issuance
process, and *for the purposes of verification and public
notification* obtain a certificate which could be shown absolutely to
have been issued in violation of the standards of at least one of the
root programs that the issuer (or issuer's trust-delegator)
participates in.

3a) If this had not been identified, someone else would have found it
eventually -- and the reaction to a gross violation of security
standards is to immediately believe that the worst-case scenario has
occurred: that it had already been found and exploited by at least one
other person.

4) The end-users (referred to as "relying parties") are well-served by
this identification of completely bogus credentialing.

5) It's also important to recognize the following (the first comment
on https://bugzilla.mozilla.org/show_bug.cgi?id=470897):

------- Comment #1 From Reed Loden [:reed] 2008-12-23 01:35:37
PST -------

The same company that Eddy was able to get the mozilla.com cert from (Certstar)
has been endlessly spamming webm...@mozilla.org since the beginning of
December complaining that one of our SSL certs had "expired" and needed to be
"renewed" (both of which were false). They have continued to spam us almost
daily. :(

------ end comment ------

Yes, the bare facts are that a user exploited the lack of verification
in your certificate issuance process. However, the lack of
verification in your process violates at least one contractual
obligation that your company is required to uphold, and reduces the
value of both Comodo's root AND the commercial Certifying Authority
structure in general. Because the user founded another commercial CA
which is trusted by Mozilla NSS by-default, this user was operating in
a manner to verify that the commercial CA structure was in fact
secure; the trust afforded this user's CA is demonstrably harmed by
your lack of verification.

I do not take this situation as lightly as you appear to wish that
everyone would. The initial fault is YOUR COMPANY'S (and that fault
reverbrates up to Comodo, since it delegated trust to your company),
and the fact that you're attempting to shift the blame onto the user
shows that you are absolutely untrustworthy to run or be the public
face of any commercial certificate-issuance service.

Because of this, my recommendation that Comodo's trust bits be removed
until a full audit of their practices (and a full audit of all issued
certificates) stands, and I am that much more resolute in my belief.
It may be that the integrity of Comodo's root has been irreparably
damaged by your company's malfeasance; if this is the case, I
certainly hope they rake you over the coals.

-Kyle H

Frank Hecker

unread,
Dec 23, 2008, 8:14:42 AM12/23/08
to
Eddy Nigg wrote:
> Disabling the trust bits of "AddTrust External CA Root" could be a
> temporary measure to prevent damage to relying parties until Mozilla
> receives full report and disclosure from Comodo about its resellers and
> conclusion of their investigation.

Do you mean the UTN-UserFirst-Hardware root? According to the screenshot
on your blog post, that's the root the bogus cert chains up to. Also, if
we were to take action of this general sort (as a hypothetical), what
about adding the PositiveSSL CA cert to NSS with the SSL trust bit
disabled; wouldn't that accomplish the same purpose, without interfering
with other parts of the hierarchy under the UTN-UserFirst-Hardware root?
(I seem to recall we've discussed this sort of thing in the past.)

Also note that any "suspension" of a root would last at last 1-3 months,
since that the typical interval between security updates for Firefox and
other Mozilla-based products.

Eddy Nigg

unread,
Dec 23, 2008, 8:15:56 AM12/23/08
to
On 12/23/2008 03:05 PM, Frank Hecker:

> Eddy Nigg wrote:
>> For those interested, Frank opened a bug to investigate this incident:
>>
>> https://bugzilla.mozilla.org/show_bug.cgi?id=470897
>
> Actually Nelson opened this bug.
>

Thanks for that. More into this story...

...all our employees coming the our network are able to retrieve the
certificates and all the rest from the "control panel" of this site.
Again, no questions asked. Apparently the only things one needs is to
fake our gateway IP address to get there.

It's getting weirder with the second. If they don't shut that site, we
can perhaps just publish the private key for the mozilla.com certificate
as well so everybody can enjoy it.

Kyle Hamilton

unread,
Dec 23, 2008, 8:17:14 AM12/23/08
to mozilla's crypto code discussion list
Either way, it's enough of a problem that a bug needs to be opened,
and it needs to be followed upon sooner than later.

(My opinion, of course. Comodo's gotta be breathing a sigh of relief
that I don't work for the Mozilla Foundation, else I would have yanked
the trust bits first and figured it out later.)

This incident shows anew the lack of policy/procedure of what to do in
the case of a gross breach of the root agreement, though. The entire
point of certificates in the first place is "user security", not "user
convenience". (If this weren't the case, none of us would have any
desire to get certificates in the first place.) Your reaction seems
to be based on placing "user convenience" over "user security", and
(again, my opinion) I don't believe that this is appropriate at all.

-Kyle H

Ian G

unread,
Dec 23, 2008, 8:58:45 AM12/23/08
to mozilla's crypto code discussion list
In the past, lots of good stuff has been done that handles the ascension
to the root list of Mozilla. c.f. the policy. But not so much is
written about *what happens afterwards*. This recent thread has been
such a case, and has afforded an opportunity to make some notes on what
might be suitable as a practices page on the wiki.

The reason for doing this, and adding yet more boring verbage to the
mass of words, is that it can prevent uneccessary legal costs by
providing clear guidepaths, and clear signals of seriousness.

Here's what I see:

1. How to file a dispute against Mozilla and/or a CA. This seems
fairly easy; file a bug in bugzilla and mark it as already described
here: https://wiki.mozilla.org/CA:How_to_apply . (With mods: Severity:
as appropriate.)

2. What is the practice on revocation or un-trusting on roots? This is
is perhaps a headline case of a dispute, so possible merits its own
notes. Frank has suggested the test:

a. there is a clear and present danger to Mozilla users, or
b. to punish a CA, and/or to deter others.

I would suggest that (b) be modifed slightly to give it a basis, which
in this case might be "for breaches of policy or practices". The
Policy, pt 4, gives the authority, as well as some examples, and adds
another test case:

c. where certificates cause technical problems with mozo software.

3. How to resolve a dispute. This is a Mozilla action &
responsibility. Reverse-engineering and referring, I would suggest this
as a teaser:

a. The CA certificate "module owner" at Mozilla foundation is
responsible. Ref, the policy, pt 15.
b. The dispute is investigated and ruled on by module owner.
c. The ruling is listed in the bug report above.
d. Many disputes will be dealt with by communication, and no ruling
will be required. This will create a default "closed, no action" ruling.

4. Finality. What happens if we disagree with the decision of the
module owner? In the policy, it says "CAs or others objecting to a
particular decision may appeal to mozilla.org staff, who will make a
final decision." Ref, policy, pt 15.

I would wonder about this; google suggests that "staff" is as listed here:
http://www.mozilla.org/about/staff
but that seems out of date. Also, due to the absence of this forum in
the public eye, I doubt it musters the credibility we need in dispute
review where the legal and contractual significance is high. E.g., is
there any way we can review the decisions they made in the past?

There are several possibilities:

(i) Ruling is final.
(ii) Mozilla.org staff, policy, pt 15.
(iii) Review by board of Mozilla Foundation.
(iv) Review by some independent party.
(v) Review by forum at law: courts, or Arbitrator.

Personally, I would plumb for (iii) and suggest the Mozo Foundation
board as the next step. It is expensive, but available. The directors
already have fiduciary responsibility, and can thus deal with the
significance. It is also aligned with the review of the manager
concerned, the policy and the general contractual issues.

End! I'd encourage others to comment!

If nobody has any objections, I can add that as a page into the wiki as
an informal document of practices, including or without those questions.


iang

Frank Hecker

unread,
Dec 23, 2008, 9:35:00 AM12/23/08
to
Kyle Hamilton wrote:
> Your reaction seems
> to be based on placing "user convenience" over "user security", and
> (again, my opinion) I don't believe that this is appropriate at all.

My intent is to balance the disruption that would be caused by pulling a
root vs. the actual security threat to users. Right now we have no real
idea as to the extent of the problem (e.g., how many certs might have
been issued without proper validation, how many of those were issued to
malicious actors, etc.).

Frank Hecker

unread,
Dec 23, 2008, 9:48:18 AM12/23/08
to
Ian G wrote:
> In the past, lots of good stuff has been done that handles the ascension
> to the root list of Mozilla. c.f. the policy. But not so much is
> written about *what happens afterwards*.

I don't have time right now to address your whole post, but I did want
to comment on one point.

> 4. Finality. What happens if we disagree with the decision of the
> module owner? In the policy, it says "CAs or others objecting to a
> particular decision may appeal to mozilla.org staff, who will make a
> final decision." Ref, policy, pt 15.
>
> I would wonder about this; google suggests that "staff" is as listed here:
> http://www.mozilla.org/about/staff
> but that seems out of date.

There is no longer a "mozilla.org staff" as traditionally constituted.
From a Mozilla community point of view the closest present-day
equivalent is the system described at

https://wiki.mozilla.org/Module_Owners_Activities_Modules

Basically there is a group of people that oversee the Mozilla module
ownership system and the module owners within it. That group in turn
operates under general policy structures set by Mitchell Baker and
Brendan Eich, and as you note the ultimate legal authority resides in
the board of the Mozilla Foundation (of which Mitchell and Brendan are
members).

For more background on this see

http://www.mozilla.org/hacking/module-ownership
http://blog.lizardwrangler.com/2008/05/30/governance-and-module-ownership/

Gervase Markham

unread,
Dec 23, 2008, 10:42:39 AM12/23/08
to Nelson B Bolyard
Frank Hecker wrote:
> Do you mean the UTN-UserFirst-Hardware root? According to the screenshot
> on your blog post, that's the root the bogus cert chains up to. Also, if
> we were to take action of this general sort (as a hypothetical), what
> about adding the PositiveSSL CA cert to NSS with the SSL trust bit
> disabled; wouldn't that accomplish the same purpose, without interfering
> with other parts of the hierarchy under the UTN-UserFirst-Hardware root?
> (I seem to recall we've discussed this sort of thing in the past.)

This was the first thing that occurred to me as well. Can anyone
(Nelson?) tell us definitively if this is possible? I would assume it
is, but it would be good to get confirmation.

Gerv

Gen Kanai

unread,
Dec 23, 2008, 1:17:18 PM12/23/08
to mozilla's crypto code discussion list
Are we going to receive information from Comodo regarding how many
other Comodo resellers may be in a similar position to Certstar?

Are we going to receive information from Certstar as to how many other
certs may have been issued in error?

How do we verify the claims from Comodo or Certstar?

Gen

Kyle Hamilton

unread,
Dec 23, 2008, 1:30:18 PM12/23/08
to mozilla's crypto code discussion list
These are, of course, the reason why I want the trust bits removed.
It's going to take more than a month to get this all straightened out,
and that's "over a month" that we have no idea if Comodo certificates
can be trusted.

-Kyle H

Frank Hecker

unread,
Dec 23, 2008, 1:43:30 PM12/23/08
to
Gen Kanai wrote:
> Are we going to receive information from Comodo regarding how many other
> Comodo resellers may be in a similar position to Certstar?
>
> Are we going to receive information from Certstar as to how many other
> certs may have been issued in error?

I've asked Robin Alden of Comodo to make an accounting regarding these
two issues. I don't expect to see that immediately (i.e., in the next
day or two), though I also don't expect to wait a month for it (as Kyle
is concerned about).

> How do we verify the claims from Comodo or Certstar?

Some things by their nature we can directly verify, and some we can't.
For example, we can certainly verify how Certstar is operating currently
(as Eddy and I did), and if certs get revoked we can verify that by
downloading the CRLs that get issued.

Hendrik Weimer

unread,
Dec 23, 2008, 2:15:25 PM12/23/08
to mozilla's crypto code discussion list
Frank Hecker <hec...@mozillafoundation.org> writes:

> My intent is to balance the disruption that would be caused by pulling
> a root vs. the actual security threat to users. Right now we have no
> real idea as to the extent of the problem (e.g., how many certs might
> have been issued without proper validation, how many of those were
> issued to malicious actors, etc.).

Isn't that, by itself, a very good reason to take immediate action?
Security should be default-fail rather than default-pass.

Hendrik

Kyle Hamilton

unread,
Dec 23, 2008, 2:23:16 PM12/23/08
to mozilla's crypto code discussion list
On Tue, Dec 23, 2008 at 10:43 AM, Frank Hecker
<hec...@mozillafoundation.org> wrote:
> I've asked Robin Alden of Comodo to make an accounting regarding these two
> issues. I don't expect to see that immediately (i.e., in the next day or
> two), though I also don't expect to wait a month for it (as Kyle is
> concerned about).

Actually, I think it's very important that the accounting include this:

for each name (not just certificate, but name in
subjectAlternativeNames) that has been certified, a connection to the
TLS ports should be made, and the certificate presented by the site
compared against the certificate that Comodo issued. This obviously
won't be a complete verification, but it should give a start to see
how widespread the problem is.

A script to do this could probably be written fairly easily, but
depending on the number of certificates Comodo has issued that are
currently valid (and I'd like to see some hard numbers on that, as
well) it could take a while to run.

From the script, the numbers I'd like to see are: the number of
unreachable/not-answering names/hosts, the number of matching
certificates, and the number of mismatched certificates. From that
output plus Comodo's records, I would also like to see how many
resellers there are and how many of them have sold mismatched
certificates.

I hate to say this, but this IS The Worst-Case Scenario. A CA has
gone rogue and issued certificates that violate its standards, and the
standards of the root programs that it's a part of -- it is true that
Comodo didn't /intend/ to go rogue, but it has, and we can't afford to
let it damage the greater PKI. Since every CA in the root store is
treated the same, there is no differentiation between them -- and this
means that Verisign and Comodo and Thawte and *every* CA share the
same reputation. If one goes rogue, it's exactly the same as if all
of them have gone rogue, in the eye of the end-user.

To put it another way, it's exactly the same problem as putting the
public keys of webservers in DNSSEC. Since the end-system can't know
if a response came from DNSSEC or just unauthenticated DNS, the
end-system can't trust anything (including RFC2538-style CERT records)
that comes from DNS.

THIS is why I want to see greater differentiation in the browser
chrome between CAs, so that one bad apple doesn't spoil the whole root
barrel. THIS is why the argument against changing the chrome (user
convenience) fails.

-Kyle H

Kyle Hamilton

unread,
Dec 23, 2008, 2:27:53 PM12/23/08
to mozilla's crypto code discussion list
I'd rather deal with disruption caused thereby (and, yes, the user
complaints generated thereby -- at least then the end-user would KNOW
that there's a problem that's being dealt with rather than having a
FALSE SENSE OF SECURITY) than let those potential security breaches
continue to wreak their quiet little havoc.

-Kyle H

On Tue, Dec 23, 2008 at 11:15 AM, Hendrik Weimer <hen...@enyo.de> wrote:
> Frank Hecker <hec...@mozillafoundation.org> writes:
>
>> My intent is to balance the disruption that would be caused by pulling
>> a root vs. the actual security threat to users. Right now we have no
>> real idea as to the extent of the problem (e.g., how many certs might
>> have been issued without proper validation, how many of those were
>> issued to malicious actors, etc.).
>

> Isn't that, by itself, a very good reason to take immediate action?
> Security should be default-fail rather than default-pass.
>
> Hendrik

Eddy Nigg

unread,
Dec 23, 2008, 2:49:20 PM12/23/08
to
On 12/23/2008 09:15 PM, Hendrik Weimer:

It should be. I realized that there are more points which will have to
be addressed in due time at Mozilla, which however can wait for now.

Concerning the disruption, Comodo has many roots and the resetting of
this specific root would affect low-assurance sites as far as I know.
The higher validated sites would not be affected. Having said that, the
roots of their low-assurance products have been previously of concern it
must be looked at in the broader context of Comodo's operations. It's
not surprising in itself.

Justin Dolske

unread,
Dec 23, 2008, 3:12:27 PM12/23/08
to
On 12/23/08 11:27 AM, Kyle Hamilton wrote:
> I'd rather deal with disruption caused thereby (and, yes, the user
> complaints generated thereby -- at least then the end-user would KNOW
> that there's a problem that's being dealt with rather than having a
> FALSE SENSE OF SECURITY)

Hmm, would they?

Justin Dolske

unread,
Dec 23, 2008, 3:20:15 PM12/23/08
to

*sigh* I somehow managed to click "Send" mid-message.

Anyway, the rest of my incomplete thought:

...I think there's some risk that if a Firefox update suddenly breaks a
large swath of legitimate SSL sites, that could end up training users to
ignore the problem. I'm not even sure how you'd present this condition
to Joe The User in a comprehensible way.

That said, the Comodo/Certstar is hugely sucky and I would hope there's
something we can do about it that helps users.

Justin

Daniel Veditz

unread,
Dec 23, 2008, 3:23:27 PM12/23/08
to
Frank Hecker wrote:
> Eddy Nigg wrote:
>> Disabling the trust bits of "AddTrust External CA Root" could be a
>> temporary measure to prevent damage to relying parties
>
> Also note that any "suspension" of a root would last at last 1-3 months,
> since that the typical interval between security updates for Firefox and
> other Mozilla-based products.

And we don't have a magic switch we can flip in the office. We'd have to
make the change, test the change, make the builds, ship the builds,
users would have to update (about a week from ship until most users have
the update).

If the sole purpose of the update was to break lots of sites (from the
user's POV) then some number of them disable updates, making them less
secure in the future.

If Comodo is acting in good faith then anything they can do would be
lightyears faster than a client update. If they're not fulfilling their
responsibilities then a permanent removal would make sense, but given
the time scales it's hard to see how a "temporary" month-or-so removal
helps.

Maybe we need to build in something like a CRL that pings back to
Mozilla that would let us revoke roots without having to ship a client
update.

Eddy Nigg

unread,
Dec 23, 2008, 3:43:17 PM12/23/08
to
On 12/23/2008 10:23 PM, Daniel Veditz:

>
> Maybe we need to build in something like a CRL that pings back to
> Mozilla that would let us revoke roots without having to ship a client
> update.

Of course we (@ mozilla) also take our lessons from this event, I'm
sure. Indeed it was previously suggested here and I think we'll have to
look at such options in due time.

Anything other than taking down their site is insufficient (as an
immediate action to prevent further damage), specially in light of
apparently more flaws in their system. Not having done that as I
requested already two days ago isn't exactly comforting. Doesn't show
really cooperation and good faith here...

...they'd do that better now before any more damage happens. I've warned
them and requested to shut them down. My warning was ignored, their bad...

dou...@theros.info

unread,
Dec 23, 2008, 3:44:13 PM12/23/08
to

I, for example, have a ssl cert from comodo reseller, and they DO have
made all the validation steps.

My site, a legitimate one, would be in trouble with this. Are you all
sure that it is a good measure to just knock off the root cert or
security bit?

please, think twice

Christoph Moormann

unread,
Dec 23, 2008, 4:08:08 PM12/23/08
to

I puzzled by the security approach on this issue. There was a severe
security problem, and at least two, maybe many certs are compromised,
who knows by this point.
Do i still have trust in the Certs of Comodo or their signing
resellers? No, until the whole issue is cleared up completely.
Now it's the users duty to check every Cert on the issuer to evaluate
the trustlevel, while his browser states, everything is fine?
This is not about punishing Comodo, their resellers or their
customers, this is about trusting my browser software.

Ian G

unread,
Dec 23, 2008, 4:12:21 PM12/23/08
to mozilla's crypto code discussion list
On 23/12/08 20:23, Kyle Hamilton wrote:
> On Tue, Dec 23, 2008 at 10:43 AM, Frank Hecker
> <hec...@mozillafoundation.org> wrote:
>> I've asked Robin Alden of Comodo to make an accounting regarding these two
>> issues. I don't expect to see that immediately (i.e., in the next day or
>> two), though I also don't expect to wait a month for it (as Kyle is
>> concerned about).
>
> Actually, I think it's very important that the accounting include this:
>
> for each name (not just certificate, but name in
> subjectAlternativeNames) that has been certified, a connection to the
> TLS ports should be made, and the certificate presented by the site
> compared against the certificate that Comodo issued. This obviously
> won't be a complete verification, but it should give a start to see
> how widespread the problem is.


Um. Some questions... On what basis are we asking for this "accounting" ?

Is there a criteria that calls for public breach accounting? Perhaps
Mozilla needs to add it? Most of the criteria call for revocation when
a false cert has been issued, and that seems to have been met right now.

Also, some of the process might instead rest with Comodo's external
auditor or its internal audit process or its internal security process.
Perhaps in the next audit, there will be an examination of the
remedial steps taken, to see if they conform to the security manual, etc.

There are some slippery slope aspects here. If the framework for such
"accounting" is outside Mozilla's area, then care might be needed.
There could be confusion in role and liabilities.

It's nice if Mozilla were minded to do all the audit and security
activities for all, but I'm not sure Frank has time ;) OTOH, if we feel
the need to usurp the CA's processes and the conventional audit
relationship with an "accounting" then perhaps we need to also simplify
some other processes?

Another aspect here is that the goal of any security or governance
process is not to eliminate or stop all breaches. The process is more
about how you keep dangers to a minimum, deal with the breaches that do
happen, and improve.


> I hate to say this, but this IS The Worst-Case Scenario.

Noooo.... With respect, I'd say worst case is that a root has been
breached, and the Carders have got hold of it and are MITMing the major
retail sites. Damages!

Earlier, Frank used the language of "clear and present danger."

* clear: we can measure the costs of it, and cost of defences.
* present: it is happening today, provably.
* danger: it can be shown capable of doing damage, at least in theory

Only the last one is shown. It is a danger in that anyone relying on a
cert wrongly issued could be harmed. But it hasn't actually caused
anyone damages, and nobody has shown it to exist.

This situation is about as harmful as the average exploit demo. In this
particular case it was caught by an insider rather than by an outsider.
However, beyond that, the situation is already sealed in that Comodo
have already taken the urgent triage steps to make sure nobody else gets
one.


> A CA has
> gone rogue and issued certificates that violate its standards, and the
> standards of the root programs that it's a part of -- it is true that
> Comodo didn't /intend/ to go rogue, but it has, and we can't afford to
> let it damage the greater PKI.

This is no different to when Verisign got schnikered into issuing a
couple of Microsoft certs. No damages ever resulted. The system worked
as it was expected to, it seems. Panic is not called for.


> Since every CA in the root store is
> treated the same, there is no differentiation between them -- and this
> means that Verisign and Comodo and Thawte and *every* CA share the
> same reputation. If one goes rogue, it's exactly the same as if all
> of them have gone rogue, in the eye of the end-user.

...

> THIS is why I want to see greater differentiation in the browser
> chrome between CAs, so that one bad apple doesn't spoil the whole root
> barrel. THIS is why the argument against changing the chrome (user
> convenience) fails.


Oh, on that, yes, strongly agree. There's no trust without brand :)

iang

Justin Dolske

unread,
Dec 23, 2008, 4:16:05 PM12/23/08
to
On 12/23/08 12:20 PM, Justin Dolske wrote:

> That said, the Comodo/Certstar is hugely sucky and I would hope there's
> something we can do about it that helps users.

I am just full of fail today: "... the Comodo/Comstar *incident* is
hugely sucky ..."

Justin

Eddy Nigg

unread,
Dec 23, 2008, 4:25:09 PM12/23/08
to
On 12/23/2008 11:12 PM, Ian G:

> Earlier, Frank used the language of "clear and present danger."
>
> * clear: we can measure the costs of it, and cost of defences.
> * present: it is happening today, provably.
> * danger: it can be shown capable of doing damage, at least in theory
>
> Only the last one is shown. It is a danger in that anyone relying on a
> cert wrongly issued could be harmed. But it hasn't actually caused
> anyone damages, and nobody has shown it to exist.

The first and second are valid as well. We can measure the costs, can't we?
The second item shows that it can be happening today, you don't have to
prove it, because I did already. The fact that I disclosed publicly
about it doesn't mean that there aren't scores of others out there.
Considering the aggressive and illegal mailing campaign they operated,
one can expect that there are many certificates issued in this way.

>
> This situation is about as harmful as the average exploit demo. In this
> particular case it was caught by an insider rather than by an outsider.
> However, beyond that, the situation is already sealed in that Comodo
> have already taken the urgent triage steps to make sure nobody else gets
> one.
>

Well, that's exactly the point! I haven't heard back from Robin anymore,
their site is still online as well. Clear statements about how many
certificates are affected (all of the ones issued through them) and
revocations (or at least urgent re-validation within 48 hours and
thereafter revocation) would be satisfactory maybe.

The handling of the situation is insufficient, response and measures are
a joke, communication doesn't exist. Apparently this resellers business
was too good for Comodo.

Paul Hoffman

unread,
Dec 23, 2008, 4:54:22 PM12/23/08
to mozilla's crypto code discussion list
At 11:27 AM -0800 12/23/08, Kyle Hamilton wrote:
>I'd rather deal with disruption caused thereby (and, yes, the user
>complaints generated thereby -- at least then the end-user would KNOW
>that there's a problem that's being dealt with rather than having a
>FALSE SENSE OF SECURITY) than let those potential security breaches
>continue to wreak their quiet little havoc.

And I would not.

Just because a few people loudly proclaim their preferences on either side, it does not mean that their preferences should be acted on in a way that affects millions of Firefox users.

Paul C. Bryan

unread,
Dec 23, 2008, 4:59:57 PM12/23/08
to
> Just because a few people loudly proclaim their preferences on either side, it does not mean that their preferences should be acted on in a way that affects millions of Firefox users.

It was Comodo that affected millions of Firefox users; it's up to
Mozilla to protect those users by failing safe.

Paul C. Bryan

unread,
Dec 23, 2008, 5:05:13 PM12/23/08
to
Presumably it was Comodo that underwent an audit to be added to
Mozilla's roots, and Comodo should not be allowed to delegate trust to
their resellers for domain validation. If, today, trust is delegated
to their resellers, then we can't trust Comodo, period.

Although disruptive, their trust bits should be suspended. The
explanation to users: "The CA purporting to provide assurance about
the site you are trying to visit cannot be trusted. Please contact the
site operator and advise them to find a trustworthy certification
authority."

Yes, perception is that Mozilla releases code expressly to "break"
access to legitimate sites, but this is because a trusted CA has gone
rogue. Users can still jump through hoops to expressly include the
site's certificate and keep going.

The trust model for browsers should be fail-safe, even if this
inconveniences users. Better that than me and countless others
inadvertently exposing my credentials to a site pretending to be my
bank, investment house, government revenue agency, etc.

If Mozilla doesn't pull the trust bits, what's it's accountability for
any breaches that occur due to keeping the bits? With assurance must
come liability, whether from the certification authority, or those who
are implicitly trusted with vetting them.

Kyle Hamilton

unread,
Dec 23, 2008, 5:20:05 PM12/23/08
to mozilla's crypto code discussion list
The only effective and appropriate response to a root that does not
have sufficient internal controls to maintain its own security is to
remove the trust in it. If you've purchased a certificate from them
because it's trusted, and then they lose that trust, I would think
that you should be clamoring for your money back and looking for an
alternate certificate issuer rather than trying to maintain the
problem.

-Kyle H

Frank Hecker

unread,
Dec 23, 2008, 5:20:52 PM12/23/08
to
Eddy Nigg wrote:
> Concerning the disruption, Comodo has many roots and the resetting of
> this specific root would affect low-assurance sites as far as I know.

I don't think that's necessarily true. I don't think it would affect EV
sites (because of the way validation for those sites is special-cased),
but it could affect non-EV sites under other Comodo brands (I mean,
other than the PositiveSSL brand) and it could also affect other CAs
whose CA certs are cross-signed by the UTN-UserFirst-Hardware root.

Eddy Nigg

unread,
Dec 23, 2008, 5:31:13 PM12/23/08
to
On 12/24/2008 12:20 AM, Frank Hecker:

How about ADDING the chained issuing CA certificate to NSS and mark it
deliberately untrusted (no bits turned on)?

But our dilemma here shows clearly that we must have tools to act to
such threats. It's probably one of the lessons to learn from this incident.

Eddy Nigg

unread,
Dec 23, 2008, 5:39:24 PM12/23/08
to
On 12/24/2008 12:05 AM, Paul C. Bryan:

> Presumably it was Comodo that underwent an audit to be added to
> Mozilla's roots, and Comodo should not be allowed to delegate trust to
> their resellers for domain validation. If, today, trust is delegated
> to their resellers, then we can't trust Comodo, period.
>

I would second that and in light of many other problematic practices
which were discovered during their inclusion/update of EV, it's simply
too much. More than 24 hours into this, I've come to the conclusion that
this is a sever incident which requires action. If Robin can assure us
of reasonable actions from their side (as suggested previously by me) it
would serve all participants the best. Inaction and non-cooperation will
leave Mozilla with not much choice I think. Ignorance by Mozilla itself
will hunt it for a long time too. But it must happen now, either way!

(I don't think we have the time to discuss each and every aspect of RA
and reseller responsibilities and what we deem as save, I'm certain
we'll take this issue up (which apparently has about the same
implications as intermediate externally operated CAs))

Kyle Hamilton

unread,
Dec 23, 2008, 5:51:21 PM12/23/08
to mozilla's crypto code discussion list
I believe that Startcom (and other certification authorites in
Mozilla's root program) would likely have cause to bring an action for
the tort of negligence against Mozilla. I feel that this is something
that Mozilla should likely ask its general counsel very quickly.

0) Comodo is plainly found to have derelicted its duty to uphold the
Mozilla CA agreement. As a result, damage to the trust in the PKI
occurs, causing people to look outside of the PKI for solutions to the
problems that encryption and authentication can solve.

1) As a result of this Startcom and other CAs in the PKI have a
reduced market for their products. (damage)

2) Mozilla has the authority and ability to remove CAs from its trust
list if and when they are found to be negligent in their duties. It,
as an organization which attempts to vet the CAs that it includes in
its PSM and which has policies which allow for it to take action
should the policies be breached, has accepted the duty to remain
diligent. (duty, and proximity of duty, since in order to get into
the program in the first place all CAs in the PKI must accede to
Mozilla's demands of trustworthiness.)

3) Mozilla fails to remain diligent, and fails to remove trust bits
upon notice of dereliction. (causation, which leads to the trust in
the PKI being eroded further.)

I am not a lawyer, this is not legal advice, etc. I'm trying to
prevent Mozilla from having problems.

-Kyle H

Where Wolf

unread,
Dec 23, 2008, 6:10:57 PM12/23/08
to
On Dec 23, 5:51 pm, "Kyle Hamilton" <aerow...@gmail.com> wrote:
> I believe that Startcom (and other certification authorites in
> Mozilla's root program) would likely have cause to bring an action for
> the tort of negligence against Mozilla.  I feel that this is something
> that Mozilla should likely ask its general counsel very quickly.
>
Apart from the legal and procedural ramifications of all this, can you
guys publish a manual procedure for those of us who want to remove
Comodo's certificates from our browser installations? Assuming one is
available?

Eddy Nigg

unread,
Dec 23, 2008, 6:16:43 PM12/23/08
to
On 12/24/2008 01:10 AM, Where Wolf:

Select Preferences -> Advanced -> View Certificates -> Authorities.
Search for AddTrust AB -> AddTrust External CA Root and click "Edit".
Remove all Flags.

This would remove the trust from the potentially affected sites and
their certificates. Comodo has many more roots if you are interested,
such as all AddTrust, Comodo CA Limited, The Usertrust Network.

Paul Hoffman

unread,
Dec 23, 2008, 6:23:39 PM12/23/08
to mozilla's crypto code discussion list
At 3:15 PM +0200 12/23/08, Eddy Nigg wrote:
>If they don't shut that site, we can perhaps just publish the private key for the mozilla.com certificate as well so everybody can enjoy it.

It is indeed unbelievable to hear the COO of a CA company making threats like this. I'm sure that making such threats is not covered by Mozilla's inclusion policies, but maybe it should be.

And, yes, I'm serious. Given that Startcom has the ability to issue bogus certificates like the kind that Eddy is threatening, I would think that a public statement like the above is relevant to Mozilla or Microsoft deciding whether or not the organization is trustworthy.

Paul Hoffman

unread,
Dec 23, 2008, 6:27:14 PM12/23/08
to mozilla's crypto code discussion list
At 2:51 PM -0800 12/23/08, Kyle Hamilton wrote:
>I believe that Startcom (and other certification authorites in
>Mozilla's root program) would likely have cause to bring an action for
>the tort of negligence against Mozilla. I feel that this is something
>that Mozilla should likely ask its general counsel very quickly.
>
>. . .

>
>I am not a lawyer, this is not legal advice, etc.

The above is most certainly legal advice.

> I'm trying to
>prevent Mozilla from having problems.

Some people might read it differently.

Where Wolf

unread,
Dec 23, 2008, 6:29:26 PM12/23/08
to

> Select Preferences -> Advanced -> View Certificates -> Authorities.
> Search for AddTrust AB -> AddTrust External CA Root and click "Edit".
> Remove all Flags.
>
> This would remove the trust from the potentially affected sites and
> their certificates. Comodo has many more roots if you are interested,
> such as all AddTrust, Comodo CA Limited, The Usertrust Network.
>
> --
> Regards
>
> Signer: Eddy Nigg, StartCom Ltd.
> Jabber: start...@startcom.org
> Blog:  https://blog.startcom.org

Thanks Eddy. I found the option screen shortly after I posted here,
but I was assuming that the affected certificates would be listed
under Comodo's branch, which apparently wouldn't have done me a bit of
good. Sometimes it pays to ask even when you think you know the answer
(or if you're a teacher or lawyer, ONLY ask if you already know the
answer).

BTW I tracked into this discussion from your blog, so thanks for
bringing this whole thing to my attention as well.

Eddy Nigg

unread,
Dec 23, 2008, 6:45:59 PM12/23/08
to
On 12/24/2008 01:23 AM, Paul Hoffman:

> At 3:15 PM +0200 12/23/08, Eddy Nigg wrote:
>> If they don't shut that site, we can perhaps just publish the private key for the mozilla.com certificate as well so everybody can enjoy it.
>

Paul, you are disappointing me! I have not heard one critical word from
you about this incident, instead you are criticize *me*? C'mon, give me
a break!

I reported that my employees can see the supposedly private control
panel of this reseller - what else is needed to get this site down?

> And, yes, I'm serious. Given that Startcom has the ability to issue bogus certificates like the kind that Eddy is threatening, I would think that a public statement like the above is relevant to Mozilla or Microsoft deciding whether or not the organization is trustworthy.

I don't need the services of Comodo for that, if I would have ever
wanted to that, I could do so long time ago. Everybody in his right mind
understands exactly what I meant to say in this post. If you didn't,
that's your problem.

--
Regards

Signer: Eddy Nigg, StartCom Ltd.

Jabber: star...@startcom.org
Blog: https://blog.startcom.org

patr...@certstar.com

unread,
Dec 23, 2008, 6:58:28 PM12/23/08
to
Dear all,

I just wanted to give you all an update from Certstar. As you all know
we failed to validate a certificate due to a flaw in our system which
is clearly unacceptable.

Having worked intensively with this case I can truly say that Comodo
is indeed taking their responsibility extremely seriously and taking
huge efforts to ensure that safety of their root. Currently we are
going through a number certificate to double ensure that no further
mis-issuances have occurred.

Personally I do feel that it would have been appropriate if we had
been contacted by StartCom Ltd when they found this flow so that it
could have been fixed faster. Being our competitor I am not sure if we
could expect this, but it would indeed have been generous move.

The technical verification procedure has been improved and is now on a
very high security level. Comodo will also review our implementation
to ensure that it comply with all standards and cannot be abused.
Again, I would like to apologize to the community. Nothing does
however indicate that the flow was abused by others – but still we
have made a huge mistake which I sincerely apologize for.


--
kind regards,
Patricia, Certstar ApS

patr...@certstar.com

unread,
Dec 23, 2008, 7:12:49 PM12/23/08
to
On Dec 24, 12:45 am, Eddy Nigg <eddy_n...@startcom.org> wrote:
> I reported that my employees can see the supposedly private control
> panel of this reseller - what else is needed to get this site down?

The certificate has been revoked for quite some time now and is not
available in your control panel.

You must be aware of this as you submitted a complaint to Paypal that
“the certificate purchased was revoked” (you did not get what you paid
for). One could think that the purpose was to get them to suspend our
account as similar complaints was made the day before.

I do not wish avoid engaging in a personal discussion. Instead I would
like to get back to how we can solve this and make sure it does
*never* happen again – for Certstar or other SSL providers.

Paul C. Bryan

unread,
Dec 23, 2008, 7:33:18 PM12/23/08
to
On Dec 23, 3:58 pm, patri...@certstar.com wrote:
> The technical verification procedure has been improved and is now on a
> very high security level. Comodo will also review our implementation
> to ensure that it comply with all standards and cannot be abused.

As far as I know, you're not the party who is responsible, nor the one
to mitigate this issue; that lies with Commodo, and ultimately
Mozilla. Comodo should never have delegated trust to anyone to a third
party to perform such validation. In so doing, they have undermined
trust that can be placed in them.

Christoph Moormann

unread,
Dec 23, 2008, 7:39:21 PM12/23/08
to
On Dec 24, 12:58 am, patri...@certstar.com wrote:

> I just wanted to give you all an update from Certstar. As you all know
> we failed to validate a certificate due to a flaw in our system which
> is clearly unacceptable.

IIIRC you failed to validate at least two certificates, coincidentally
the ones by the thread starter.

> Having worked intensively with this case I can truly say that Comodo
> is indeed taking their responsibility extremely seriously and taking
> huge efforts to ensure that safety of their root. Currently we are
> going through a number certificate to double ensure that no further
> mis-issuances have occurred.

I'm anxious for the results, and i would be suprised if this was an
exception.

>Comodo will also review our implementation
> to ensure that it comply with all standards and cannot be abused.

You call it abuse to check whether someone hands out trusted
unvalidated certificates?
Do you realise that half of your text tries to blame other people.
Go ahead, shoot the messenger.
*shaking his head in disbelief*


Ian G

unread,
Dec 23, 2008, 7:40:36 PM12/23/08
to mozilla's crypto code discussion list
On 23/12/08 22:25, Eddy Nigg wrote:
> On 12/23/2008 11:12 PM, Ian G:
>> Earlier, Frank used the language of "clear and present danger."
>>
>> * clear: we can measure the costs of it, and cost of defences.
>> * present: it is happening today, provably.
>> * danger: it can be shown capable of doing damage, at least in theory
>>
>> Only the last one is shown. It is a danger in that anyone relying on a
>> cert wrongly issued could be harmed. But it hasn't actually caused
>> anyone damages, and nobody has shown it to exist.
>
> The first and second are valid as well. We can measure the costs, can't we?


First point: Who lost money? What are the damages?

You don't count, or more precisely, the money you spent getting the cert
doesn't count; sorry about that :)


> The second item shows that it can be happening today, you don't have to
> prove it, because I did already.


Yes, it's an "exploit" not a "breach".

This is a breach, this proves certs are proved to use bad stuff:

http://blogs.technet.com/mmpc/archive/2008/11/06/malware-and-signed-code.aspx

Indeed, some stats to underscore *presence* ...

"In the first six months of 2008, the MMPC received reports of 22M
instances of distinct malware files, of which about 173,000 were
distinct malware files with code signatures. Of this malware with code
signatures, about 38,000 were not validly code signed, so approximately
*135,000 validly signed malware files* were reported to Microsoft.
Approximately 0.6% of detected malware were validly code signed."

But they don't go far not enough to show the first point. (You might
validly ask why Microsoft did not go that far, and deny us the clarity
in dealing with this situation! Another subject, another day!)


> The fact that I disclosed publicly
> about it doesn't mean that there aren't scores of others out there.


It doesn't mean there aren't and it doesn't mean there are. On the
basis of a reasonable judgement, given the info we have today, there
aren't. Also, by the same logic, there aren't any mice in your
computer, even though you looked yesterday.

Which isn't to say that it won't change tomorrow. Security work is done
with facts, because attackers will use your conjectures against you.

> Considering the aggressive and illegal mailing campaign they operated,
> one can expect that there are many certificates issued in this way.


Sure. All highly targeted to existing owners of SSL certs ... who
surprise, surprise, didn't notice that there was no validation :)

What damages are likely here, to the targeted users? Are there any
"non-targeted users" ?


>> This situation is about as harmful as the average exploit demo. In this
>> particular case it was caught by an insider rather than by an outsider.
>> However, beyond that, the situation is already sealed in that Comodo
>> have already taken the urgent triage steps to make sure nobody else gets
>> one.
>>
>
> Well, that's exactly the point! I haven't heard back from Robin anymore,

> their site is still online as well.Clear statements about how many


> certificates are affected (all of the ones issued through them) and
> revocations (or at least urgent re-validation within 48 hours and
> thereafter revocation) would be satisfactory maybe.

> The handling of the situation is insufficient, response and measures are
> a joke, communication doesn't exist.


Right. Over to Comodo. It's their job, let them do it. Any statements
you desire to them should tie into some criteria, practice or policy.


> Apparently this resellers business
> was too good for Comodo.


Well, concerns on the concept of resellers have been raised before.
This becomes a case in point, which should perhaps allow us to
re-address the issue.

But, CAs issue bad certs, that's a fact, so we can't damn them entirely.
Also, it's their job to deal with it. Adding a lynching party helps
no-one.

iang

Eddy Nigg

unread,
Dec 23, 2008, 7:41:53 PM12/23/08
to patr...@certstar.com
Hi Particia,

On 12/24/2008 01:58 AM, patr...@certstar.com:
> Dear all,
>

Upfront, as such for me, I accept your apology - sincerely. However your
apology is only for half the story, please read on...

>
> Having worked intensively with this case I can truly say that Comodo
> is indeed taking their responsibility extremely seriously and taking
> huge efforts to ensure that safety of their root. Currently we are
> going through a number certificate to double ensure that no further
> mis-issuances have occurred.

Look, the problem isn't your implementations alone, this failure is a
broader set of failures not only by your company, but that of Comodo.
Comodo has clearly failed in their duties, in their concepts, controls
and policies. The responsibility of domain validation shouldn't be with
you, but part of Comodo's procedures and data flow. As much as your
system perhaps failed - you aren't the only one to take blame and
responsibility.

Besides that, I want to repeat that there was no validation performed at
all. There was no such step, not even a hint of it.

>
> Personally I do feel that it would have been appropriate if we had
> been contacted by StartCom Ltd when they found this flow so that it
> could have been fixed faster. Being our competitor I am not sure if we
> could expect this, but it would indeed have been generous move.

I tell you something about generosity hereby. You have spammed, phished
and mislead our customers by sending those emails, with it everything
started. StartCom has contacted you (myself personally) and requested to
talk to the owners of your company and somebody with the name Mark
answered me. I replied with full disclosure about those emails you are
sending and with immediate request to shut down your activities.
Unfortunately *OUR* email was ignored and we never received a reply
thereafter.

My request was reasonable in light of the potential damage you have
caused us and other certification authorities and whoever makes a
mistake has to pay for it. Your site should have been taken down then.

Also Comodo replied laconically to my complaint. In that respect, and
the way Comodo handled this incident, nobody should be surprised that
who make foes, will not be treated friendly either. It would have been
generous from *YOU* to address the first problem appropriately and come
to terms with us when we requested it.

>
> The technical verification procedure has been improved and is now on a
> very high security level. Comodo will also review our implementation
> to ensure that it comply with all standards and cannot be abused.

As long as those validations are performed at a hosted site at some
hosting provider (yes, I made my researches in this respect) and not by
Comodo's infrastructure, there can't be any talk of "improvement",
period. This is simply not serious! I hope that everybody takes note on
this.

Eddy Nigg

unread,
Dec 23, 2008, 8:35:00 PM12/23/08
to
On 12/24/2008 02:40 AM, Ian G:

> You don't count, or more precisely, the money you spent getting the cert
> doesn't count; sorry about that :)

At least I got it refunded ;-)

>
> Well, concerns on the concept of resellers have been raised before. This
> becomes a case in point, which should perhaps allow us to re-address the
> issue.

Yes indeed. When the right time comes, this must be addressed appropriately.

>
> But, CAs issue bad certs, that's a fact, so we can't damn them entirely.
> Also, it's their job to deal with it. Adding a lynching party helps no-one.
>

Nope, and flaws can happen, no system is immune. However in this case
the handling was a sever failure from top to bottom, this isn't a bug or
isolated failure.

David E. Ross

unread,
Dec 23, 2008, 8:45:19 PM12/23/08
to

I've turned off all trust bits for all Comodo certs in my own personal,
home installation of SeaMonkey. That was step one. I might turn some
back on when I see the consequences. So far, however, I have not
encountered any secure site using Comodo.

The bigger issue is what will be done by the Mozilla organization. Now
that Mozill knows about the problem, I believe they have a liability to
their users if they fail to take action to mitigate the risks to those
users. This is analogous to laws in many U.S. states that hold a city
liable for automobile damage from a pothole AFTER the pothole is
reported to the city; there is no liability before knowledge but much
liability after knowledge (even if the city did not cause the pothole).

--
David E. Ross
<http://www.rossde.com/>

Go to Mozdev at <http://www.mozdev.org/> for quick access to
extensions for Firefox, Thunderbird, SeaMonkey, and other
Mozilla-related applications. You can access Mozdev much
more quickly than you can Mozilla Add-Ons.

ro...@comodo.com

unread,
Dec 23, 2008, 8:56:59 PM12/23/08
to
On Dec 23, 5:09 am, Frank Hecker <hec...@mozillafoundation.org> wrote:
> There are two general reasons for pulling a root, to address a clear and
> present danger to Mozilla users, and to punish a CA and deter others. My
> concern right now is with the former. I see at least three issues in
> relation to that:
>
> 1. Issuance of further non-validated certs by this reseller. Comodo
> seems to have addressed this by suspending the reseller's ability to get
> certs issued. (I can testify that this is the case, as I tried to
> duplicate Eddy's feat earlier today and got my uploaded CSR rejected.)
>
> 2. Potential problems with certs already sold through this reseller.
> Comodo should investigate this and take action if needed. (This need not
> necessarily require revoking all certificates associated with the
> reseller; for example, the existing certs and their associated domains
> could be re-validated, the registered domain owners could be notified of
> the potential for bogus certs floating around, etc.)
>
Frank,
We are in the process of reviewing all of the certificates that have
been issued where Certstar served as a RA for Comodo. Fortunately they
have only been involved in the validation of 111 certificates. As I
previously mentioned as soon as we discovered the error with the
Mozilla.com certificate we suspended Certicom’s RA privileges.
Certstar’s RA activities remain suspended.

At this point it appears that the Mozilla and Startcom certificates
we