Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Unbelievable!

1,241 views
Skip to first unread message

Eddy Nigg

unread,
Dec 22, 2008, 5:25:23 PM12/22/08
to
https://blog.startcom.org/?p=145

--
Regards

Signer: Eddy Nigg, StartCom Ltd.
Jabber: star...@startcom.org
Blog: https://blog.startcom.org

Robin Alden

unread,
Dec 22, 2008, 8:15:35 PM12/22/08
to mozilla's crypto code discussion list
Eddy,
That reseller's ability to sell Comodo certificates has been
suspended while we investigate why they are apparently not fulfilling their
contractual obligations to us.
We revoked your certificate for mozilla.com.

Regards
Robin Alden
Comodo

> _______________________________________________
> dev-tech-crypto mailing list
> dev-tec...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tech-crypto

Eddy Nigg

unread,
Dec 22, 2008, 8:33:46 PM12/22/08
to
On 12/23/2008 03:15 AM, Robin Alden:

> Eddy,
> That reseller's ability to sell Comodo certificates has been
> suspended while we investigate why they are apparently not fulfilling their
> contractual obligations to us.

How can you outsource such a critical part as domain control validation
to a reseller is a complete mystery to me! Your controls (if any) have
completly failed.

And apparently if the fish stinks at the tail, chances that the head is
rotten too are pretty high.

> We revoked your certificate for mozilla.com.
>

I suggest to revoke ALL certificates from this reseller and perform an
urgent review of your policies and implementations in relation to
resellers at all. Other steps might be needed as well, you know better
than me.

Robin Alden

unread,
Dec 22, 2008, 8:38:14 PM12/22/08
to mozilla's crypto code discussion list
Eddy,

As I noted in my prior correspondence, Comodo has undertaken an internal
review of the Certstar reseller account. We have informed CertStar that
their email violates their contractual obligation to refrain from sending
unsolicited emails and that their email could be interpreted as misleading
and confusing to the customer. During our review, we discovered that
Certstar had apparently issued a certificate to mozilla.com without
validating control of the domain. We immediately revoked the certificate
(prior to your posting) and have suspended Certstar's reseller activities
until our investigation has been completed. Please let me know if you have
any further problems.

Regards
Robin Alden
Comodo

> -----Original Message-----
> From: dev-tech-crypto-bounces+robin=comod...@lists.mozilla.org
> [mailto:dev-tech-crypto-bounces+robin=comod...@lists.mozilla.org] On
> Behalf Of Eddy Nigg

Eddy Nigg

unread,
Dec 22, 2008, 8:47:16 PM12/22/08
to
On 12/23/2008 03:38 AM, Robin Alden:

> Eddy,
>
> As I noted in my prior correspondence, Comodo has undertaken an internal
> review of the Certstar reseller account. We have informed CertStar that
> their email violates their contractual obligation to refrain from sending
> unsolicited emails and that their email could be interpreted as misleading
> and confusing to the customer. During our review, we discovered that
> Certstar had apparently issued a certificate to mozilla.com without
> validating control of the domain. We immediately revoked the certificate
> (prior to your posting) and have suspended Certstar's reseller activities
> until our investigation has been completed.

A.) The certificate was revoked after posting to this list.
B.) Your CA also issued a certificate to startcom.org without validating
anything.
C.) Yes, I think we have a problem of a wider scale here. This is not
about me personally.

Kyle Hamilton

unread,
Dec 22, 2008, 8:53:07 PM12/22/08
to mozilla's crypto code discussion list
I believe that this is a symptom of faulty internal control, and that
this is direct evidence of not living up to the Mozilla CA policy.

I advocate at least temporarily removing the trust bits from Comodo
until a new external audit can be completed, with an eye toward
ensuring that Comodo, not the reseller, perform the domain
validations.

-Kyle H

Frank Hecker

unread,
Dec 23, 2008, 12:09:50 AM12/23/08
to
Kyle Hamilton wrote:
> I advocate at least temporarily removing the trust bits from Comodo
> until a new external audit can be completed, with an eye toward
> ensuring that Comodo, not the reseller, perform the domain
> validations.

There are two general reasons for pulling a root, to address a clear and
present danger to Mozilla users, and to punish a CA and deter others. My
concern right now is with the former. I see at least three issues in
relation to that:

1. Issuance of further non-validated certs by this reseller. Comodo
seems to have addressed this by suspending the reseller's ability to get
certs issued. (I can testify that this is the case, as I tried to
duplicate Eddy's feat earlier today and got my uploaded CSR rejected.)

2. Potential problems with certs already sold through this reseller.
Comodo should investigate this and take action if needed. (This need not
necessarily require revoking all certificates associated with the
reseller; for example, the existing certs and their associated domains
could be re-validated, the registered domain owners could be notified of
the potential for bogus certs floating around, etc.)

3. Potential problems with other Comodo resellers. I'm not going to tell
Comodo how to operate its reseller network, but they certainly should
take a look at whether and where this might be a problem with other
resellers, and how they could revamp their systems to reduce potential
problems with resellers.

Pulling a Comodo root will knock out Firefox, etc., access to thousands
of SSL sites, maybe tens of thousands. Given the disruption that would
cause, the final decision on this IMO should be made in conjunction with
the Firefox security folks. From my point of view I'd wait on more
information regarding items 2 and 3 above before making a recommendation.

Frank

--
Frank Hecker
hec...@mozillafoundation.org

Kyle Hamilton

unread,
Dec 23, 2008, 2:09:35 AM12/23/08
to mozilla's crypto code discussion list, mozilla-...@mozilla.org
I agree with all three of these issues -- it's not *just* this
reseller, it's potentially the entire reseller network, and it's
entirely possible that the problem exists across multiples.
(Especially if Comodo delegates full Registration Authority capability
without verification, which seems to be the case -- though they could
have simply issued a sub-CA certificate.)

It occurs to me that there is no facility in Firefox or other Mozilla
products to provide an explanatory dialog that there's an issue, and
such a facility would be extremely useful at this point. Being able
to print a message to the user like "The Mozilla Foundation has
identified issues with the trusted root that issued this certificate
which prevent Firefox from being able to guarantee that this is truly
the site to which you intended to go. While it is unlikely that this
is a widespread problem, and an attack would rely on more technical
intrusions into the network, the nature of these issues requires that
you be warned of this circumstance so that you can exercise
appropriate levels of caution. The Mozilla Foundation is working with
the trusted root to resolve these issues." would help a lot.

(I word it like that because in order for an attacker to succeed he
would need to also hijack DNS, or place a entry in the user's hosts
file.)

Of course, this would be an NSS change (the addition of a 'trust
suspended' bit, in addition to the trust flags), a Firefox/Thunderbird
code change (to look for that bit), and a chrome change (to explain
what's going on). Or even a check for the name in the hosts file to
see if it's overridden from DNS. Placing an entry in a user's hosts
file is easier than hijacking DNS, or at least it's supposed to be.
I'm pretty sure we're all aware of the fairly recent cache poisoning
attack, but the people who write BIND pushed out a change to protect
against it very rapidly.

The addition of a 'trust suspended' bit is primarily unlikely to
happen, though, because there's too much inertia in the way things are
to do what needs to be done to fix the authentication system to allow
a root to stay in while the user is potentially at risk.

-Kyle H

patr...@certstar.com

unread,
Dec 23, 2008, 3:48:12 AM12/23/08
to
Hi all,

A glitch in our validation system has today caused a certificate to be
issued to a person who successfully abused our system.

We have now strengthened our domain validation system so that such
abuse cannot happen again. Comodo has handled this issue in a
professional way by invoking the certificate immediately after issuing
and contacting Certstar.

Again, I cannot stress enough how seriously we take this issue and I
would like to apologize to the Mozilla organization for the mis-issue.

--
kind regards,
Patricia, Certstar ApS

Thorsten Becker

unread,
Dec 23, 2008, 3:54:07 AM12/23/08
to
Hi Patricia,

patr...@certstar.com schrieb:

> We have now strengthened our domain validation system so that such
> abuse cannot happen again.

just curious: How do you normally validate domain ownership?

TIA,

Thorsten

Eddy Nigg

unread,
Dec 23, 2008, 4:42:49 AM12/23/08
to
On 12/23/2008 10:48 AM, patr...@certstar.com:

> Hi all,
>
> A glitch in our validation system has today caused a certificate to be
> issued to a person who successfully abused our system.

It's not me who abused your system, it's your company which sent out
illegal, misleading emails to our customers. Since I couldn't find out
who stands behind your site I had to get a certificate. There was no
abuse, there was only use-and-get.

- No affiliation published, the fact that you are a reseller for Comodo
is nowhere noted.
- No subscriber agreement or notice presented to subscribers.
- No control validation of the domain name. Just pay-and-go.

I didn't had to do anything to overcome any validation whatsoever. There
was no abuse!

> We have now strengthened our domain validation system so that such
> abuse cannot happen again. Comodo has handled this issue in a
> professional way by invoking the certificate immediately after issuing
> and contacting Certstar.

Yes, after I posted my article here at the mailing list.

> Again, I cannot stress enough how seriously we take this issue and I
> would like to apologize to the Mozilla organization for the mis-issue.
>


--

Eddy Nigg

unread,
Dec 23, 2008, 5:03:18 AM12/23/08
to
For those interested, Frank opened a bug to investigate this incident:

https://bugzilla.mozilla.org/show_bug.cgi?id=470897

Eddy Nigg

unread,
Dec 23, 2008, 5:39:03 AM12/23/08
to
On 12/23/2008 07:09 AM, Frank Hecker:

> There are two general reasons for pulling a root, to address a clear and
> present danger to Mozilla users, and to punish a CA and deter others. My
> concern right now is with the former. I see at least three issues in
> relation to that:
>
> 1. Issuance of further non-validated certs by this reseller. Comodo
> seems to have addressed this by suspending the reseller's ability to get
> certs issued. (I can testify that this is the case, as I tried to
> duplicate Eddy's feat earlier today and got my uploaded CSR rejected.)

As long as this site keeps operating, our customers are still being let
to believe that they have to renew their certificate with them. This is
only a reminder about how it started at all. CAs and their customers are
still taking damage from the previously sent messages.

>
> 2. Potential problems with certs already sold through this reseller.
> Comodo should investigate this and take action if needed. (This need not
> necessarily require revoking all certificates associated with the
> reseller; for example, the existing certs and their associated domains
> could be re-validated, the registered domain owners could be notified of
> the potential for bogus certs floating around, etc.)

You shouldn't notify the subscribers or domain name owners, but the
relying parties. How to do that is up to you and Comodo I guess.

Comodo not only shouldn't just investigate and take action, Comodo needs
to report publicly about their findings and full report about the
actions taken. This isn't a suggestion of resolution about this
incident, it's the transparency I expect from them at this hour.

> Pulling a Comodo root will knock out Firefox, etc., access to thousands
> of SSL sites, maybe tens of thousands.

I'm not advocating removing their root, however we must assess the risk
which is potentially caused to the relying parties. There may be
thousands of sites which received certificates without validating them.

> Given the disruption that would
> cause, the final decision on this IMO should be made in conjunction with
> the Firefox security folks.

Disabling the trust bits of "AddTrust External CA Root" could be a
temporary measure to prevent damage to relying parties until Mozilla
receives full report and disclosure from Comodo about its resellers and
conclusion of their investigation.

Additionally instead of just yanking a root as a deterrent and
punishment, as you mentioned above, I'd prefer to receive a commitment
from Comodo to address other issues noted during the last discussion -
mainly those listed in the "Problematic Practices" document. Considering
that OCSP in Firefox is set to soft-fail, an issue with their OCSP
server could still circumvent bogus sites for potentially the next five
years. A full review of their current status in NSS and their practices
might be advocated too.

Eddy Nigg

unread,
Dec 23, 2008, 6:36:47 AM12/23/08
to
On 12/23/2008 09:09 AM, Kyle Hamilton:

> (I word it like that because in order for an attacker to succeed he
> would need to also hijack DNS, or place a entry in the user's hosts
> file.)

Or be a WiFi operator. This was the attack vector of
https://bugzilla.mozilla.org/show_bug.cgi?id=460374

>
> Of course, this would be an NSS change (the addition of a 'trust
> suspended' bit,

I think this to be an interesting idea and should be considered.

Frank Hecker

unread,
Dec 23, 2008, 8:05:35 AM12/23/08
to
Eddy Nigg wrote:
> For those interested, Frank opened a bug to investigate this incident:
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=470897

Actually Nelson opened this bug.

Kyle Hamilton

unread,
Dec 23, 2008, 8:09:42 AM12/23/08
to mozilla's crypto code discussion list
Patricia, I believe it's important to realize a couple of things:

1) An unsolicited commercial email (UCE) message was sent from your
company to the party in question suggesting that there already existed
a relationship between your company and the party in question. This
is obvious from the verb 'renew' in the original message -- a
non-party to the original certificate can't renew "on behalf of" the
original certificate issuer. If they can, there's a major problem.

1a) The message from your company, and in fact the entire process up
to and including paying for the certificate which your company issued,
did not expressly claim an affiliation, and the party in question went
through the process of "renewal" with the intent of figuring out which
CA's services were being advertised via UCE.

2) The party in question (Eddy Nigg) is the founder of another
Certifying Authority, with a root which participates in the Mozilla
root program and which is also included in the root store.

3) It falls within the concept of "due diligence" for a
security-conscious warden of a public trust to recognize the lack of
actual domain control verification in a trusted certificate issuance
process, and *for the purposes of verification and public
notification* obtain a certificate which could be shown absolutely to
have been issued in violation of the standards of at least one of the
root programs that the issuer (or issuer's trust-delegator)
participates in.

3a) If this had not been identified, someone else would have found it
eventually -- and the reaction to a gross violation of security
standards is to immediately believe that the worst-case scenario has
occurred: that it had already been found and exploited by at least one
other person.

4) The end-users (referred to as "relying parties") are well-served by
this identification of completely bogus credentialing.

5) It's also important to recognize the following (the first comment
on https://bugzilla.mozilla.org/show_bug.cgi?id=470897):

------- Comment #1 From Reed Loden [:reed] 2008-12-23 01:35:37
PST -------

The same company that Eddy was able to get the mozilla.com cert from (Certstar)
has been endlessly spamming webm...@mozilla.org since the beginning of
December complaining that one of our SSL certs had "expired" and needed to be
"renewed" (both of which were false). They have continued to spam us almost
daily. :(

------ end comment ------

Yes, the bare facts are that a user exploited the lack of verification
in your certificate issuance process. However, the lack of
verification in your process violates at least one contractual
obligation that your company is required to uphold, and reduces the
value of both Comodo's root AND the commercial Certifying Authority
structure in general. Because the user founded another commercial CA
which is trusted by Mozilla NSS by-default, this user was operating in
a manner to verify that the commercial CA structure was in fact
secure; the trust afforded this user's CA is demonstrably harmed by
your lack of verification.

I do not take this situation as lightly as you appear to wish that
everyone would. The initial fault is YOUR COMPANY'S (and that fault
reverbrates up to Comodo, since it delegated trust to your company),
and the fact that you're attempting to shift the blame onto the user
shows that you are absolutely untrustworthy to run or be the public
face of any commercial certificate-issuance service.

Because of this, my recommendation that Comodo's trust bits be removed
until a full audit of their practices (and a full audit of all issued
certificates) stands, and I am that much more resolute in my belief.
It may be that the integrity of Comodo's root has been irreparably
damaged by your company's malfeasance; if this is the case, I
certainly hope they rake you over the coals.

-Kyle H

Frank Hecker

unread,
Dec 23, 2008, 8:14:42 AM12/23/08
to
Eddy Nigg wrote:
> Disabling the trust bits of "AddTrust External CA Root" could be a
> temporary measure to prevent damage to relying parties until Mozilla
> receives full report and disclosure from Comodo about its resellers and
> conclusion of their investigation.

Do you mean the UTN-UserFirst-Hardware root? According to the screenshot
on your blog post, that's the root the bogus cert chains up to. Also, if
we were to take action of this general sort (as a hypothetical), what
about adding the PositiveSSL CA cert to NSS with the SSL trust bit
disabled; wouldn't that accomplish the same purpose, without interfering
with other parts of the hierarchy under the UTN-UserFirst-Hardware root?
(I seem to recall we've discussed this sort of thing in the past.)

Also note that any "suspension" of a root would last at last 1-3 months,
since that the typical interval between security updates for Firefox and
other Mozilla-based products.

Eddy Nigg

unread,
Dec 23, 2008, 8:15:56 AM12/23/08
to
On 12/23/2008 03:05 PM, Frank Hecker:

> Eddy Nigg wrote:
>> For those interested, Frank opened a bug to investigate this incident:
>>
>> https://bugzilla.mozilla.org/show_bug.cgi?id=470897
>
> Actually Nelson opened this bug.
>

Thanks for that. More into this story...

...all our employees coming the our network are able to retrieve the
certificates and all the rest from the "control panel" of this site.
Again, no questions asked. Apparently the only things one needs is to
fake our gateway IP address to get there.

It's getting weirder with the second. If they don't shut that site, we
can perhaps just publish the private key for the mozilla.com certificate
as well so everybody can enjoy it.

Kyle Hamilton

unread,
Dec 23, 2008, 8:17:14 AM12/23/08
to mozilla's crypto code discussion list
Either way, it's enough of a problem that a bug needs to be opened,
and it needs to be followed upon sooner than later.

(My opinion, of course. Comodo's gotta be breathing a sigh of relief
that I don't work for the Mozilla Foundation, else I would have yanked
the trust bits first and figured it out later.)

This incident shows anew the lack of policy/procedure of what to do in
the case of a gross breach of the root agreement, though. The entire
point of certificates in the first place is "user security", not "user
convenience". (If this weren't the case, none of us would have any
desire to get certificates in the first place.) Your reaction seems
to be based on placing "user convenience" over "user security", and
(again, my opinion) I don't believe that this is appropriate at all.

-Kyle H

Ian G

unread,
Dec 23, 2008, 8:58:45 AM12/23/08
to mozilla's crypto code discussion list
In the past, lots of good stuff has been done that handles the ascension
to the root list of Mozilla. c.f. the policy. But not so much is
written about *what happens afterwards*. This recent thread has been
such a case, and has afforded an opportunity to make some notes on what
might be suitable as a practices page on the wiki.

The reason for doing this, and adding yet more boring verbage to the
mass of words, is that it can prevent uneccessary legal costs by
providing clear guidepaths, and clear signals of seriousness.

Here's what I see:

1. How to file a dispute against Mozilla and/or a CA. This seems
fairly easy; file a bug in bugzilla and mark it as already described
here: https://wiki.mozilla.org/CA:How_to_apply . (With mods: Severity:
as appropriate.)

2. What is the practice on revocation or un-trusting on roots? This is
is perhaps a headline case of a dispute, so possible merits its own
notes. Frank has suggested the test:

a. there is a clear and present danger to Mozilla users, or
b. to punish a CA, and/or to deter others.

I would suggest that (b) be modifed slightly to give it a basis, which
in this case might be "for breaches of policy or practices". The
Policy, pt 4, gives the authority, as well as some examples, and adds
another test case:

c. where certificates cause technical problems with mozo software.

3. How to resolve a dispute. This is a Mozilla action &
responsibility. Reverse-engineering and referring, I would suggest this
as a teaser:

a. The CA certificate "module owner" at Mozilla foundation is
responsible. Ref, the policy, pt 15.
b. The dispute is investigated and ruled on by module owner.
c. The ruling is listed in the bug report above.
d. Many disputes will be dealt with by communication, and no ruling
will be required. This will create a default "closed, no action" ruling.

4. Finality. What happens if we disagree with the decision of the
module owner? In the policy, it says "CAs or others objecting to a
particular decision may appeal to mozilla.org staff, who will make a
final decision." Ref, policy, pt 15.

I would wonder about this; google suggests that "staff" is as listed here:
http://www.mozilla.org/about/staff
but that seems out of date. Also, due to the absence of this forum in
the public eye, I doubt it musters the credibility we need in dispute
review where the legal and contractual significance is high. E.g., is
there any way we can review the decisions they made in the past?

There are several possibilities:

(i) Ruling is final.
(ii) Mozilla.org staff, policy, pt 15.
(iii) Review by board of Mozilla Foundation.
(iv) Review by some independent party.
(v) Review by forum at law: courts, or Arbitrator.

Personally, I would plumb for (iii) and suggest the Mozo Foundation
board as the next step. It is expensive, but available. The directors
already have fiduciary responsibility, and can thus deal with the
significance. It is also aligned with the review of the manager
concerned, the policy and the general contractual issues.

End! I'd encourage others to comment!

If nobody has any objections, I can add that as a page into the wiki as
an informal document of practices, including or without those questions.


iang

Frank Hecker

unread,
Dec 23, 2008, 9:35:00 AM12/23/08
to
Kyle Hamilton wrote:
> Your reaction seems
> to be based on placing "user convenience" over "user security", and
> (again, my opinion) I don't believe that this is appropriate at all.

My intent is to balance the disruption that would be caused by pulling a
root vs. the actual security threat to users. Right now we have no real
idea as to the extent of the problem (e.g., how many certs might have
been issued without proper validation, how many of those were issued to
malicious actors, etc.).

Frank Hecker

unread,
Dec 23, 2008, 9:48:18 AM12/23/08
to
Ian G wrote:
> In the past, lots of good stuff has been done that handles the ascension
> to the root list of Mozilla. c.f. the policy. But not so much is
> written about *what happens afterwards*.

I don't have time right now to address your whole post, but I did want
to comment on one point.

> 4. Finality. What happens if we disagree with the decision of the
> module owner? In the policy, it says "CAs or others objecting to a
> particular decision may appeal to mozilla.org staff, who will make a
> final decision." Ref, policy, pt 15.
>
> I would wonder about this; google suggests that "staff" is as listed here:
> http://www.mozilla.org/about/staff
> but that seems out of date.

There is no longer a "mozilla.org staff" as traditionally constituted.
From a Mozilla community point of view the closest present-day
equivalent is the system described at

https://wiki.mozilla.org/Module_Owners_Activities_Modules

Basically there is a group of people that oversee the Mozilla module
ownership system and the module owners within it. That group in turn
operates under general policy structures set by Mitchell Baker and
Brendan Eich, and as you note the ultimate legal authority resides in
the board of the Mozilla Foundation (of which Mitchell and Brendan are
members).

For more background on this see

http://www.mozilla.org/hacking/module-ownership
http://blog.lizardwrangler.com/2008/05/30/governance-and-module-ownership/

Gervase Markham

unread,
Dec 23, 2008, 10:42:39 AM12/23/08
to Nelson B Bolyard
Frank Hecker wrote:
> Do you mean the UTN-UserFirst-Hardware root? According to the screenshot
> on your blog post, that's the root the bogus cert chains up to. Also, if
> we were to take action of this general sort (as a hypothetical), what
> about adding the PositiveSSL CA cert to NSS with the SSL trust bit
> disabled; wouldn't that accomplish the same purpose, without interfering
> with other parts of the hierarchy under the UTN-UserFirst-Hardware root?
> (I seem to recall we've discussed this sort of thing in the past.)

This was the first thing that occurred to me as well. Can anyone
(Nelson?) tell us definitively if this is possible? I would assume it
is, but it would be good to get confirmation.

Gerv

Gen Kanai

unread,
Dec 23, 2008, 1:17:18 PM12/23/08
to mozilla's crypto code discussion list
Are we going to receive information from Comodo regarding how many
other Comodo resellers may be in a similar position to Certstar?

Are we going to receive information from Certstar as to how many other
certs may have been issued in error?

How do we verify the claims from Comodo or Certstar?

Gen

Kyle Hamilton

unread,
Dec 23, 2008, 1:30:18 PM12/23/08
to mozilla's crypto code discussion list
These are, of course, the reason why I want the trust bits removed.
It's going to take more than a month to get this all straightened out,
and that's "over a month" that we have no idea if Comodo certificates
can be trusted.

-Kyle H

Frank Hecker

unread,
Dec 23, 2008, 1:43:30 PM12/23/08
to
Gen Kanai wrote:
> Are we going to receive information from Comodo regarding how many other
> Comodo resellers may be in a similar position to Certstar?
>
> Are we going to receive information from Certstar as to how many other
> certs may have been issued in error?

I've asked Robin Alden of Comodo to make an accounting regarding these
two issues. I don't expect to see that immediately (i.e., in the next
day or two), though I also don't expect to wait a month for it (as Kyle
is concerned about).

> How do we verify the claims from Comodo or Certstar?

Some things by their nature we can directly verify, and some we can't.
For example, we can certainly verify how Certstar is operating currently
(as Eddy and I did), and if certs get revoked we can verify that by
downloading the CRLs that get issued.

Hendrik Weimer

unread,
Dec 23, 2008, 2:15:25 PM12/23/08
to mozilla's crypto code discussion list
Frank Hecker <hec...@mozillafoundation.org> writes:

> My intent is to balance the disruption that would be caused by pulling
> a root vs. the actual security threat to users. Right now we have no
> real idea as to the extent of the problem (e.g., how many certs might
> have been issued without proper validation, how many of those were
> issued to malicious actors, etc.).

Isn't that, by itself, a very good reason to take immediate action?
Security should be default-fail rather than default-pass.

Hendrik

Kyle Hamilton

unread,
Dec 23, 2008, 2:23:16 PM12/23/08
to mozilla's crypto code discussion list
On Tue, Dec 23, 2008 at 10:43 AM, Frank Hecker
<hec...@mozillafoundation.org> wrote:
> I've asked Robin Alden of Comodo to make an accounting regarding these two
> issues. I don't expect to see that immediately (i.e., in the next day or
> two), though I also don't expect to wait a month for it (as Kyle is
> concerned about).

Actually, I think it's very important that the accounting include this:

for each name (not just certificate, but name in
subjectAlternativeNames) that has been certified, a connection to the
TLS ports should be made, and the certificate presented by the site
compared against the certificate that Comodo issued. This obviously
won't be a complete verification, but it should give a start to see
how widespread the problem is.

A script to do this could probably be written fairly easily, but
depending on the number of certificates Comodo has issued that are
currently valid (and I'd like to see some hard numbers on that, as
well) it could take a while to run.

From the script, the numbers I'd like to see are: the number of
unreachable/not-answering names/hosts, the number of matching
certificates, and the number of mismatched certificates. From that
output plus Comodo's records, I would also like to see how many
resellers there are and how many of them have sold mismatched
certificates.

I hate to say this, but this IS The Worst-Case Scenario. A CA has
gone rogue and issued certificates that violate its standards, and the
standards of the root programs that it's a part of -- it is true that
Comodo didn't /intend/ to go rogue, but it has, and we can't afford to
let it damage the greater PKI. Since every CA in the root store is
treated the same, there is no differentiation between them -- and this
means that Verisign and Comodo and Thawte and *every* CA share the
same reputation. If one goes rogue, it's exactly the same as if all
of them have gone rogue, in the eye of the end-user.

To put it another way, it's exactly the same problem as putting the
public keys of webservers in DNSSEC. Since the end-system can't know
if a response came from DNSSEC or just unauthenticated DNS, the
end-system can't trust anything (including RFC2538-style CERT records)
that comes from DNS.

THIS is why I want to see greater differentiation in the browser
chrome between CAs, so that one bad apple doesn't spoil the whole root
barrel. THIS is why the argument against changing the chrome (user
convenience) fails.

-Kyle H

Kyle Hamilton

unread,
Dec 23, 2008, 2:27:53 PM12/23/08
to mozilla's crypto code discussion list
I'd rather deal with disruption caused thereby (and, yes, the user
complaints generated thereby -- at least then the end-user would KNOW
that there's a problem that's being dealt with rather than having a
FALSE SENSE OF SECURITY) than let those potential security breaches
continue to wreak their quiet little havoc.

-Kyle H

On Tue, Dec 23, 2008 at 11:15 AM, Hendrik Weimer <hen...@enyo.de> wrote:
> Frank Hecker <hec...@mozillafoundation.org> writes:
>
>> My intent is to balance the disruption that would be caused by pulling
>> a root vs. the actual security threat to users. Right now we have no
>> real idea as to the extent of the problem (e.g., how many certs might
>> have been issued without proper validation, how many of those were
>> issued to malicious actors, etc.).
>

> Isn't that, by itself, a very good reason to take immediate action?
> Security should be default-fail rather than default-pass.
>
> Hendrik

Eddy Nigg

unread,
Dec 23, 2008, 2:49:20 PM12/23/08
to
On 12/23/2008 09:15 PM, Hendrik Weimer:

It should be. I realized that there are more points which will have to
be addressed in due time at Mozilla, which however can wait for now.

Concerning the disruption, Comodo has many roots and the resetting of
this specific root would affect low-assurance sites as far as I know.
The higher validated sites would not be affected. Having said that, the
roots of their low-assurance products have been previously of concern it
must be looked at in the broader context of Comodo's operations. It's
not surprising in itself.

Justin Dolske

unread,
Dec 23, 2008, 3:12:27 PM12/23/08
to
On 12/23/08 11:27 AM, Kyle Hamilton wrote:
> I'd rather deal with disruption caused thereby (and, yes, the user
> complaints generated thereby -- at least then the end-user would KNOW
> that there's a problem that's being dealt with rather than having a
> FALSE SENSE OF SECURITY)

Hmm, would they?

Justin Dolske

unread,
Dec 23, 2008, 3:20:15 PM12/23/08
to

*sigh* I somehow managed to click "Send" mid-message.

Anyway, the rest of my incomplete thought:

...I think there's some risk that if a Firefox update suddenly breaks a
large swath of legitimate SSL sites, that could end up training users to
ignore the problem. I'm not even sure how you'd present this condition
to Joe The User in a comprehensible way.

That said, the Comodo/Certstar is hugely sucky and I would hope there's
something we can do about it that helps users.

Justin

Daniel Veditz

unread,
Dec 23, 2008, 3:23:27 PM12/23/08
to
Frank Hecker wrote:
> Eddy Nigg wrote:
>> Disabling the trust bits of "AddTrust External CA Root" could be a
>> temporary measure to prevent damage to relying parties
>
> Also note that any "suspension" of a root would last at last 1-3 months,
> since that the typical interval between security updates for Firefox and
> other Mozilla-based products.

And we don't have a magic switch we can flip in the office. We'd have to
make the change, test the change, make the builds, ship the builds,
users would have to update (about a week from ship until most users have
the update).

If the sole purpose of the update was to break lots of sites (from the
user's POV) then some number of them disable updates, making them less
secure in the future.

If Comodo is acting in good faith then anything they can do would be
lightyears faster than a client update. If they're not fulfilling their
responsibilities then a permanent removal would make sense, but given
the time scales it's hard to see how a "temporary" month-or-so removal
helps.

Maybe we need to build in something like a CRL that pings back to
Mozilla that would let us revoke roots without having to ship a client
update.

Eddy Nigg

unread,
Dec 23, 2008, 3:43:17 PM12/23/08
to
On 12/23/2008 10:23 PM, Daniel Veditz:

>
> Maybe we need to build in something like a CRL that pings back to
> Mozilla that would let us revoke roots without having to ship a client
> update.

Of course we (@ mozilla) also take our lessons from this event, I'm
sure. Indeed it was previously suggested here and I think we'll have to
look at such options in due time.

Anything other than taking down their site is insufficient (as an
immediate action to prevent further damage), specially in light of
apparently more flaws in their system. Not having done that as I
requested already two days ago isn't exactly comforting. Doesn't show
really cooperation and good faith here...

...they'd do that better now before any more damage happens. I've warned
them and requested to shut them down. My warning was ignored, their bad...

dou...@theros.info

unread,
Dec 23, 2008, 3:44:13 PM12/23/08
to

I, for example, have a ssl cert from comodo reseller, and they DO have
made all the validation steps.

My site, a legitimate one, would be in trouble with this. Are you all
sure that it is a good measure to just knock off the root cert or
security bit?

please, think twice

Christoph Moormann

unread,
Dec 23, 2008, 4:08:08 PM12/23/08
to

I puzzled by the security approach on this issue. There was a severe
security problem, and at least two, maybe many certs are compromised,
who knows by this point.
Do i still have trust in the Certs of Comodo or their signing
resellers? No, until the whole issue is cleared up completely.
Now it's the users duty to check every Cert on the issuer to evaluate
the trustlevel, while his browser states, everything is fine?
This is not about punishing Comodo, their resellers or their
customers, this is about trusting my browser software.

Ian G

unread,
Dec 23, 2008, 4:12:21 PM12/23/08
to mozilla's crypto code discussion list
On 23/12/08 20:23, Kyle Hamilton wrote:
> On Tue, Dec 23, 2008 at 10:43 AM, Frank Hecker
> <hec...@mozillafoundation.org> wrote:
>> I've asked Robin Alden of Comodo to make an accounting regarding these two
>> issues. I don't expect to see that immediately (i.e., in the next day or
>> two), though I also don't expect to wait a month for it (as Kyle is
>> concerned about).
>
> Actually, I think it's very important that the accounting include this:
>
> for each name (not just certificate, but name in
> subjectAlternativeNames) that has been certified, a connection to the
> TLS ports should be made, and the certificate presented by the site
> compared against the certificate that Comodo issued. This obviously
> won't be a complete verification, but it should give a start to see
> how widespread the problem is.


Um. Some questions... On what basis are we asking for this "accounting" ?

Is there a criteria that calls for public breach accounting? Perhaps
Mozilla needs to add it? Most of the criteria call for revocation when
a false cert has been issued, and that seems to have been met right now.

Also, some of the process might instead rest with Comodo's external
auditor or its internal audit process or its internal security process.
Perhaps in the next audit, there will be an examination of the
remedial steps taken, to see if they conform to the security manual, etc.

There are some slippery slope aspects here. If the framework for such
"accounting" is outside Mozilla's area, then care might be needed.
There could be confusion in role and liabilities.

It's nice if Mozilla were minded to do all the audit and security
activities for all, but I'm not sure Frank has time ;) OTOH, if we feel
the need to usurp the CA's processes and the conventional audit
relationship with an "accounting" then perhaps we need to also simplify
some other processes?

Another aspect here is that the goal of any security or governance
process is not to eliminate or stop all breaches. The process is more
about how you keep dangers to a minimum, deal with the breaches that do
happen, and improve.


> I hate to say this, but this IS The Worst-Case Scenario.

Noooo.... With respect, I'd say worst case is that a root has been
breached, and the Carders have got hold of it and are MITMing the major
retail sites. Damages!

Earlier, Frank used the language of "clear and present danger."

* clear: we can measure the costs of it, and cost of defences.
* present: it is happening today, provably.
* danger: it can be shown capable of doing damage, at least in theory

Only the last one is shown. It is a danger in that anyone relying on a
cert wrongly issued could be harmed. But it hasn't actually caused
anyone damages, and nobody has shown it to exist.

This situation is about as harmful as the average exploit demo. In this
particular case it was caught by an insider rather than by an outsider.
However, beyond that, the situation is already sealed in that Comodo
have already taken the urgent triage steps to make sure nobody else gets
one.


> A CA has
> gone rogue and issued certificates that violate its standards, and the
> standards of the root programs that it's a part of -- it is true that
> Comodo didn't /intend/ to go rogue, but it has, and we can't afford to
> let it damage the greater PKI.

This is no different to when Verisign got schnikered into issuing a
couple of Microsoft certs. No damages ever resulted. The system worked
as it was expected to, it seems. Panic is not called for.


> Since every CA in the root store is
> treated the same, there is no differentiation between them -- and this
> means that Verisign and Comodo and Thawte and *every* CA share the
> same reputation. If one goes rogue, it's exactly the same as if all
> of them have gone rogue, in the eye of the end-user.

...

> THIS is why I want to see greater differentiation in the browser
> chrome between CAs, so that one bad apple doesn't spoil the whole root
> barrel. THIS is why the argument against changing the chrome (user
> convenience) fails.


Oh, on that, yes, strongly agree. There's no trust without brand :)

iang

Justin Dolske

unread,
Dec 23, 2008, 4:16:05 PM12/23/08
to
On 12/23/08 12:20 PM, Justin Dolske wrote:

> That said, the Comodo/Certstar is hugely sucky and I would hope there's
> something we can do about it that helps users.

I am just full of fail today: "... the Comodo/Comstar *incident* is
hugely sucky ..."

Justin

Eddy Nigg

unread,
Dec 23, 2008, 4:25:09 PM12/23/08
to
On 12/23/2008 11:12 PM, Ian G:

> Earlier, Frank used the language of "clear and present danger."
>
> * clear: we can measure the costs of it, and cost of defences.
> * present: it is happening today, provably.
> * danger: it can be shown capable of doing damage, at least in theory
>
> Only the last one is shown. It is a danger in that anyone relying on a
> cert wrongly issued could be harmed. But it hasn't actually caused
> anyone damages, and nobody has shown it to exist.

The first and second are valid as well. We can measure the costs, can't we?
The second item shows that it can be happening today, you don't have to
prove it, because I did already. The fact that I disclosed publicly
about it doesn't mean that there aren't scores of others out there.
Considering the aggressive and illegal mailing campaign they operated,
one can expect that there are many certificates issued in this way.

>
> This situation is about as harmful as the average exploit demo. In this
> particular case it was caught by an insider rather than by an outsider.
> However, beyond that, the situation is already sealed in that Comodo
> have already taken the urgent triage steps to make sure nobody else gets
> one.
>

Well, that's exactly the point! I haven't heard back from Robin anymore,
their site is still online as well. Clear statements about how many
certificates are affected (all of the ones issued through them) and
revocations (or at least urgent re-validation within 48 hours and
thereafter revocation) would be satisfactory maybe.

The handling of the situation is insufficient, response and measures are
a joke, communication doesn't exist. Apparently this resellers business
was too good for Comodo.

Paul Hoffman

unread,
Dec 23, 2008, 4:54:22 PM12/23/08
to mozilla's crypto code discussion list
At 11:27 AM -0800 12/23/08, Kyle Hamilton wrote:
>I'd rather deal with disruption caused thereby (and, yes, the user
>complaints generated thereby -- at least then the end-user would KNOW
>that there's a problem that's being dealt with rather than having a
>FALSE SENSE OF SECURITY) than let those potential security breaches
>continue to wreak their quiet little havoc.

And I would not.

Just because a few people loudly proclaim their preferences on either side, it does not mean that their preferences should be acted on in a way that affects millions of Firefox users.

Paul C. Bryan

unread,
Dec 23, 2008, 4:59:57 PM12/23/08
to
> Just because a few people loudly proclaim their preferences on either side, it does not mean that their preferences should be acted on in a way that affects millions of Firefox users.

It was Comodo that affected millions of Firefox users; it's up to
Mozilla to protect those users by failing safe.

Paul C. Bryan

unread,
Dec 23, 2008, 5:05:13 PM12/23/08
to
Presumably it was Comodo that underwent an audit to be added to
Mozilla's roots, and Comodo should not be allowed to delegate trust to
their resellers for domain validation. If, today, trust is delegated
to their resellers, then we can't trust Comodo, period.

Although disruptive, their trust bits should be suspended. The
explanation to users: "The CA purporting to provide assurance about
the site you are trying to visit cannot be trusted. Please contact the
site operator and advise them to find a trustworthy certification
authority."

Yes, perception is that Mozilla releases code expressly to "break"
access to legitimate sites, but this is because a trusted CA has gone
rogue. Users can still jump through hoops to expressly include the
site's certificate and keep going.

The trust model for browsers should be fail-safe, even if this
inconveniences users. Better that than me and countless others
inadvertently exposing my credentials to a site pretending to be my
bank, investment house, government revenue agency, etc.

If Mozilla doesn't pull the trust bits, what's it's accountability for
any breaches that occur due to keeping the bits? With assurance must
come liability, whether from the certification authority, or those who
are implicitly trusted with vetting them.

Kyle Hamilton

unread,
Dec 23, 2008, 5:20:05 PM12/23/08
to mozilla's crypto code discussion list
The only effective and appropriate response to a root that does not
have sufficient internal controls to maintain its own security is to
remove the trust in it. If you've purchased a certificate from them
because it's trusted, and then they lose that trust, I would think
that you should be clamoring for your money back and looking for an
alternate certificate issuer rather than trying to maintain the
problem.

-Kyle H

Frank Hecker

unread,
Dec 23, 2008, 5:20:52 PM12/23/08
to
Eddy Nigg wrote:
> Concerning the disruption, Comodo has many roots and the resetting of
> this specific root would affect low-assurance sites as far as I know.

I don't think that's necessarily true. I don't think it would affect EV
sites (because of the way validation for those sites is special-cased),
but it could affect non-EV sites under other Comodo brands (I mean,
other than the PositiveSSL brand) and it could also affect other CAs
whose CA certs are cross-signed by the UTN-UserFirst-Hardware root.

Eddy Nigg

unread,
Dec 23, 2008, 5:31:13 PM12/23/08
to
On 12/24/2008 12:20 AM, Frank Hecker:

How about ADDING the chained issuing CA certificate to NSS and mark it
deliberately untrusted (no bits turned on)?

But our dilemma here shows clearly that we must have tools to act to
such threats. It's probably one of the lessons to learn from this incident.

Eddy Nigg

unread,
Dec 23, 2008, 5:39:24 PM12/23/08
to
On 12/24/2008 12:05 AM, Paul C. Bryan:

> Presumably it was Comodo that underwent an audit to be added to
> Mozilla's roots, and Comodo should not be allowed to delegate trust to
> their resellers for domain validation. If, today, trust is delegated
> to their resellers, then we can't trust Comodo, period.
>

I would second that and in light of many other problematic practices
which were discovered during their inclusion/update of EV, it's simply
too much. More than 24 hours into this, I've come to the conclusion that
this is a sever incident which requires action. If Robin can assure us
of reasonable actions from their side (as suggested previously by me) it
would serve all participants the best. Inaction and non-cooperation will
leave Mozilla with not much choice I think. Ignorance by Mozilla itself
will hunt it for a long time too. But it must happen now, either way!

(I don't think we have the time to discuss each and every aspect of RA
and reseller responsibilities and what we deem as save, I'm certain
we'll take this issue up (which apparently has about the same
implications as intermediate externally operated CAs))

Kyle Hamilton

unread,
Dec 23, 2008, 5:51:21 PM12/23/08
to mozilla's crypto code discussion list
I believe that Startcom (and other certification authorites in
Mozilla's root program) would likely have cause to bring an action for
the tort of negligence against Mozilla. I feel that this is something
that Mozilla should likely ask its general counsel very quickly.

0) Comodo is plainly found to have derelicted its duty to uphold the
Mozilla CA agreement. As a result, damage to the trust in the PKI
occurs, causing people to look outside of the PKI for solutions to the
problems that encryption and authentication can solve.

1) As a result of this Startcom and other CAs in the PKI have a
reduced market for their products. (damage)

2) Mozilla has the authority and ability to remove CAs from its trust
list if and when they are found to be negligent in their duties. It,
as an organization which attempts to vet the CAs that it includes in
its PSM and which has policies which allow for it to take action
should the policies be breached, has accepted the duty to remain
diligent. (duty, and proximity of duty, since in order to get into
the program in the first place all CAs in the PKI must accede to
Mozilla's demands of trustworthiness.)

3) Mozilla fails to remain diligent, and fails to remove trust bits
upon notice of dereliction. (causation, which leads to the trust in
the PKI being eroded further.)

I am not a lawyer, this is not legal advice, etc. I'm trying to
prevent Mozilla from having problems.

-Kyle H

Where Wolf

unread,
Dec 23, 2008, 6:10:57 PM12/23/08
to
On Dec 23, 5:51 pm, "Kyle Hamilton" <aerow...@gmail.com> wrote:
> I believe that Startcom (and other certification authorites in
> Mozilla's root program) would likely have cause to bring an action for
> the tort of negligence against Mozilla.  I feel that this is something
> that Mozilla should likely ask its general counsel very quickly.
>
Apart from the legal and procedural ramifications of all this, can you
guys publish a manual procedure for those of us who want to remove
Comodo's certificates from our browser installations? Assuming one is
available?

Eddy Nigg

unread,
Dec 23, 2008, 6:16:43 PM12/23/08
to
On 12/24/2008 01:10 AM, Where Wolf:

Select Preferences -> Advanced -> View Certificates -> Authorities.
Search for AddTrust AB -> AddTrust External CA Root and click "Edit".
Remove all Flags.

This would remove the trust from the potentially affected sites and
their certificates. Comodo has many more roots if you are interested,
such as all AddTrust, Comodo CA Limited, The Usertrust Network.

Paul Hoffman

unread,
Dec 23, 2008, 6:23:39 PM12/23/08
to mozilla's crypto code discussion list
At 3:15 PM +0200 12/23/08, Eddy Nigg wrote:
>If they don't shut that site, we can perhaps just publish the private key for the mozilla.com certificate as well so everybody can enjoy it.

It is indeed unbelievable to hear the COO of a CA company making threats like this. I'm sure that making such threats is not covered by Mozilla's inclusion policies, but maybe it should be.

And, yes, I'm serious. Given that Startcom has the ability to issue bogus certificates like the kind that Eddy is threatening, I would think that a public statement like the above is relevant to Mozilla or Microsoft deciding whether or not the organization is trustworthy.

Paul Hoffman

unread,
Dec 23, 2008, 6:27:14 PM12/23/08
to mozilla's crypto code discussion list
At 2:51 PM -0800 12/23/08, Kyle Hamilton wrote:
>I believe that Startcom (and other certification authorites in
>Mozilla's root program) would likely have cause to bring an action for
>the tort of negligence against Mozilla. I feel that this is something
>that Mozilla should likely ask its general counsel very quickly.
>
>. . .

>
>I am not a lawyer, this is not legal advice, etc.

The above is most certainly legal advice.

> I'm trying to
>prevent Mozilla from having problems.

Some people might read it differently.

Where Wolf

unread,
Dec 23, 2008, 6:29:26 PM12/23/08
to

> Select Preferences -> Advanced -> View Certificates -> Authorities.
> Search for AddTrust AB -> AddTrust External CA Root and click "Edit".
> Remove all Flags.
>
> This would remove the trust from the potentially affected sites and
> their certificates. Comodo has many more roots if you are interested,
> such as all AddTrust, Comodo CA Limited, The Usertrust Network.
>
> --
> Regards
>
> Signer: Eddy Nigg, StartCom Ltd.
> Jabber: start...@startcom.org
> Blog:  https://blog.startcom.org

Thanks Eddy. I found the option screen shortly after I posted here,
but I was assuming that the affected certificates would be listed
under Comodo's branch, which apparently wouldn't have done me a bit of
good. Sometimes it pays to ask even when you think you know the answer
(or if you're a teacher or lawyer, ONLY ask if you already know the
answer).

BTW I tracked into this discussion from your blog, so thanks for
bringing this whole thing to my attention as well.

Eddy Nigg

unread,
Dec 23, 2008, 6:45:59 PM12/23/08
to
On 12/24/2008 01:23 AM, Paul Hoffman:

> At 3:15 PM +0200 12/23/08, Eddy Nigg wrote:
>> If they don't shut that site, we can perhaps just publish the private key for the mozilla.com certificate as well so everybody can enjoy it.
>

Paul, you are disappointing me! I have not heard one critical word from
you about this incident, instead you are criticize *me*? C'mon, give me
a break!

I reported that my employees can see the supposedly private control
panel of this reseller - what else is needed to get this site down?

> And, yes, I'm serious. Given that Startcom has the ability to issue bogus certificates like the kind that Eddy is threatening, I would think that a public statement like the above is relevant to Mozilla or Microsoft deciding whether or not the organization is trustworthy.

I don't need the services of Comodo for that, if I would have ever
wanted to that, I could do so long time ago. Everybody in his right mind
understands exactly what I meant to say in this post. If you didn't,
that's your problem.

--
Regards

Signer: Eddy Nigg, StartCom Ltd.

Jabber: star...@startcom.org
Blog: https://blog.startcom.org

patr...@certstar.com

unread,
Dec 23, 2008, 6:58:28 PM12/23/08
to
Dear all,

I just wanted to give you all an update from Certstar. As you all know
we failed to validate a certificate due to a flaw in our system which
is clearly unacceptable.

Having worked intensively with this case I can truly say that Comodo
is indeed taking their responsibility extremely seriously and taking
huge efforts to ensure that safety of their root. Currently we are
going through a number certificate to double ensure that no further
mis-issuances have occurred.

Personally I do feel that it would have been appropriate if we had
been contacted by StartCom Ltd when they found this flow so that it
could have been fixed faster. Being our competitor I am not sure if we
could expect this, but it would indeed have been generous move.

The technical verification procedure has been improved and is now on a
very high security level. Comodo will also review our implementation
to ensure that it comply with all standards and cannot be abused.
Again, I would like to apologize to the community. Nothing does
however indicate that the flow was abused by others – but still we
have made a huge mistake which I sincerely apologize for.


--
kind regards,
Patricia, Certstar ApS

patr...@certstar.com

unread,
Dec 23, 2008, 7:12:49 PM12/23/08
to
On Dec 24, 12:45 am, Eddy Nigg <eddy_n...@startcom.org> wrote:
> I reported that my employees can see the supposedly private control
> panel of this reseller - what else is needed to get this site down?

The certificate has been revoked for quite some time now and is not
available in your control panel.

You must be aware of this as you submitted a complaint to Paypal that
“the certificate purchased was revoked” (you did not get what you paid
for). One could think that the purpose was to get them to suspend our
account as similar complaints was made the day before.

I do not wish avoid engaging in a personal discussion. Instead I would
like to get back to how we can solve this and make sure it does
*never* happen again – for Certstar or other SSL providers.

Paul C. Bryan

unread,
Dec 23, 2008, 7:33:18 PM12/23/08
to
On Dec 23, 3:58 pm, patri...@certstar.com wrote:
> The technical verification procedure has been improved and is now on a
> very high security level. Comodo will also review our implementation
> to ensure that it comply with all standards and cannot be abused.

As far as I know, you're not the party who is responsible, nor the one
to mitigate this issue; that lies with Commodo, and ultimately
Mozilla. Comodo should never have delegated trust to anyone to a third
party to perform such validation. In so doing, they have undermined
trust that can be placed in them.

Christoph Moormann

unread,
Dec 23, 2008, 7:39:21 PM12/23/08
to
On Dec 24, 12:58 am, patri...@certstar.com wrote:

> I just wanted to give you all an update from Certstar. As you all know
> we failed to validate a certificate due to a flaw in our system which
> is clearly unacceptable.

IIIRC you failed to validate at least two certificates, coincidentally
the ones by the thread starter.

> Having worked intensively with this case I can truly say that Comodo
> is indeed taking their responsibility extremely seriously and taking
> huge efforts to ensure that safety of their root. Currently we are
> going through a number certificate to double ensure that no further
> mis-issuances have occurred.

I'm anxious for the results, and i would be suprised if this was an
exception.

>Comodo will also review our implementation
> to ensure that it comply with all standards and cannot be abused.

You call it abuse to check whether someone hands out trusted
unvalidated certificates?
Do you realise that half of your text tries to blame other people.
Go ahead, shoot the messenger.
*shaking his head in disbelief*


Ian G

unread,
Dec 23, 2008, 7:40:36 PM12/23/08
to mozilla's crypto code discussion list
On 23/12/08 22:25, Eddy Nigg wrote:
> On 12/23/2008 11:12 PM, Ian G:
>> Earlier, Frank used the language of "clear and present danger."
>>
>> * clear: we can measure the costs of it, and cost of defences.
>> * present: it is happening today, provably.
>> * danger: it can be shown capable of doing damage, at least in theory
>>
>> Only the last one is shown. It is a danger in that anyone relying on a
>> cert wrongly issued could be harmed. But it hasn't actually caused
>> anyone damages, and nobody has shown it to exist.
>
> The first and second are valid as well. We can measure the costs, can't we?


First point: Who lost money? What are the damages?

You don't count, or more precisely, the money you spent getting the cert
doesn't count; sorry about that :)


> The second item shows that it can be happening today, you don't have to
> prove it, because I did already.


Yes, it's an "exploit" not a "breach".

This is a breach, this proves certs are proved to use bad stuff:

http://blogs.technet.com/mmpc/archive/2008/11/06/malware-and-signed-code.aspx

Indeed, some stats to underscore *presence* ...

"In the first six months of 2008, the MMPC received reports of 22M
instances of distinct malware files, of which about 173,000 were
distinct malware files with code signatures. Of this malware with code
signatures, about 38,000 were not validly code signed, so approximately
*135,000 validly signed malware files* were reported to Microsoft.
Approximately 0.6% of detected malware were validly code signed."

But they don't go far not enough to show the first point. (You might
validly ask why Microsoft did not go that far, and deny us the clarity
in dealing with this situation! Another subject, another day!)


> The fact that I disclosed publicly
> about it doesn't mean that there aren't scores of others out there.


It doesn't mean there aren't and it doesn't mean there are. On the
basis of a reasonable judgement, given the info we have today, there
aren't. Also, by the same logic, there aren't any mice in your
computer, even though you looked yesterday.

Which isn't to say that it won't change tomorrow. Security work is done
with facts, because attackers will use your conjectures against you.

> Considering the aggressive and illegal mailing campaign they operated,
> one can expect that there are many certificates issued in this way.


Sure. All highly targeted to existing owners of SSL certs ... who
surprise, surprise, didn't notice that there was no validation :)

What damages are likely here, to the targeted users? Are there any
"non-targeted users" ?


>> This situation is about as harmful as the average exploit demo. In this
>> particular case it was caught by an insider rather than by an outsider.
>> However, beyond that, the situation is already sealed in that Comodo
>> have already taken the urgent triage steps to make sure nobody else gets
>> one.
>>
>
> Well, that's exactly the point! I haven't heard back from Robin anymore,

> their site is still online as well.Clear statements about how many


> certificates are affected (all of the ones issued through them) and
> revocations (or at least urgent re-validation within 48 hours and
> thereafter revocation) would be satisfactory maybe.

> The handling of the situation is insufficient, response and measures are
> a joke, communication doesn't exist.


Right. Over to Comodo. It's their job, let them do it. Any statements
you desire to them should tie into some criteria, practice or policy.


> Apparently this resellers business
> was too good for Comodo.


Well, concerns on the concept of resellers have been raised before.
This becomes a case in point, which should perhaps allow us to
re-address the issue.

But, CAs issue bad certs, that's a fact, so we can't damn them entirely.
Also, it's their job to deal with it. Adding a lynching party helps
no-one.

iang

Eddy Nigg

unread,
Dec 23, 2008, 7:41:53 PM12/23/08
to patr...@certstar.com
Hi Particia,

On 12/24/2008 01:58 AM, patr...@certstar.com:
> Dear all,
>

Upfront, as such for me, I accept your apology - sincerely. However your
apology is only for half the story, please read on...

>
> Having worked intensively with this case I can truly say that Comodo
> is indeed taking their responsibility extremely seriously and taking
> huge efforts to ensure that safety of their root. Currently we are
> going through a number certificate to double ensure that no further
> mis-issuances have occurred.

Look, the problem isn't your implementations alone, this failure is a
broader set of failures not only by your company, but that of Comodo.
Comodo has clearly failed in their duties, in their concepts, controls
and policies. The responsibility of domain validation shouldn't be with
you, but part of Comodo's procedures and data flow. As much as your
system perhaps failed - you aren't the only one to take blame and
responsibility.

Besides that, I want to repeat that there was no validation performed at
all. There was no such step, not even a hint of it.

>
> Personally I do feel that it would have been appropriate if we had
> been contacted by StartCom Ltd when they found this flow so that it
> could have been fixed faster. Being our competitor I am not sure if we
> could expect this, but it would indeed have been generous move.

I tell you something about generosity hereby. You have spammed, phished
and mislead our customers by sending those emails, with it everything
started. StartCom has contacted you (myself personally) and requested to
talk to the owners of your company and somebody with the name Mark
answered me. I replied with full disclosure about those emails you are
sending and with immediate request to shut down your activities.
Unfortunately *OUR* email was ignored and we never received a reply
thereafter.

My request was reasonable in light of the potential damage you have
caused us and other certification authorities and whoever makes a
mistake has to pay for it. Your site should have been taken down then.

Also Comodo replied laconically to my complaint. In that respect, and
the way Comodo handled this incident, nobody should be surprised that
who make foes, will not be treated friendly either. It would have been
generous from *YOU* to address the first problem appropriately and come
to terms with us when we requested it.

>
> The technical verification procedure has been improved and is now on a
> very high security level. Comodo will also review our implementation
> to ensure that it comply with all standards and cannot be abused.

As long as those validations are performed at a hosted site at some
hosting provider (yes, I made my researches in this respect) and not by
Comodo's infrastructure, there can't be any talk of "improvement",
period. This is simply not serious! I hope that everybody takes note on
this.

Eddy Nigg

unread,
Dec 23, 2008, 8:35:00 PM12/23/08
to
On 12/24/2008 02:40 AM, Ian G:

> You don't count, or more precisely, the money you spent getting the cert
> doesn't count; sorry about that :)

At least I got it refunded ;-)

>
> Well, concerns on the concept of resellers have been raised before. This
> becomes a case in point, which should perhaps allow us to re-address the
> issue.

Yes indeed. When the right time comes, this must be addressed appropriately.

>
> But, CAs issue bad certs, that's a fact, so we can't damn them entirely.
> Also, it's their job to deal with it. Adding a lynching party helps no-one.
>

Nope, and flaws can happen, no system is immune. However in this case
the handling was a sever failure from top to bottom, this isn't a bug or
isolated failure.

David E. Ross

unread,
Dec 23, 2008, 8:45:19 PM12/23/08
to

I've turned off all trust bits for all Comodo certs in my own personal,
home installation of SeaMonkey. That was step one. I might turn some
back on when I see the consequences. So far, however, I have not
encountered any secure site using Comodo.

The bigger issue is what will be done by the Mozilla organization. Now
that Mozill knows about the problem, I believe they have a liability to
their users if they fail to take action to mitigate the risks to those
users. This is analogous to laws in many U.S. states that hold a city
liable for automobile damage from a pothole AFTER the pothole is
reported to the city; there is no liability before knowledge but much
liability after knowledge (even if the city did not cause the pothole).

--
David E. Ross
<http://www.rossde.com/>

Go to Mozdev at <http://www.mozdev.org/> for quick access to
extensions for Firefox, Thunderbird, SeaMonkey, and other
Mozilla-related applications. You can access Mozdev much
more quickly than you can Mozilla Add-Ons.

ro...@comodo.com

unread,
Dec 23, 2008, 8:56:59 PM12/23/08
to
On Dec 23, 5:09 am, Frank Hecker <hec...@mozillafoundation.org> wrote:
> There are two general reasons for pulling a root, to address a clear and
> present danger to Mozilla users, and to punish a CA and deter others. My
> concern right now is with the former. I see at least three issues in
> relation to that:
>
> 1. Issuance of further non-validated certs by this reseller. Comodo
> seems to have addressed this by suspending the reseller's ability to get
> certs issued. (I can testify that this is the case, as I tried to
> duplicate Eddy's feat earlier today and got my uploaded CSR rejected.)
>
> 2. Potential problems with certs already sold through this reseller.
> Comodo should investigate this and take action if needed. (This need not
> necessarily require revoking all certificates associated with the
> reseller; for example, the existing certs and their associated domains
> could be re-validated, the registered domain owners could be notified of
> the potential for bogus certs floating around, etc.)
>
Frank,
We are in the process of reviewing all of the certificates that have
been issued where Certstar served as a RA for Comodo. Fortunately they
have only been involved in the validation of 111 certificates. As I
previously mentioned as soon as we discovered the error with the
Mozilla.com certificate we suspended Certicom’s RA privileges.
Certstar’s RA activities remain suspended.

At this point it appears that the Mozilla and Startcom certificates
were an anomaly, the result of the automatic DCV mechanism being
bypassed by accident. Certstar’s account will remain suspended until
we are satisfied that the certificates issued for Mozilla and Startcom
were as Certstar maintains, the result of unintentional mistakes that
have been corrected with safeguards in place to prevent a
reoccurrence. Should Comodo decide to reinstate Certstar as an RA we
will review each certificate request until we are satisfied that
Certstar has been adequately trained and that yesterday’s mistake
cannot be duplicated.

Comodo has been able to verify that 73 of the 111 orders processed by
Certstar were processed pursuant to the requirements of our CPS and
our webhost RA terms and conditions. We should have our review of the
remaining 38 certificates completed by tomorrow evening. If we are
unable to verify any validation, we will revoke the applicable
certificate.


> 3. Potential problems with other Comodo resellers. I'm not going to tell
> Comodo how to operate its reseller network, but they certainly should
> take a look at whether and where this might be a problem with other
> resellers, and how they could revamp their systems to reduce potential
> problems with resellers.
>
Comodo takes it responsibility to supervise RAs very seriously and we
actively audit their performance. While it is not practical to audit
100% of their work, we audit a representative sample. In the past we
have discovered only a few isolated incidents where our sub CAs or RAs
failed to follow the applicable guidelines. When this has occurred we
have suspended the account, performed additional audit activities and
where appropriate revoked certificates. We are confident that our
existing system works well and are constantly looking for ways to
improve it.

We apologize for Certstar’s mistake and assure you that we will
redouble our self-auditing efforts to insure the problem does not
repeat itself.

Regards
Robin Alden
Comodo

Eddy Nigg

unread,
Dec 23, 2008, 8:59:03 PM12/23/08
to
On 12/24/2008 02:12 AM, patr...@certstar.com:

> On Dec 24, 12:45 am, Eddy Nigg<eddy_n...@startcom.org> wrote:
>> I reported that my employees can see the supposedly private control
>> panel of this reseller - what else is needed to get this site down?
>
> The certificate has been revoked for quite some time now and is not
> available in your control panel.

Nevertheless I saw it with my own eyes at my co-workers PC. And no, it's
not a proxy on our end. This was about 24 hours after I last visited
your site.

>
> You must be aware of this as you submitted a complaint to Paypal that
> “the certificate purchased was revoked” (you did not get what you paid
> for). One could think that the purpose was to get them to suspend our
> account as similar complaints was made the day before.
>

I bought the certificate for proving that Comodo issues unvalidated
certificates after realizing this during my initial attempt about who
stands behind those shameless and fraudulent emails - and got one for
startcom.org, no questions asked.

And yes, I requested from any possible party to shut your operation
down: Godaddy (which also operates a CA), Paypal (which owns a CA),
Servage (your hosting provider), Comodo (of which you are a reseller).

Dan Colascione

unread,
Dec 23, 2008, 9:02:54 PM12/23/08
to
On Dec 23, 8:56 pm, ro...@comodo.com wrote:
> Comodo has been able to verify that 73 of the 111 orders processed by
> Certstar were processed pursuant to the requirements of our CPS and
> our webhost RA terms and conditions.
> [snip]
> In the past we
> have *discovered* only a few isolated incidents where our sub CA*s* or RA*s*

> failed to follow the applicable guidelines.

[emphasis mine]

Frankly, that's even *more* disturbing. It means that there are almost
certainly unverified certificates in the wild, and that this problem
is pervasive.

Paul C. Bryan

unread,
Dec 23, 2008, 9:13:57 PM12/23/08
to
On Dec 23, 5:56 pm, ro...@comodo.com wrote:
> Comodo takes it responsibility to supervise RAs very seriously and we
> actively audit their performance. While it is not practical to audit
> 100% of their work, we audit a representative sample.

By delegating RA functions (including domain verification) to third
parties, you appear to be making them the weakest links in PKI chains
of trust.

> We apologize for Certstar’s mistake and assure you that we will
> redouble our self-auditing efforts to insure the problem does not
> repeat itself.

Some questions:

1. Does Comodo take full responsibility for the actions of its
resellers? If so, how should the repercussions of such failures be to
Comodo?

2. Are resellers subject to the same audits that Comodo presumably had
to undergo to get its root certs added to Mozilla? Who performs, and
who verifies such audits? How often are they performed?

3. Are you willing to openly, continually disclose your list of
resellers, the frequency of audits, audit methodology, and actual
audit reports so that third parties can evaluate whether Comodo is
trustworthy as a CA?

Nelson B Bolyard

unread,
Dec 23, 2008, 9:16:54 PM12/23/08
to mozilla's crypto code discussion list
Ian G wrote, On 2008-12-23 05:58:

> 3. How to resolve a dispute. This is a Mozilla action &
> responsibility. Reverse-engineering and referring, I would suggest this
> as a teaser:
>
> a. The CA certificate "module owner" at Mozilla foundation is
> responsible. Ref, the policy, pt 15.
> b. The dispute is investigated and ruled on by module owner.

For there to be a "CA certificate module owner", there would have to be
a "CA certificate Module". Presently, there is no such module. Maybe
there should be. As you probably know, Mozilla now has "code modules"
and "non code modules". I think the CA certificate module would be a
non-code module. Yeah, ultimately whatever certs are accepted by Mozilla
do get stored in NSS, but the NSS developers just abide with Mozilla's
decisions about that. There's only ever been one real disagreement
between Mozilla and NSS about root CA certs, and that was how the whole
process we have now got started. Since then, it's been smooth sailing.
I expect it will continue to be that for a long time.

Anyway, I would support the creation of a "CA certificate" non-code module.

Eddy Nigg

unread,
Dec 23, 2008, 9:23:11 PM12/23/08
to
On 12/24/2008 04:16 AM, Nelson B Bolyard:

Seconded +1

Eddy Nigg

unread,
Dec 23, 2008, 9:51:53 PM12/23/08
to
My blog article and exposure has provoked somebody to come forward with
additional evidences concerning the reseller activities of Comodo. In
order to protect the innocent I decided to provide this information
confidentially to Frank Hecker for now. Stay tuned.

David E. Ross

unread,
Dec 23, 2008, 10:14:14 PM12/23/08
to

I've already cleared all trust bits for the three Comodo certificates in
my home installation of SeaMonkey. Should I also do the same for the
certificates under those other names? In other words, what have YOU
done with respect to this issue?

Paul Hoffman

unread,
Dec 23, 2008, 10:32:54 PM12/23/08
to mozilla's crypto code discussion list
At 1:45 AM +0200 12/24/08, Eddy Nigg wrote:
>Paul, you are disappointing me! I have not heard one critical word from you about this incident,

What would be added by me joining the choir? Clearly, Comodo made a mistake in trusting (at least) one of its resellers. The mistake was laid bare, and the folks who might remove Comodo from the root pile are following the issue, probably more closely than they are letting on. Do you really think "oh, but if only Paul Hoffman would be critical, then things will really change"?

FWIW, I would be shocked if you could not get the same result (a cert without sufficient checking of the domain) for a lower-profile domain name from at least five other resellers of other CAs in the root pile. You tried to find this one because this particular reseller tried to steal your customers in a slimy fashion, but you could probably find other resellers (possibly even Comodo resellers) who are just as lax.

> instead you are criticize *me*?

Yes.

>C'mon, give me a break!

You are repeatedly using this list as a springboard to criticize a competitor. When you didn't get your way instantly, you made threats against Mozilla, an organization for which many of us have a lot of respect. No break is justified.

>I reported that my employees can see the supposedly private control panel of this reseller - what else is needed to get this site down?

I guess you aren't reading the responses from the people on the thread that might not be as upset as you are. That question was already answered.

>>And, yes, I'm serious. Given that Startcom has the ability to issue bogus certificates like the kind that Eddy is threatening, I would think that a public statement like the above is relevant to Mozilla or Microsoft deciding whether or not the organization is trustworthy.
>
>I don't need the services of Comodo for that, if I would have ever wanted to that, I could do so long time ago.

Yes, exactly. And you, the COO/CTO of a trusted CA, are making public threats that would be the equivalent of that. I understand that you don't think that is a problem; please understand that other might think it is.

Paul Hoffman

unread,
Dec 23, 2008, 10:33:00 PM12/23/08
to mozilla's crypto code discussion list
At 1:16 AM +0200 12/24/08, Eddy Nigg wrote:
>Select Preferences -> Advanced -> View Certificates -> Authorities. Search for AddTrust AB -> AddTrust External CA Root and click "Edit". Remove all Flags.
>
>This would remove the trust from the potentially affected sites and their certificates. Comodo has many more roots if you are interested, such as all AddTrust, Comodo CA Limited, The Usertrust Network.

I'm glad that Eddy was the one who posted this. Doesn't this seem like a better solution than "sue Mozilla for theoretical future damages incurred by using their free software"? Put more rudely, why do you expect Daddy to fix it for you when you can fix it yourself, more easily and more quickly, if you want to?

Eddy Nigg

unread,
Dec 23, 2008, 10:50:23 PM12/23/08
to
On 12/24/2008 05:32 AM, Paul Hoffman:

> At 1:45 AM +0200 12/24/08, Eddy Nigg wrote:
>> Paul, you are disappointing me! I have not heard one critical word from you about this incident,
>
> You tried to find this one because this particular reseller tried to steal your customers in a slimy fashion...

Well, I didn't tried to find the flaw, I tried to find the CA
responsible. It took me a while to recap how I got the cert for my
domain and went like...gosh, this can't be, I must have overlooked
something. I decided to check it out and bingo.

As to my personality, what I wrote in my blog article is what I meant. I
was clearly disappointed, because I'm also a strong supporter of domain
validated certificates and that they have their place in PKI (provided
that they are applied correctly). Obviously, this incident makes it
harder if we don't act accordingly.


> but you could probably find other resellers (possibly even Comodo resellers) who are just as lax.

I really hope not. And it should serve others to re-check their controls
and procedures.

>
> You are repeatedly using this list as a springboard to criticize a competitor.

That's how you look at it. I think I've contributed and proved that I'm
not against our competition, I'm against devaluing our work
collectively. One CA is enough to get us looking for other jobs if you
don't mind.

> When you didn't get your way instantly, you made threats against Mozilla, an organization for which many of us have a lot of respect.

I never threatened Mozilla. If at all, I made clear what the results may
be. Including this specific incident shows clearly that my arguments and
concerns were real in many ways. Here we have a CA which issues domain
validated wild cards up to ten years. Now you tell me how this is OK -
specially in light of what happened.

I care about what I'm doing, yes! And I have no intention apologizing
for it.

> Yes, exactly. And you, the COO/CTO of a trusted CA, are making public threats that would be the equivalent of that. I understand that you don't think that is a problem; please understand that other might think it is.

OK, maybe I should have made my point different.

jack...@gmail.com

unread,
Dec 23, 2008, 11:31:48 PM12/23/08
to
On Dec 22, 7:15 pm, "Robin Alden" <ro...@comodo.com> wrote:
> Eddy,
>         That reseller's ability to sell Comodo certificates has been
> suspended while we investigate why they are apparently not fulfilling their
> contractual obligations to us.
>         We revoked your certificate for mozilla.com.
>
> Regards
> Robin Alden
> Comodo
>
> > -----Original Message-----
> > From: dev-tech-crypto-bounces+robin=comodo....@lists.mozilla.org
> > [mailto:dev-tech-crypto-bounces+robin=comodo....@lists.mozilla.org] On
> > Behalf Of Eddy Nigg
> > Sent: Monday, December 22, 2008 10:25 PM
> > To: dev-tech-cry...@lists.mozilla.org
> > Subject: Unbelievable!
>
> >https://blog.startcom.org/?p=145

>
> > --
> > Regards
>
> > Signer: Eddy Nigg, StartCom Ltd.
> > Jabber: start...@startcom.org
> > Blog:      https://blog.startcom.org
> > _______________________________________________
> > dev-tech-crypto mailing list
> > dev-tech-cry...@lists.mozilla.org
> >https://lists.mozilla.org/listinfo/dev-tech-crypto

I guess Comodo didn't do their own due diligence. How can anyone trust
Comodo?

Gen Kanai

unread,
Dec 23, 2008, 11:54:48 PM12/23/08
to mozilla's crypto code discussion list

Kyle Hamilton

unread,
Dec 24, 2008, 12:20:01 AM12/24/08
to mozilla's crypto code discussion list
On Tue, Dec 23, 2008 at 6:16 PM, Nelson B Bolyard <nel...@bolyard.me> wrote:
>
> Anyway, I would support the creation of a "CA certificate" non-code module.

I think this would be a really good idea. I'm aware that my opinion
carries little weight, but I think that since it relies on business-
and legal-side undertakings, it shouldn't be managed by the coders.

How would this work? Split nssckbi out to be managed by the non-code
module owner, though a coder would need to be enlisted to finalize the
decisions made by that person?

-Kyle H

Ian G

unread,
Dec 24, 2008, 5:53:25 AM12/24/08
to mozilla's crypto code discussion list
On 24/12/08 03:16, Nelson B Bolyard wrote:
> Ian G wrote, On 2008-12-23 05:58:
>
>> 3. How to resolve a dispute. This is a Mozilla action&
>> responsibility. Reverse-engineering and referring, I would suggest this
>> as a teaser:
>>
>> a. The CA certificate "module owner" at Mozilla foundation is
>> responsible. Ref, the policy, pt 15.
>> b. The dispute is investigated and ruled on by module owner.
>
> For there to be a "CA certificate module owner", there would have to be
> a "CA certificate Module". Presently, there is no such module. Maybe
> there should be.

I was searching for the role, and found that term in the policy, I
think. Mentally, I call it "the CA desk" and sometimes use that term.
The earlier text had a term "CA Manager" which is a bit old-fashioned.

It's whatever it ends up being called, the point for me was "one person
at Mozo who knows this stuff, decides!"

> As you probably know, Mozilla now has "code modules"
> and "non code modules". I think the CA certificate module would be a
> non-code module. Yeah, ultimately whatever certs are accepted by Mozilla
> do get stored in NSS, but the NSS developers just abide with Mozilla's
> decisions about that. There's only ever been one real disagreement
> between Mozilla and NSS about root CA certs, and that was how the whole
> process we have now got started. Since then, it's been smooth sailing.
> I expect it will continue to be that for a long time.
>

> Anyway, I would support the creation of a "CA certificate" non-code module.


OK.

To address Frank's comment:

>> 4. Finality. ... In the policy, it says "... mozilla.org staff..."

> There is no longer a "mozilla.org staff" ...

(In which case, a policy change will be needed one day.)

I would let my comment stand. The point here is that if the day-to-day
manager is disputed, an immediate next-desk-warmer isn't going to help
much. Go for the top, hand it to the ultimate responsible party, the
board. The need here is to send a serious signal, short circuit the
process, and avoid any additional layers of bureaucracy.

iang

Frank Hecker

unread,
Dec 24, 2008, 9:17:00 AM12/24/08
to
Gen Kanai wrote:
> More discussion on this topic over at Programming Reddit:
>
> http://www.reddit.com/r/programming/comments/7lb96/ssl_certificate_for_mozillacom_issued_without/

Unfortunately the discussion devolved (as it always does :-) into the
merits of self-signed certificates. Oh well.

Frank

--
Frank Hecker
hec...@mozillafoundation.org

Frank Hecker

unread,
Dec 24, 2008, 10:37:50 AM12/24/08
to
Eddy Nigg wrote:
> My blog article and exposure has provoked somebody to come forward with
> additional evidences concerning the reseller activities of Comodo. In
> order to protect the innocent I decided to provide this information
> confidentially to Frank Hecker for now. Stay tuned.

To expand on what I wrote to Eddy via email, I really don't want to be
dealing with information sent to me in confidence. I'm skeptical about
dealing with reports where the originators of the reports aren't willing
to go on public record with their complaints, especially when there are
possibly mixed motives on the part of the person or organization
reporting the problem. (For example, CAs reporting on either CAs, or
resellers reporting on other resellers.)

Kyle Hamilton

unread,
Dec 24, 2008, 12:14:32 PM12/24/08
to mozilla's crypto code discussion list
On Wed, Dec 24, 2008 at 6:17 AM, Frank Hecker
<hec...@mozillafoundation.org> wrote:
> Gen Kanai wrote:
>>
>> More discussion on this topic over at Programming Reddit:
>>
>>
>> http://www.reddit.com/r/programming/comments/7lb96/ssl_certificate_for_mozillacom_issued_without/
>
> Unfortunately the discussion devolved (as it always does :-) into the merits
> of self-signed certificates. Oh well.
>
> Frank

Honestly, I get more mileage out of doing my own diligence. This is
why I want to have my own root certificate that I can cross-sign the
various commercial CA certificates with, so that I can revoke those
cross-signatures if they turn out to have a problem.

I'd like to see an extension that allows other certificates (for the
same public key) to be included in a certificate (self-signed or not).
This would protect against one of the threats against the OpenPGP
model, would create a means for certifications to be cherry-picked for
any given application, and would allow other identities for that
public key to be added by the identity collector -- but would also
create a linkage back to the identity collector's public key.
(realistically, the only reason for the signature on an external
certificate that contains an extension like this would be because no
X.509 software handles the TBSCertificate structure without the
signature.)

-Kyle H

Paul Hoffman

unread,
Dec 24, 2008, 12:55:34 PM12/24/08
to mozilla's crypto code discussion list
At 9:14 AM -0800 12/24/08, Kyle Hamilton wrote:
>I'd like to see an extension that allows other certificates (for the
>same public key) to be included in a certificate (self-signed or not).

Are you asking for a Mozilla extension or a PKIX extension? If the latter, none is needed: it is already inherent in PKIX. In fact, I am not sure that anything needs to be done by Mozilla. The following should theoretically work:

- Remove all trust anchors one-by-one
- Add your single trust anchor
- Sign the certs of any CA you want
- Add those signed certs to the pre-loaded validation path (not root) cert list

I haven't tried this myself, but it should work. I have been told that something very similar to it works fine in XP/Vista for IE.

--Paul Hoffman

Kyle Hamilton

unread,
Dec 24, 2008, 2:34:55 PM12/24/08
to mozilla's crypto code discussion list
On Tue, Dec 23, 2008 at 5:14 AM, Frank Hecker
<hec...@mozillafoundation.org> wrote:
> Eddy Nigg wrote:
>>
>> Disabling the trust bits of "AddTrust External CA Root" could be a
>> temporary measure to prevent damage to relying parties until Mozilla
>> receives full report and disclosure from Comodo about its resellers and
>> conclusion of their investigation.
>
> Do you mean the UTN-UserFirst-Hardware root? According to the screenshot on
> your blog post, that's the root the bogus cert chains up to. Also, if we
> were to take action of this general sort (as a hypothetical), what about
> adding the PositiveSSL CA cert to NSS with the SSL trust bit disabled;
> wouldn't that accomplish the same purpose, without interfering with other
> parts of the hierarchy under the UTN-UserFirst-Hardware root? (I seem to
> recall we've discussed this sort of thing in the past.)

What is the effect of this problem on the request to enable the
UTN-UserFirst-Hardware root for EV,
https://bugzilla.mozilla.org/show_bug.cgi?id=401587 ?

-Kyle H

Kyle Hamilton

unread,
Dec 24, 2008, 2:35:56 PM12/24/08
to mozilla's crypto code discussion list
In the terminology of ASN.1 and PKIX, I want a standardized PKIX
extension that allows for a SEQUENCE OF Certificate within the
tbsCertificate structure.

I'm trying to figure out how I'm supposed to extract all the
certificates from my database without any version of keytool that I
can find available for OSX 10.5 (Leopard).

-Kyle H

> _______________________________________________
> dev-tech-crypto mailing list
> dev-tec...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tech-crypto
>

John Nagle

unread,
Dec 24, 2008, 1:41:41 PM12/24/08
to dev-tec...@lists.mozilla.org
As a user of SSL certificates in our SiteTruth system, which
attempts to identify and rate the business behind a web site, we're
concerned about CA reliability and trust. We've been using Mozilla's
approved root cert list for our system, and are considering whether
we should continue to do so. Given the situation described in

"http://markmail.org/message/rje3lssql55l2rev?q=unbelievable"

the use of Mozila's root CA list is now questionable.

There seem to be several problems here.

1. AddTrust, a company which apparently no longer exists, has an approved
root CA certificate. This in itself is troublesome. Comodo does
not seem to have taken on the obligations of AddTrust; see
"http://markmail.org/message/3zr4e5hxwmxjbgnp?q=Comodo+AddTrust".
Yet Comodo is still issuing certificates under the root CA of
this dead company.

2. Comodo is apparently not only allowing resellers like CertStar,
but is allowing them to do their own validation of the legitimacy
of the certificate requestor. Who takes financial responsibility
for such errors? CertStar itself disclaims financial responsibility
at "http://www.certstar.com/terms.html".

3. Microsoft requires an annual audit for root CAs:
"http://technet.microsoft.com/en-us/library/cc751157.aspx".
Mozilla seems willing to accept a one-time audit. That seems
to be why the disappearance of AddTrust wasn't noticed.
Microsoft's audit requirements extend all the way down the
chain of trust. Their policy is:
"The CA must complete an audit and submit audit results to Microsoft
every twelve (12) months. The Audit must cover the full PKI hierarchy
that will be enabled by Microsoft through the assignment of Extended
Key Usages (EKUs). All certificate usages that we enable must be
audited periodically. The audit report must document the full scope
of the PKI hierarchy including any sub-CA that issues a specific type
of certificate covered by an audit." Microsoft uses the standards of
the WebTrust Program for Certification Authorities
("http://www.cica.ca/download.cfm?ci_id=45239&la_id=1&re_id=0")
managed by the Canadian Society of Chartered Accountants. That's
a good guideline for Mozilla to follow.

At this point, I would suggest that the following actions are appropriate
to bring Comodo and Mozilla into compliance with the WebTrust standards:

1. Comodo must undergo an audit to WebTrust standards, and the audit
report must be published. An in-house self-investigation is not
acceptable. The audit must be conducted by a recognized outside
auditing firm.
2. CertStar must separately undergo an audit to WebTrust standards,
and the audit report must be published. An in-house
self-investigation is not acceptable. The audit must be conducted
by a recognized outside auditing firm. CertStar should not be
permitted to issue any new certificates until this process is
complete and the audit results are satisfactory. If this process
is not complete within 3 months, all CertStar-issued certificates
should be revoked.
3. Comodo must disclose the identities of all "resellers" to which
it has outsourced any validation functions.
4. All AddTrust root CA certificates must be phased out. An
expiration date in Q1 or Q2 2009 is suggested. Since no company
stands behind the AddTrust name, it's necessary to force expiration
of that root.
5. The Mozilla Foundation should contact Microsoft's CA Root Certificate
group to coordinate their actions.

John Nagle
SiteTruth

Nelson B Bolyard

unread,
Dec 24, 2008, 4:42:45 PM12/24/08
to mozilla's crypto code discussion list
Kyle Hamilton wrote, On 2008-12-23 21:20:
> On Tue, Dec 23, 2008 at 6:16 PM, Nelson B Bolyard <nel...@bolyard.me> wrote:
>> Anyway, I would support the creation of a "CA certificate" non-code module.
>
> I think this would be a really good idea. I'm aware that my opinion
> carries little weight, but I think that since it relies on business-
> and legal-side undertakings, it shouldn't be managed by the coders.
>
> How would this work? Split nssckbi out to be managed by the non-code
> module owner, though a coder would need to be enlisted to finalize the
> decisions made by that person?

No, it would be a NON-CODE module. It would not contain any code.
Its output would be the list of trusted root certs, perhaps as a web page,
and/or also as a set of requests (in the form of Bugzilla bugs) to have
certs inserted into nssckbi.

nssckbi is just a medium for the conveyance of that list, potentially one
of several. The task for the NSS module owner would be to ensure that
the copy of the list in nssckbi is kept reasonably up to date, and doesn't
differ from the official list (or a very recent version of that list) as of
the date on which it is released.

That's really how things operate now. I'm merely suggesting that that
separation of responsibility be formalized by making the maintenance of
the official CA list be a separate "module".

Nelson B Bolyard

unread,
Dec 24, 2008, 4:46:41 PM12/24/08
to mozilla's crypto code discussion list
Paul Hoffman wrote, On 2008-12-24 09:55:
> At 9:14 AM -0800 12/24/08, Kyle Hamilton wrote:
>> I'd like to see an extension that allows other certificates (for the
>> same public key) to be included in a certificate (self-signed or not).
>
> Are you asking for a Mozilla extension or a PKIX extension? If the
> latter, none is needed: it is already inherent in PKIX. In fact, I am not
> sure that anything needs to be done by Mozilla. The following should
> theoretically work:
>
> - Remove all trust anchors one-by-one
> - Add your single trust anchor
> - Sign the certs of any CA you want
> - Add those signed certs to the pre-loaded validation path (not root)
> cert list
>
> I haven't tried this myself, but it should work. I have been told that
> something very similar to it works fine in XP/Vista for IE.

Of course, that is COMPLETELY equivalent to simply setting trust flags on
the CA certs you want to trust, and removing those flags from the ones you
don't want to trust, which is already a part of Mozilla browsers (and
Netscape browsers, before them) for over 14 years.

Kyle Hamilton

unread,
Dec 24, 2008, 5:36:40 PM12/24/08
to mozilla's crypto code discussion list

To be honest, Mozilla doesn't distribute keytool with Firefox, which
means that I have to try to go into the (unbatchable) interface and
remove the flags one. by. one. by. one. and then select the next
certificate and remove those trust flags, and the next, and the next,
and the next...

...for all hundred or so certs that Firefox includes.

And then, once I DO manage to do that, then with the "new and
improved" user interface updates, I then have to click at least six
times to try to figure out what's going on, and then when I do find a
site that's protected by an unknown CA certificate (OR that I've
removed the trust bits on), I have to do the following:

1) Click 'add an exception'
2) click 'get certificate' (why I should have to do this is beyond me,
since firefox obviously already has the certificate downloaded since
it told me 'sec_error_untrusted_issuer', which it couldn't have known
without the certificate in its possession ANYWAY)
3) click 'view'
4) get the name of the Issuer
5) hope to all the gods that there's enough information in the chain
to figure out what root it's supposed to be going to
6) close the window
7) go into Preferences
8) click Advanced
9) click Encryption
10) click 'View Certificates'
11) Scroll through the list, with each click giving me approximately
0.6 useful results (given the preponderance of 'section headings by
root owner', which by the way doesn't work at all with the Addtrust AB
stuff since those are Comodo roots)
12) find the appropriate root and re-enable it for identification of websites
13) refresh the page.

How 'bout this, Nelson (and I invite Frank and the entire security UI
team to do this, as well): YOU do it. Create a new profile and
manually remove the trust on every CA. Then, browse around, and see
which CAs are actually used by you in your day-to-day browsing,
reenabling them manually (since you're trying to emulate not having
keytool around).

Furthermore, even when keytool IS available, it's entirely likely that
its name conflicts with Java's keytool. (especially on Mac OSX.)

This is completely unworkable, and discourages users that want to from
taking their security into their own hands.

-Kyle H

Kyle Hamilton

unread,
Dec 24, 2008, 5:38:00 PM12/24/08
to mozilla's crypto code discussion list
I'm also going to state that yes, I know this, because I HAVE DONE IT.
And I wouldn't wish that hell on anyone who didn't have a DETAILED
knowledge of how the X.509 model operates, and I wouldn't wish the
user-interface hell on ANYONE.

-Kyle H

Kyle Hamilton

unread,
Dec 24, 2008, 5:42:25 PM12/24/08
to mozilla's crypto code discussion list

Thanks for the explanation.

I do agree that the separation of responsibility would be good, since
Frank (appears to?) does the actual CA approval and you appear to be
the one primarily who implements his directives as regards the updates
to nssckbi?

-Kyle H

Eddy Nigg

unread,
Dec 24, 2008, 5:46:08 PM12/24/08
to
On 12/25/2008 12:36 AM, Kyle Hamilton:

> To be honest, Mozilla doesn't distribute keytool with Firefox, which
> means that I have to try to go into the (unbatchable) interface and
> remove the flags one. by. one. by. one. and then select the next
> certificate and remove those trust flags, and the next, and the next,
> and the next...

Kyle, why don't you blow that libnssckbi away from your Firefox
installation? Would make it easier I think. (Hope I picked the right one
;-) )

--
Regards

Signer: Eddy Nigg, StartCom Ltd.

Jabber: star...@startcom.org
Blog: https://blog.startcom.org

Kyle Hamilton

unread,
Dec 24, 2008, 5:53:08 PM12/24/08
to mozilla's crypto code discussion list
On Wed, Dec 24, 2008 at 2:46 PM, Eddy Nigg <eddy...@startcom.org> wrote:
> On 12/25/2008 12:36 AM, Kyle Hamilton:
>>
>> To be honest, Mozilla doesn't distribute keytool with Firefox, which
>> means that I have to try to go into the (unbatchable) interface and
>> remove the flags one. by. one. by. one. and then select the next
>> certificate and remove those trust flags, and the next, and the next,
>> and the next...
>
> Kyle, why don't you blow that libnssckbi away from your Firefox
> installation? Would make it easier I think. (Hope I picked the right one ;-)
> )

Primarily because I want those certs on one profile, but not another,
and disk space is kind of at a premium right now. :)

(Oh yeah, if one person who uses a computer doesn't want the built-in
roots, but another does, they have to have separate Firefox
installations.)

-Kyle H

Nelson B Bolyard

unread,
Dec 24, 2008, 5:53:39 PM12/24/08
to mozilla's crypto code discussion list
Kyle Hamilton wrote, On 2008-12-24 14:42:

> Thanks for the explanation.
>
> I do agree that the separation of responsibility would be good, since
> Frank (appears to?) does the actual CA approval

Yes

> and you appear to be the one primarily who implements his directives as
> regards the updates to nssckbi?

I did it at one time. Presently, and for the last year or more, keeping
nssckbi up to date with Frank's list has been done by Kai Engert who
also works on PSM, the browser component that interfaces to NSS.

Nelson B Bolyard

unread,
Dec 24, 2008, 6:14:51 PM12/24/08
to mozilla's crypto code discussion list
Kyle Hamilton wrote, On 2008-12-24 14:53:
> On Wed, Dec 24, 2008 at 2:46 PM, Eddy Nigg <eddy...@startcom.org> wrote:
>> On 12/25/2008 12:36 AM, Kyle Hamilton:
>>> To be honest, Mozilla doesn't distribute keytool with Firefox, which
>>> means that I have to try to go into the (unbatchable) interface and
>>> remove the flags one. by. one. by. one. and then select the next
>>> certificate and remove those trust flags, and the next, and the next,
>>> and the next...

What is the relevance of keytool here?
The tool relevant to NSS is certutil. It is scriptable.
It's quite possible to write scripts to use certutil to do mass trust
modification. I've done it. But there's probably a much easier way
than that.

>> Kyle, why don't you blow that libnssckbi away from your Firefox
>> installation? Would make it easier I think. (Hope I picked the right one ;-)
>> )
>

> Primarily because I want those certs on one profile, but not another,
> and disk space is kind of at a premium right now. :)
>
> (Oh yeah, if one person who uses a computer doesn't want the built-in
> roots, but another does, they have to have separate Firefox
> installations.)

No, it is possible to accommodate both with a single installation.
There are several ways to do it. Here's one.
Move the nssckbi file from the Firefox code directory where it normally
lives to the profile directory (where the cert8.db file lives) for the
user profile that wants to use it.

sayrer

unread,
Dec 24, 2008, 7:50:46 PM12/24/08
to
On Dec 23, 10:33 pm, Paul Hoffman <phoff...@proper.com> wrote:
> At 1:16 AM +0200 12/24/08, Eddy Nigg wrote:
>
> >Select Preferences -> Advanced -> View Certificates -> Authorities. Search for AddTrust AB -> AddTrust External CA Root and click "Edit". Remove all Flags.
>
> Put more rudely, why do you expect Daddy to fix it for you when you can fix it yourself, more easily and more quickly, if you want to?

Mozilla is in the business of protecting all users, not just those
with sysadmin levels of skill. I'm not advocating drastic action with
the Comodo roots, but a workarounds of this sort are probably not
distinguishable from total failure when the entire user base is taken
into account.

The truth is that we are basically unable to act without a lot of
collateral damage. We should keep this in mind with future security
technology. Relying on companies willing to take money for doing
absolutely nothing (not even the bare minimum they agreed to) is not a
pleasant thing to do to our users. We didn't learn this lesson with
EV--maybe next time! :)

- Rob

Paul Hoffman

unread,
Dec 24, 2008, 7:58:48 PM12/24/08
to mozilla's crypto code discussion list
At 1:46 PM -0800 12/24/08, Nelson B Bolyard wrote:
>Paul Hoffman wrote, On 2008-12-24 09:55:
> > - Remove all trust anchors one-by-one
>> - Add your single trust anchor
>> - Sign the certs of any CA you want
>> - Add those signed certs to the pre-loaded validation path (not root)
> > cert list
>
>Of course, that is COMPLETELY equivalent to simply setting trust flags on
>the CA certs you want to trust, and removing those flags from the ones you
>don't want to trust, which is already a part of Mozilla browsers (and
>Netscape browsers, before them) for over 14 years.

Not "COMPLETELY", but close. What I proposed has a signature at the top of the hierarchy, which is what I thought that Kyle was asking for. The result is completely equivalent, but the format is slightly different.

Of course, it is much easier for the people on this list to Insist With Exclamation Marks! that Mozilla fix this for them instead of them doing it themselves, but that problem is at different layer.

Paul Hoffman

unread,
Dec 24, 2008, 7:54:44 PM12/24/08
to mozilla's crypto code discussion list
At 11:35 AM -0800 12/24/08, Kyle Hamilton wrote:
>In the terminology of ASN.1 and PKIX, I want a standardized PKIX
>extension that allows for a SEQUENCE OF Certificate within the
>tbsCertificate structure.

That makes no sense to me, but I would have to see a complete proposal to understand why you would want that.

>I'm trying to figure out how I'm supposed to extract all the
>certificates from my database without any version of keytool that I
>can find available for OSX 10.5 (Leopard).

That's a completely different problem, of course.

Daniel Veditz

unread,
Dec 25, 2008, 2:13:55 AM12/25/08
to mozilla's crypto code discussion list
Paul Hoffman wrote:
> At 1:16 AM +0200 12/24/08, Eddy Nigg wrote:
>> Select Preferences -> Advanced -> View Certificates -> Authorities.
>> Search for AddTrust AB -> AddTrust External CA Root and click
>> "Edit". Remove all Flags.
>
> Doesn't this seem like a better solution than "sue Mozilla for
> theoretical future damages incurred by using their free software"?
> Put more rudely, why do you expect Daddy to fix it for you when you
> can fix it yourself, more easily and more quickly, if you want to?

Isn't that a bit like an auto maker issuing a notice "If you don't want
your car to explode in a fender bender you can crawl under the right
rear and remove the screw marked 34A on the following diagram."

Everyone should be able to do that so I'm sure that gets the auto maker
off the hook, right? Assuming all 200 million Firefox users even hear
about the problem.

Daniel Veditz

unread,
Dec 25, 2008, 4:56:20 AM12/25/08
to
Kyle Hamilton wrote:
> I then have to click at least six
> times to try to figure out what's going on, and then when I do find a
> site that's protected by an unknown CA certificate (OR that I've
> removed the trust bits on), I have to do the following:
>
> 1) Click 'add an exception'
> 2) click 'get certificate' (why I should have to do this is beyond me,
> since firefox obviously already has the certificate downloaded since
> it told me 'sec_error_untrusted_issuer', which it couldn't have known
> without the certificate in its possession ANYWAY)

Not that it changes your point any, but if you set the pref
browser.ssl_override_behavior to 2 then you won't need to "get
Certificate". Firefox 3.1 will have this behavior by default.

If you set browser.xul.error_pages.expert_bad_cert to true then you
won't have to click a link to reveal the "add exception" button, saving
a click at that step, too. Firefox 3.1 won't be adopting that default
though.

Michael Ströder

unread,
Dec 25, 2008, 6:15:42 AM12/25/08
to
Kyle Hamilton wrote:
> (Especially if Comodo delegates full Registration Authority capability
> without verification, which seems to be the case -- though they could
> have simply issued a sub-CA certificate.)

Delegating the RA's tasks is still different from issuing a sub-CA cert
since with a delegated RA the CA can look at all issued EE certs and
revoke some of them if needed. A sub-CA typically runs completely on its
own so the CA could only revoke the sub-CA cert and not certain EE certs.

> It occurs to me that there is no facility in Firefox or other Mozilla
> products to provide an explanatory dialog that there's an issue, and
> such a facility would be extremely useful at this point. Being able
> to print a message to the user like "The Mozilla Foundation has
> identified issues with the trusted root that issued this certificate
> which prevent Firefox from being able to guarantee that this is truly
> the site to which you intended to go. While it is unlikely that this
> is a widespread problem, and an attack would rely on more technical
> intrusions into the network, the nature of these issues requires that
> you be warned of this circumstance so that you can exercise
> appropriate levels of caution. The Mozilla Foundation is working with
> the trusted root to resolve these issues." would help a lot.

Either the trust bits are removed or not. Such a dialogue wouldn't help
at all. It's too complicated.

Ciao, Michael.

Michael Ströder

unread,
Dec 25, 2008, 6:21:02 AM12/25/08
to
Eddy Nigg wrote:
> On 12/23/2008 09:09 AM, Kyle Hamilton:
>> Of course, this would be an NSS change (the addition of a 'trust
>> suspended' bit,
>
> I think this to be an interesting idea and should be considered.

I really wonder why there should be one state more. And how is it going
to be set (updated)? A cert is either trusted or not. Period.

Ciao, Michael.

Michael Ströder

unread,
Dec 25, 2008, 6:04:51 AM12/25/08
to
Frank Hecker wrote:
> From my point of view I'd wait on more
> information regarding items 2 and 3 above before making a recommendation.

Could you please define a time-frame within Comodo MUST react?

Ciao, Michael.

It is loading more messages.
0 new messages