Signer: Eddy Nigg, StartCom Ltd.
> dev-tech-crypto mailing list
How can you outsource such a critical part as domain control validation
to a reseller is a complete mystery to me! Your controls (if any) have
And apparently if the fish stinks at the tail, chances that the head is
rotten too are pretty high.
> We revoked your certificate for mozilla.com.
I suggest to revoke ALL certificates from this reseller and perform an
urgent review of your policies and implementations in relation to
resellers at all. Other steps might be needed as well, you know better
As I noted in my prior correspondence, Comodo has undertaken an internal
review of the Certstar reseller account. We have informed CertStar that
their email violates their contractual obligation to refrain from sending
unsolicited emails and that their email could be interpreted as misleading
and confusing to the customer. During our review, we discovered that
Certstar had apparently issued a certificate to mozilla.com without
validating control of the domain. We immediately revoked the certificate
(prior to your posting) and have suspended Certstar's reseller activities
until our investigation has been completed. Please let me know if you have
any further problems.
> -----Original Message-----
> From: dev-tech-crypto-bounces+robin=comod...@lists.mozilla.org
> [mailto:dev-tech-crypto-bounces+robin=comod...@lists.mozilla.org] On
> Behalf Of Eddy Nigg
A.) The certificate was revoked after posting to this list.
B.) Your CA also issued a certificate to startcom.org without validating
C.) Yes, I think we have a problem of a wider scale here. This is not
about me personally.
I advocate at least temporarily removing the trust bits from Comodo
until a new external audit can be completed, with an eye toward
ensuring that Comodo, not the reseller, perform the domain
There are two general reasons for pulling a root, to address a clear and
present danger to Mozilla users, and to punish a CA and deter others. My
concern right now is with the former. I see at least three issues in
relation to that:
1. Issuance of further non-validated certs by this reseller. Comodo
seems to have addressed this by suspending the reseller's ability to get
certs issued. (I can testify that this is the case, as I tried to
duplicate Eddy's feat earlier today and got my uploaded CSR rejected.)
2. Potential problems with certs already sold through this reseller.
Comodo should investigate this and take action if needed. (This need not
necessarily require revoking all certificates associated with the
reseller; for example, the existing certs and their associated domains
could be re-validated, the registered domain owners could be notified of
the potential for bogus certs floating around, etc.)
3. Potential problems with other Comodo resellers. I'm not going to tell
Comodo how to operate its reseller network, but they certainly should
take a look at whether and where this might be a problem with other
resellers, and how they could revamp their systems to reduce potential
problems with resellers.
Pulling a Comodo root will knock out Firefox, etc., access to thousands
of SSL sites, maybe tens of thousands. Given the disruption that would
cause, the final decision on this IMO should be made in conjunction with
the Firefox security folks. From my point of view I'd wait on more
information regarding items 2 and 3 above before making a recommendation.
It occurs to me that there is no facility in Firefox or other Mozilla
products to provide an explanatory dialog that there's an issue, and
such a facility would be extremely useful at this point. Being able
to print a message to the user like "The Mozilla Foundation has
identified issues with the trusted root that issued this certificate
which prevent Firefox from being able to guarantee that this is truly
the site to which you intended to go. While it is unlikely that this
is a widespread problem, and an attack would rely on more technical
intrusions into the network, the nature of these issues requires that
you be warned of this circumstance so that you can exercise
appropriate levels of caution. The Mozilla Foundation is working with
the trusted root to resolve these issues." would help a lot.
(I word it like that because in order for an attacker to succeed he
would need to also hijack DNS, or place a entry in the user's hosts
Of course, this would be an NSS change (the addition of a 'trust
suspended' bit, in addition to the trust flags), a Firefox/Thunderbird
code change (to look for that bit), and a chrome change (to explain
what's going on). Or even a check for the name in the hosts file to
see if it's overridden from DNS. Placing an entry in a user's hosts
file is easier than hijacking DNS, or at least it's supposed to be.
I'm pretty sure we're all aware of the fairly recent cache poisoning
attack, but the people who write BIND pushed out a change to protect
against it very rapidly.
The addition of a 'trust suspended' bit is primarily unlikely to
happen, though, because there's too much inertia in the way things are
to do what needs to be done to fix the authentication system to allow
a root to stay in while the user is potentially at risk.
A glitch in our validation system has today caused a certificate to be
issued to a person who successfully abused our system.
We have now strengthened our domain validation system so that such
abuse cannot happen again. Comodo has handled this issue in a
professional way by invoking the certificate immediately after issuing
and contacting Certstar.
Again, I cannot stress enough how seriously we take this issue and I
would like to apologize to the Mozilla organization for the mis-issue.
Patricia, Certstar ApS
> We have now strengthened our domain validation system so that such
> abuse cannot happen again.
just curious: How do you normally validate domain ownership?
It's not me who abused your system, it's your company which sent out
illegal, misleading emails to our customers. Since I couldn't find out
who stands behind your site I had to get a certificate. There was no
abuse, there was only use-and-get.
- No affiliation published, the fact that you are a reseller for Comodo
is nowhere noted.
- No subscriber agreement or notice presented to subscribers.
- No control validation of the domain name. Just pay-and-go.
I didn't had to do anything to overcome any validation whatsoever. There
was no abuse!
> We have now strengthened our domain validation system so that such
> abuse cannot happen again. Comodo has handled this issue in a
> professional way by invoking the certificate immediately after issuing
> and contacting Certstar.
Yes, after I posted my article here at the mailing list.
> Again, I cannot stress enough how seriously we take this issue and I
> would like to apologize to the Mozilla organization for the mis-issue.
As long as this site keeps operating, our customers are still being let
to believe that they have to renew their certificate with them. This is
only a reminder about how it started at all. CAs and their customers are
still taking damage from the previously sent messages.
> 2. Potential problems with certs already sold through this reseller.
> Comodo should investigate this and take action if needed. (This need not
> necessarily require revoking all certificates associated with the
> reseller; for example, the existing certs and their associated domains
> could be re-validated, the registered domain owners could be notified of
> the potential for bogus certs floating around, etc.)
You shouldn't notify the subscribers or domain name owners, but the
relying parties. How to do that is up to you and Comodo I guess.
Comodo not only shouldn't just investigate and take action, Comodo needs
to report publicly about their findings and full report about the
actions taken. This isn't a suggestion of resolution about this
incident, it's the transparency I expect from them at this hour.
> Pulling a Comodo root will knock out Firefox, etc., access to thousands
> of SSL sites, maybe tens of thousands.
I'm not advocating removing their root, however we must assess the risk
which is potentially caused to the relying parties. There may be
thousands of sites which received certificates without validating them.
> Given the disruption that would
> cause, the final decision on this IMO should be made in conjunction with
> the Firefox security folks.
Disabling the trust bits of "AddTrust External CA Root" could be a
temporary measure to prevent damage to relying parties until Mozilla
receives full report and disclosure from Comodo about its resellers and
conclusion of their investigation.
Additionally instead of just yanking a root as a deterrent and
punishment, as you mentioned above, I'd prefer to receive a commitment
from Comodo to address other issues noted during the last discussion -
mainly those listed in the "Problematic Practices" document. Considering
that OCSP in Firefox is set to soft-fail, an issue with their OCSP
server could still circumvent bogus sites for potentially the next five
years. A full review of their current status in NSS and their practices
might be advocated too.
Or be a WiFi operator. This was the attack vector of
> Of course, this would be an NSS change (the addition of a 'trust
> suspended' bit,
I think this to be an interesting idea and should be considered.
Actually Nelson opened this bug.
1) An unsolicited commercial email (UCE) message was sent from your
company to the party in question suggesting that there already existed
a relationship between your company and the party in question. This
is obvious from the verb 'renew' in the original message -- a
non-party to the original certificate can't renew "on behalf of" the
original certificate issuer. If they can, there's a major problem.
1a) The message from your company, and in fact the entire process up
to and including paying for the certificate which your company issued,
did not expressly claim an affiliation, and the party in question went
through the process of "renewal" with the intent of figuring out which
CA's services were being advertised via UCE.
2) The party in question (Eddy Nigg) is the founder of another
Certifying Authority, with a root which participates in the Mozilla
root program and which is also included in the root store.
3) It falls within the concept of "due diligence" for a
security-conscious warden of a public trust to recognize the lack of
actual domain control verification in a trusted certificate issuance
process, and *for the purposes of verification and public
notification* obtain a certificate which could be shown absolutely to
have been issued in violation of the standards of at least one of the
root programs that the issuer (or issuer's trust-delegator)
3a) If this had not been identified, someone else would have found it
eventually -- and the reaction to a gross violation of security
standards is to immediately believe that the worst-case scenario has
occurred: that it had already been found and exploited by at least one
4) The end-users (referred to as "relying parties") are well-served by
this identification of completely bogus credentialing.
5) It's also important to recognize the following (the first comment
------- Comment #1 From Reed Loden [:reed] 2008-12-23 01:35:37
The same company that Eddy was able to get the mozilla.com cert from (Certstar)
has been endlessly spamming webm...@mozilla.org since the beginning of
December complaining that one of our SSL certs had "expired" and needed to be
"renewed" (both of which were false). They have continued to spam us almost
------ end comment ------
Yes, the bare facts are that a user exploited the lack of verification
in your certificate issuance process. However, the lack of
verification in your process violates at least one contractual
obligation that your company is required to uphold, and reduces the
value of both Comodo's root AND the commercial Certifying Authority
structure in general. Because the user founded another commercial CA
which is trusted by Mozilla NSS by-default, this user was operating in
a manner to verify that the commercial CA structure was in fact
secure; the trust afforded this user's CA is demonstrably harmed by
your lack of verification.
I do not take this situation as lightly as you appear to wish that
everyone would. The initial fault is YOUR COMPANY'S (and that fault
reverbrates up to Comodo, since it delegated trust to your company),
and the fact that you're attempting to shift the blame onto the user
shows that you are absolutely untrustworthy to run or be the public
face of any commercial certificate-issuance service.
Because of this, my recommendation that Comodo's trust bits be removed
until a full audit of their practices (and a full audit of all issued
certificates) stands, and I am that much more resolute in my belief.
It may be that the integrity of Comodo's root has been irreparably
damaged by your company's malfeasance; if this is the case, I
certainly hope they rake you over the coals.
Do you mean the UTN-UserFirst-Hardware root? According to the screenshot
on your blog post, that's the root the bogus cert chains up to. Also, if
we were to take action of this general sort (as a hypothetical), what
about adding the PositiveSSL CA cert to NSS with the SSL trust bit
disabled; wouldn't that accomplish the same purpose, without interfering
with other parts of the hierarchy under the UTN-UserFirst-Hardware root?
(I seem to recall we've discussed this sort of thing in the past.)
Also note that any "suspension" of a root would last at last 1-3 months,
since that the typical interval between security updates for Firefox and
other Mozilla-based products.
Thanks for that. More into this story...
...all our employees coming the our network are able to retrieve the
certificates and all the rest from the "control panel" of this site.
Again, no questions asked. Apparently the only things one needs is to
fake our gateway IP address to get there.
It's getting weirder with the second. If they don't shut that site, we
can perhaps just publish the private key for the mozilla.com certificate
as well so everybody can enjoy it.
(My opinion, of course. Comodo's gotta be breathing a sigh of relief
that I don't work for the Mozilla Foundation, else I would have yanked
the trust bits first and figured it out later.)
This incident shows anew the lack of policy/procedure of what to do in
the case of a gross breach of the root agreement, though. The entire
point of certificates in the first place is "user security", not "user
convenience". (If this weren't the case, none of us would have any
desire to get certificates in the first place.) Your reaction seems
to be based on placing "user convenience" over "user security", and
(again, my opinion) I don't believe that this is appropriate at all.
The reason for doing this, and adding yet more boring verbage to the
mass of words, is that it can prevent uneccessary legal costs by
providing clear guidepaths, and clear signals of seriousness.
Here's what I see:
1. How to file a dispute against Mozilla and/or a CA. This seems
fairly easy; file a bug in bugzilla and mark it as already described
here: https://wiki.mozilla.org/CA:How_to_apply . (With mods: Severity:
2. What is the practice on revocation or un-trusting on roots? This is
is perhaps a headline case of a dispute, so possible merits its own
notes. Frank has suggested the test:
a. there is a clear and present danger to Mozilla users, or
b. to punish a CA, and/or to deter others.
I would suggest that (b) be modifed slightly to give it a basis, which
in this case might be "for breaches of policy or practices". The
Policy, pt 4, gives the authority, as well as some examples, and adds
another test case:
c. where certificates cause technical problems with mozo software.
3. How to resolve a dispute. This is a Mozilla action &
responsibility. Reverse-engineering and referring, I would suggest this
as a teaser:
a. The CA certificate "module owner" at Mozilla foundation is
responsible. Ref, the policy, pt 15.
b. The dispute is investigated and ruled on by module owner.
c. The ruling is listed in the bug report above.
d. Many disputes will be dealt with by communication, and no ruling
will be required. This will create a default "closed, no action" ruling.
4. Finality. What happens if we disagree with the decision of the
module owner? In the policy, it says "CAs or others objecting to a
particular decision may appeal to mozilla.org staff, who will make a
final decision." Ref, policy, pt 15.
I would wonder about this; google suggests that "staff" is as listed here:
but that seems out of date. Also, due to the absence of this forum in
the public eye, I doubt it musters the credibility we need in dispute
review where the legal and contractual significance is high. E.g., is
there any way we can review the decisions they made in the past?
There are several possibilities:
(i) Ruling is final.
(ii) Mozilla.org staff, policy, pt 15.
(iii) Review by board of Mozilla Foundation.
(iv) Review by some independent party.
(v) Review by forum at law: courts, or Arbitrator.
Personally, I would plumb for (iii) and suggest the Mozo Foundation
board as the next step. It is expensive, but available. The directors
already have fiduciary responsibility, and can thus deal with the
significance. It is also aligned with the review of the manager
concerned, the policy and the general contractual issues.
End! I'd encourage others to comment!
If nobody has any objections, I can add that as a page into the wiki as
an informal document of practices, including or without those questions.
My intent is to balance the disruption that would be caused by pulling a
root vs. the actual security threat to users. Right now we have no real
idea as to the extent of the problem (e.g., how many certs might have
been issued without proper validation, how many of those were issued to
malicious actors, etc.).
I don't have time right now to address your whole post, but I did want
to comment on one point.
> 4. Finality. What happens if we disagree with the decision of the
> module owner? In the policy, it says "CAs or others objecting to a
> particular decision may appeal to mozilla.org staff, who will make a
> final decision." Ref, policy, pt 15.
> I would wonder about this; google suggests that "staff" is as listed here:
> but that seems out of date.
There is no longer a "mozilla.org staff" as traditionally constituted.
From a Mozilla community point of view the closest present-day
equivalent is the system described at
Basically there is a group of people that oversee the Mozilla module
ownership system and the module owners within it. That group in turn
operates under general policy structures set by Mitchell Baker and
Brendan Eich, and as you note the ultimate legal authority resides in
the board of the Mozilla Foundation (of which Mitchell and Brendan are
For more background on this see
This was the first thing that occurred to me as well. Can anyone
(Nelson?) tell us definitively if this is possible? I would assume it
is, but it would be good to get confirmation.
Are we going to receive information from Certstar as to how many other
certs may have been issued in error?
How do we verify the claims from Comodo or Certstar?
I've asked Robin Alden of Comodo to make an accounting regarding these
two issues. I don't expect to see that immediately (i.e., in the next
day or two), though I also don't expect to wait a month for it (as Kyle
is concerned about).
> How do we verify the claims from Comodo or Certstar?
Some things by their nature we can directly verify, and some we can't.
For example, we can certainly verify how Certstar is operating currently
(as Eddy and I did), and if certs get revoked we can verify that by
downloading the CRLs that get issued.
> My intent is to balance the disruption that would be caused by pulling
> a root vs. the actual security threat to users. Right now we have no
> real idea as to the extent of the problem (e.g., how many certs might
> have been issued without proper validation, how many of those were
> issued to malicious actors, etc.).
Isn't that, by itself, a very good reason to take immediate action?
Security should be default-fail rather than default-pass.
Actually, I think it's very important that the accounting include this:
for each name (not just certificate, but name in
subjectAlternativeNames) that has been certified, a connection to the
TLS ports should be made, and the certificate presented by the site
compared against the certificate that Comodo issued. This obviously
won't be a complete verification, but it should give a start to see
how widespread the problem is.
A script to do this could probably be written fairly easily, but
depending on the number of certificates Comodo has issued that are
currently valid (and I'd like to see some hard numbers on that, as
well) it could take a while to run.
From the script, the numbers I'd like to see are: the number of
unreachable/not-answering names/hosts, the number of matching
certificates, and the number of mismatched certificates. From that
output plus Comodo's records, I would also like to see how many
resellers there are and how many of them have sold mismatched
I hate to say this, but this IS The Worst-Case Scenario. A CA has
gone rogue and issued certificates that violate its standards, and the
standards of the root programs that it's a part of -- it is true that
Comodo didn't /intend/ to go rogue, but it has, and we can't afford to
let it damage the greater PKI. Since every CA in the root store is
treated the same, there is no differentiation between them -- and this
means that Verisign and Comodo and Thawte and *every* CA share the
same reputation. If one goes rogue, it's exactly the same as if all
of them have gone rogue, in the eye of the end-user.
To put it another way, it's exactly the same problem as putting the
public keys of webservers in DNSSEC. Since the end-system can't know
if a response came from DNSSEC or just unauthenticated DNS, the
end-system can't trust anything (including RFC2538-style CERT records)
that comes from DNS.
THIS is why I want to see greater differentiation in the browser
chrome between CAs, so that one bad apple doesn't spoil the whole root
barrel. THIS is why the argument against changing the chrome (user
On Tue, Dec 23, 2008 at 11:15 AM, Hendrik Weimer <hen...@enyo.de> wrote:
> Frank Hecker <hec...@mozillafoundation.org> writes:
>> My intent is to balance the disruption that would be caused by pulling
>> a root vs. the actual security threat to users. Right now we have no
>> real idea as to the extent of the problem (e.g., how many certs might
>> have been issued without proper validation, how many of those were
>> issued to malicious actors, etc.).
> Isn't that, by itself, a very good reason to take immediate action?
> Security should be default-fail rather than default-pass.
It should be. I realized that there are more points which will have to
be addressed in due time at Mozilla, which however can wait for now.
Concerning the disruption, Comodo has many roots and the resetting of
this specific root would affect low-assurance sites as far as I know.
The higher validated sites would not be affected. Having said that, the
roots of their low-assurance products have been previously of concern it
must be looked at in the broader context of Comodo's operations. It's
not surprising in itself.