Many of you are probably already following this story.  This is my
current roundup:
http://freedom-to-tinker.com/blog/sjs/web-browsers-and-comodo-announc...

On 03/23/2011 09:16 PM, From Steve Schultze:

....and we are waiting patiently for a posting by a Mozilla
representative here.

--
Regards

Signer:  Eddy Nigg, StartCom Ltd.
XMPP:    start...@startcom.org
Blog:    http://blog.startcom.org/
Twitter: http://twitter.com/eddy_nigg

On 3/23/11 6:07 PM, Eddy Nigg wrote:

I appreciate your patience...
(I've been ill, so not online much the past couple of days.)

A good description about what happened has been posted here:
https://www.eff.org/deeplinks/2011/03/iranian-hackers-obtain-fraudule...

Comodo has provided their summary of the events here:
http://blogs.comodo.com/category/it-security/
http://www.comodo.com/Comodo-Fraud-Incident-2011-03-23.html

We were informed by Comodo promptly, and they followed the Security Bugs
process: http://www.mozilla.org/projects/security/security-bugs-policy.html

The Security Bug is here:
https://bugzilla.mozilla.org/show_bug.cgi?id=642395

The bad certs were all revoked promptly, but blacklisting allows us to
cope with the possibility that an attacker might block OCSP if they
control the victim's network. OCSP failures aren't fatal by default
(though they do downgrade EV to DV) so we wanted to make sure these
certs never worked.

Mozilla requested that Comodo take the following actions.

1) Revoke the RA rights of the RA concerned.  (Done)

2) Monitor their OCSP logs for evidence of use of these certificates, or
evidence that access to their OCSP responders is being blocked from
certain geographical locations

3) Publish a full account of exactly what happened. (See comodo.com
links above)

4) Provide compelling public evidence that:
- the root private key was not compromised
- the channel of mis-issuance is now shut
- all mis-issued certs have been found and listed
- the list of other certs to come through that channel recently has been
carefully audited, and representatives of each organization are
being/have been contacted to confirm correct issuance.
(partially complete, Comodo plans to engage WebTrust auditors)

5) Move, in a short time-frame (small number of months) towards a model
where each RA issues from a separate subordinate CA certificate.
(started)

We are still evaluating the situation, and may request further action
from Comodo.

Additionally, this issue has brought several concerns about
implementation of CRL and OCSP to the forefront, and some new bugs have
been filed. Also, it turned out that an NSS-based fix for this instance
was not as simple as we would like. A new bug has been created for
improving NSS to allow us to add certs to the store which are
specifically untrusted:
https://bugzilla.mozilla.org/show_bug.cgi?id=642503

Comodo responded appropriately to this incident by revoking the
certificates immediately, and notifying the major browser vendors so
that more proactive actions could be taken.

This incident alone is concerning enough, but as we are all aware, there
are previous incidents that have been logged in Bugzilla regarding certs
issued within Comodo’s hierarchy, for example:

https://bugzilla.mozilla.org/show_bug.cgi?id=470897
https://bugzilla.mozilla.org/show_bug.cgi?id=526560
https://bugzilla.mozilla.org/show_bug.cgi?id=599856

Of particular note is the comment in
https://bugzilla.mozilla.org/show_bug.cgi?id=470897#c68
"… we have retrofitted our own DV process into the RA's ordering process
in the vast majority of cases.
By 'our own DV process', I mean that Comodo performs an automated check
of domain control by sending (and confirming receipt of) an email to an
address which is either on the domain to be validated or is explicitly
mentioned in the WHOIS entry for the domain to be validated."

We have asked Comodo to explain why this mechanism was not active (or
did not work) for this RA.

I am extremely concerned about the history of questionable certs being
issued within the Comodo hierarchy. One possible result is for Mozilla
to turn off all of the trust bits for the Comodo root certificates that
are included by default in NSS.  However, I think that the CAs actions
in response to the most recent situation should also be taken under
consideration. It would have been much easier for the CA to not say
anything to browser vendors about their mistake – they had caught it
quickly and revoked the certs already. Comodo was very much aware of the
previous issues that had been logged in Bugzilla, and of the possible
outcome of making this mistake public. Regardless, Comodo chose to do
the right thing and disclose this situation to the browser vendors so
that they could also take action.

We could focus on punishment, but I think it would be more beneficial
for us to focus on actions that will prevent such mistakes in the
future. As well as sharing the lessons-learned, to help prevent other
CAs from making the same mistakes.

I ask the representatives of Comodo to contribute actively to this
discussion group. In particular, please described the actions Comodo is
taking now (e.g. the list of requested actions above), as well as the
actions that Comodo plans to take to prevent such a mistake from
happening again.

I have added a few items to
https://wiki.mozilla.org/CA:CertPolicyUpdates#Second_Phase
Under the heading “RAs and Subordinate CAs”
There are some practices, such as some roots signing end-entity certs
directly, that this incident has demonstrated should no longer be
allowed, and need to be corrected soon.

I am also following the CAB Forum and EFF SSL Observatory discussions on
this topic. So no need to copy info from them, unless there are specific
action items that pertain to Mozilla.

Kathleen

....

Kathleen,

Is your stipulation that the "parent" CA should still be in full control of those subordinates and able to audit/control all activities of the subordinates?  Maybe that is implicit in your requirements, but I don't see it specifically called out.  Comodo's current model allowed them to see these certs, had they used a "fully delegated" subordinate with their RAs this activity wouldn't have had the same visibility.  

- Andy

On 3/24/11 5:24 PM, Kathleen Wilson wrote:

What is the definition of "revoke"?  Does it have a cryptographic or
operational implication?

I do not believe that the account is yet "full," as Jake observed:
https://blog.torproject.org/blog/detecting-certificate-authority-comp...

As Andy noted, this point...

....seems mutually exclusive with this point...

....and the latter seems preferable.  The combination, in which the
"parent" does validation and issuance, but uses a different sub-CA for
each RA seems ideal.

This episode raises concerns not just about Comodo's practices, but
about Mozilla's.  How is it that a well-understood problem from 3 years
ago (Comodo's unsafe RA practices) was not confirmed to be resolved by
now?  I'll ask again...

http://groups.google.com/group/mozilla.dev.security.policy/browse_thr...

....will Mozilla mandate public disclosure of all RAs, and the CA's
precise practices with respect to delegating authority
(cryptographically or operationally) to them?

This logic only incentivizes CAs to not perform due diligence under the
assumption that as long as they admit their mistakes publicly they will
not be held accountable.  You're right, Comodo was very much aware of
the previous issues that had been logged in Bugzilla... and they even
knew what they had to do to mitigate their harm, and claimed that they
were doing so, but evidently did not.

How is this time different from last time?

On Mar 24, 2:31 pm, "Steingruebl, Andy" <asteingru...@paypal-inc.com>
wrote:

Hmmm. I'm not seeing this posting in Thunderbird, only in Google
Groups.

Yes, for the case where an intermediate CA corresponds to an RA, I was
assuming that the "parent" CA would be in full control of the
intermediate CAs.

I add this clarification to the related bullet point in the wiki page:
https://wiki.mozilla.org/CA:CertPolicyUpdates#RAs_and_Subordinate_CAs

Thanks,
Kathleen

On 3/24/11 4:19 PM, Steve Schultze wrote:

Operational.

Fair enough.

The intent is that Comodo operates these intermediate CAs and maintains
full control/auditing of them. The point is that if an RA does make a
mistake, Comodo can revoke that particular intermediate CA.

I think these practices should be incorporated by any CA who uses
external RAs. I've added points about both of these to
https://wiki.mozilla.org/CA:CertPolicyUpdates#RAs_and_Subordinate_CAs

Fair enough. At this point I think we should ask Comodo to do a full
audit of all of their RAs. It appears that their updated practices
haven't been adopted by all of their RAs.

I don't know about publicly disclosing the list of their RAs.

But I do agree that Comodo should publicly disclose their practices with
respect to delegating authority to their RAs. They should also provide a
description of the changes that they are going to make, have new
training for all of their RAs, and audit all of their RAs.

It's a shame if CAs view it that way.

I'll wait for Comodo's answers to pass judgment, but I agree that it's
concerning that practices that were said to be put into place don't
appear to have been enforced with all of their RAs

Well, the way it happened is different. Also, Comodo caught it
immediately, and took action immediately -- it didn't take someone else
to notice. So apparently they do have a practice of monitoring their RAs
in place that did work. Some of that monitoring should be automated, and
should happen before the cert gets issued...

Kathleen

On 3/24/11 8:31 PM, Kathleen Wilson wrote:

How do you plan to ensure that they are doing what they are claiming to
do without requiring disclosure?  Disclosure and the resulting
accountability is a fundamental part of Mozilla's cert management
process. *Public* disclosure is essential because Mozilla relies on
*this* public community for due diligence.

How is the accountability for their claims or Mozilla's wishes different
from last time?  Mozilla was evidently operating under the assumption
that Comodo was doing something that they were not doing.  You do not
propose any more disclosure than last time, so in that respect I don't
see how it is different this time.

On 3/24/11 5:39 PM, Steve Schultze wrote:

Even if they do give us the full list, how are we to know that they've
updated the processes and monitoring for all of their RAs?

I think the only feasible way is to rely on an audit. Perhaps we should
require Comodo to go through a comprehensive independent audit of all of
their RAs to prove that the updated practices and monitoring is in place
across all of their RAs.

Yes, I agree. What do you think of the idea of requiring an independent
audit of all of their RAs?

Kathleen

On 3/24/11 8:56 PM, Kathleen Wilson wrote:

Well, you'd have to require them to disclose the audit... thereby
disclosing the list.  Or did you envision an audit where the result was
just some auditor's attestation that all of the RAs were complying?

But what possible legitimate reason, from the perspective of users that
rely on these entities, could there be for not requiring disclosure?  At
times, there seems to be a posture on the part of the Moz cert team
against things that are thought to inconvenience the CAs.

I can imagine a variety of ways that public disclosure would help with
policing.  It would mean that people could conduct tests against the
full list to verify actual validation.  People could report RAs that do
not have disclosed audits. Etc...

Obviously from the above, I think that disclosure is necessary.  Audits
are probably also necessary.  How often would you propose that they
happen?  Must an audit be done of every RA before they begin operations?

However, I'm not sure that applying these policies against "RAs" is the
right approach.  Or maybe it is, depending on what the definition of
"RA" is.  According to WebTrust (http://www.webtrust.org/item27804.pdf):

"The CA or RA verifies the identity of the subscriber in accordance with
the CA’s established business practices (that may be contained in a
certification practice statement), and then issues a digital certificate."
  and
"An RA is an entity that is responsible for the identification and
authentication of subscribers, but does not sign or issue certificates."

So perhaps even WebTrust isn't consistent... does an RA issue certs?
Does it verify identity?  I think that any entity that has the
operational and/or cryptographic ability to issue certs (directly, or by
asserting identity to an entity that issues the certs) should be audited
and publicly disclosed.  Is that unreasonable?

How do we know that there haven't been dozens of similar compromises that
weren't caught?  This seems to be saying "lax practice are OK, we'll pretend
that they're not happening until forced by public disclosure to do something,
and even then it'll be nothing more than a slap on the wrist with a wet bus
ticket".  Even before the various public failures, Comodo had a bit of a
reputation in the PKI community for lax practices, to the extent that I'd
heard it referred to privately as "Commode-o" by other PKI people (a name that
I may have accidentally let slip into a public post in the past :-).  So there
appears to have been some unease about them for a long time, predating any of
the public disclosures of problems, but it was just ignored.  

What we have is an industry that claims to be selling trust, but that has
repeatedly demonstrated its untrustworthiness in a very public manner.  And
while other trust-based industries rely on "trust but verify", the browsers
just give us "trust and hope".  When the PKI fails, as it has repeatedly,
there's nothing else there to fall back on. If you read the discussion on
various web forums (admittedly not a representative survey, but the best we
have to go on at the moment), most people have pretty much zero confidence in
browser PKI, and very low confidence in browser vendors' seriousness about
protecting users from harm.  This is the real problem that browsers should be
looking at, not "how wet shall we make the bus ticket before we slap the CA's
wrist with it" but "how do we deal with the fact that our chosen (and only)
web security mechanism is repeatedly, and publicly, failing, and our only
response to date has been PKI-me-harder".

Peter.

On 03/24/2011 11:24 PM, From Kathleen Wilson:

Luckily they detected it before more damage could be done. Unfortunately
disclosure to the public has not been prompt by any party involved and
this should not have happened in this way. The reasons are obvious.

I believe that this is a major issue which clearly shows a failure
across the band. If Comodo would have implemented their domain control
validation process as claimed in that bug, this attempt would have been
most likely successfully foiled.

It gets me that this wasn't the case, that this hasn't been made the
declared policy and that this hasn't been audited accordingly! More than
that, if we can't rely on what a CA claims, we have a far bigger issue
than nine fraudulent certs, seriously.

It is very clear to me that no system is 100% secure all the time -
obviously  an RA somewhere out there is more likely to fail than the CA
with its resources. It's however not understandable to me that no
sincere best effort has been made since the many issues preceding this one.

It's not understandable that the API for RAs hasn't been requiring the
DV process at the CA side as promised.

It's not understandable that no better measures for authentication
against the API have been implemented.

It's not understandable that Comodo wasn't able to implement a lousy
flagging system for the top 100 targets which would be most likely
requested in such an event.

There are CAs that by choice do NOT work with such RAs - with the
expected result of lesser revenues. There are CAs that have strict
access controls to the CA systems, to those parts that can issue a
certificate - with the expected inconvenience and added costs. There are
CAs that have multiple controls in place in order to detect fraudulent
attempts should a particular lawyer fail in the chain of events and
infrastructure - with the expected result of higher development costs
and operating resources and man-power required.

But what is it worth all that, if there are those that simple don't care
or are incapable to care?!?! And Comodo might be not alone with that.

Even though this should be really the last defense of Mozilla, some
action must happen that clearly makes a point. It will be probably not
an easy decision on your part and most likely you won't make everybody
happy. But something got to happen.

It might be that those certificates would have surfaced at some point
anyway, so better this.

Nope, this works for some CAs, for others it simply doesn't, period. The
screws must be tightened here and now.

I have been saying this for years, back at the time preceding EV already
(probably arguing with Frank).

The same goes for outsourcing domain control validation.

The same goes for unrestricted externally operated intermediate CAs.

Requiring RAs to be channeled through dedicated intermediate CAs is in
my opinion only partly helpful in case of a compromise - the issuance
processes must be improved, strictly, without mercy and conflicting
business interest.

--
Regards

Signer:  Eddy Nigg, StartCom Ltd.
XMPP:    start...@startcom.org
Blog:    http://blog.startcom.org/
Twitter: http://twitter.com/eddy_nigg

On 25 mar, 04:44, Peter Gutmann <pgut...@cs.auckland.ac.nz> wrote:
[...]

I agree that measures MUST be taken against recidivist liars, instead
of continuing to trust them.
Comodo clearly lied when they wrote that technical countermeasures had
been installed to prevent such problems.
I personally won't trust such a lier anymore.

On the other hand, I've heard arguments against the trust revocation
for some browsers (but not all), in the sense that such a situation
could cause loss of market share for those browsers. I can understand
this. But this means that the power relationship changes completely as
soon as a CA is included in a browser; before acceptance, a CA needs
its root to be deployed in order to sell certificates; after
acceptance, the browser needs to keep  the biggest number of roots to
keep their users happy. That's insane.
Could it be possible to temporarily disable all trust in Comodo
certificates, wait for them to be positively audited on the technical
modifications they might implement to prevent other human failures to
happen so easily (with a public disclosure of the audit report, of
course), and re-enable them after a public discussion?

About the "anti-X.509" buzz, I've also read a lot of such things,
comparing "evil money-based CA (X.509)" and "good PGP-like users", but
honestly (and sadly, yes), they haven't understood what the real
problem was. This has nothing to do with the chosen trust model, it
could have happened with a web-of-trust (and it still does happen,
today).

--
Erwann.

Maybe we just need to think of it differently from a UI perspective.  When you visit a suspected malware site, Firefox, Chrome, etc. popup up a nice nasty warning saying that the site probably has something awfully wrong with it, and you shouldn't visit it.  

Maybe in the cases of a site that has a certificate issued by a CA that is being de-listed, or has been de-listed, the browser shouldn't show normal certificate warnings, it should advertise something more about *why* it has made this decision.  In this case it is quite straightforward and a lot easier to explain than signature expiry, mismatch of CN/Hostname, etc.  It is that we don't trust a given CA that has been de-certified.

BTW, this isn't a statement saying what ought to be done with the current CA under discussion, just suggesting an approach UI-wise that might make browses more nimble.

Just a thought.

- Andy

On 25.03.2011 01:56, Kathleen Wilson wrote:

Yes! All RAs are effectively a CA, because they can issue certs, and
must be subject to the same scrutiny as CAs.

It's not acceptable that we all trust on a bunch of companies, and
Comodo refuses to even say who they are. (Even in an event like this,
they still refuse to say who is the offender.)

I understand their business model relies on hundreds of resellers.
There's no problem with resellers. There's a problem with RAs, third
party entities which perform verification, i.e. a core function of the
CA. That must stop. IMHO, the whole concept of RAs is directly against
our policy, because they are notpart of the audit (as you say), neither
are they are not approved by us (which is what would still be missing).

A policy which says "I can let anyone in the world do the verification
for me" is obviously not very useful, but that's preciselywhat their
policy is.

Again, I have no problem with resellers, as long as Comodo does all the
CA functions, including all verifications, itself.

Ben

On 25.03.2011 02:48, Steve Schultze wrote:

Exactly.

No, it's not unreasonable, but the idea of the Mozilla CA policy. It was
just circumvented here. I don't think we should retrospectively
legitimize it.

Ben

On 25.03.2011 01:08, Kathleen Wilson wrote:

I would like to replace this entire section with:
"RAs and sub-CAs must not exist. All verifications and issuance happen
by the CA itself and cannot be delegated or outsourced. All persons and
machinery that is able to issue certs in any way must be subject to and
part of the audit, physically."

In view of the current section, this change may sound extreme, but I
believe it is justified for the reasons above, and anything else would
simply circumvent the whole policy.

Otherwise I can make a Über-CA, write in my policy that I delegate RAs
as I wish, and then every new CA doesn't have to go through our policy
and approval, but just pays me $$.
Exactly that seems to be the business model of Comodo, and I claim it's
in direct contradiction of the idea of our policy.

Ben

On 03/25/2011 03:55 PM, From Erwann Abalea:

Thanks for bringing this up in this way. I believe too that it's
unacceptable that we have CAs of new root inclusion requests jump
through extensive scrutiny, modifications of policies, re-auditing and
so forth - whereas there are some CA roots included that have repeating
failures without any review of the relevant policies, public discussions
and attestations of requested corrective measures.

If the same equation for existing roots is applied as for new ones, this
should be the correct action at this time. But Mozilla would not be able
to do it on its own without risking a degradation of its own product.

--
Regards

Signer:  Eddy Nigg, StartCom Ltd.
XMPP:    start...@startcom.org
Blog:    http://blog.startcom.org/
Twitter: http://twitter.com/eddy_nigg

Kathleen,
        I apologize for not responding more quickly to your questions.
I will dedicate some time at the start of next week to respond properly.

Best regards
Robin Alden
Comodo

On 3/25/11 11:11 AM, Eddy Nigg wrote [in part]:

However, the Mozilla policy has been updated to require periodic audits.
 I assume someone in Mozilla will be looking at the audit reports.

--

David E. Ross
<http://www.rossde.com/>

On occasion, I might filter and ignore all newsgroup messages
posted through GoogleGroups via Google's G2/1.0 user agent
because of spam from that source.

On 3/25/11 8:23 PM, David E. Ross wrote:

It doesn't appear that this is done comprehensively, or that not
providing an updated audit is grounds for removal.  For example, the
spreadsheet shows the most recent CNNIC audit as being August 2009:

http://spreadsheets.google.com/pub?key=ttwCVzDVuWzZYaDosdU6e3w&single...
  https://cert.webtrust.org/SealFile?seal=935&file=pdf

The Moz Cert Policy 2.0 says that "CAs are expected to maintain the
level of service that was established in the Inclusion Section of the
Mozilla CA Certificate Policy."

http://www.mozilla.org/projects/security/certs/policy/MaintenancePoli...

However, there doesn't seem to be any mechanism for active policing of
this expectation, other than yearly audit updates. Of course, the only
output of the audit updates is the two-page attestation from an auditor
that a new audit was done... none of the details from the actual report.
  And of course, the audit requirement is only one of many requirements
for acceptance.
  http://www.mozilla.org/projects/security/certs/policy/InclusionPolicy...

In addition, Kathleen has said:
"We don't usually go back and retroactively apply root inclusion process
changes to roots. The exception being that we have started to require
regularly updated audit documents for all included root certificate
authorities."

http://groups.google.com/group/mozilla.dev.security.policy/browse_thr...

So I believe that the original critique stands -- there is a de-facto
difference between the requirements (and the incentives on CAs) for
acceptance vs maintenance.  There is even more difference between
recently approved roots and less recently approved roots.

Steve

On 03/26/2011 03:35 AM, From Steve Schultze:

That would be fine if we'd have the ability to monitor the accompanying
CA policies (and changes thereof). I believe it would be impossible for
this group here to maintain that - however it will become easier for us
(and Kathleen) to handle once the Basic Guidelines have been adopted and
incorporated into the Mozilla CA Policy requirements.

This will require to look at two place - compliance to the basic
guidelines and attestation by audit. Possible improvements to the
guidelines will be maintained by the CAB Forum, a process that has
already kind of started. I believe this tool will become very important
for all of us.

--
Regards

Signer:  Eddy Nigg, StartCom Ltd.
XMPP:    start...@startcom.org
Blog:    http://blog.startcom.org/
Twitter: http://twitter.com/eddy_nigg

On 3/25/11 10:16 PM, Eddy Nigg wrote:

So in summary, "we can't actually enforce the expectations we just wrote
into the new guidelines."

It's also not just a matter of CA policies... it's a matter of
practices.  However, without disclosure of things like SubCA's and RA's,
it's impossible for anyone to even try in a comprehensive way.

The CAB Forum doesn't own the requirements on CA's.  The browsers and OS
vendors do.  If they wish to adopt the CAB Forum guidelines, they can.

If the CAB Forum would like to begin conducting its business in a more
public way, this outcome might be more likely.

Kathleen Wilson <kathleen95...@yahoo.com> wrote on Thu, 24 Mar 2011 14:24:24 -0700
 >
 > On 3/23/11 6:07 PM, Eddy Nigg wrote:
 >> On 03/23/2011 09:16 PM, From Steve Schultze:
 >>> Many of you are probably already following this story.
 >>
 >> ....and we are waiting patiently for a posting by a Mozilla
 >> representative here.
 >>
<snip/>
 > Mozilla requested that Comodo take the following actions.
 >

The legitimate certs of two of the targeted webapps are apparently EV certs:
<https://login.live.com/> and <https://addons.mozilla.org/>.

The generated impostor certs were not EV -- but /could/ they have been? I.e. is
Comodo (and other CAs) facilitating their RAs to issue EV certs?  Could the
compromised RA user account have actually issued an EV certs for login.live.com
and addons.mozilla.org ?

If Comodo is facilitating EV issuance by any of their RAs, are they following
(where their RAs are concerned) the procedures in Sections 12.2 (and section 12
in general) and section 10.12 of <http://cabforum.org/Guidelines_v1_3.pdf> ?

Perhaps these are questions to add to the list for them to answer.

thanks,

=JeffH

On 03/26/2011 04:23 AM, From Steve Schultze:

Of course this auditors have to confirm - failing that, they fail too.

One doesn't exclude the other obviously. Luckily we are free to add our
own requirements :-)

No, it's a group that can (and does) propose the guidelines.

Obviously. And of course I'd urge exactly that to do once it becomes
relevant.

--
Regards

Signer:  Eddy Nigg, StartCom Ltd.
XMPP:    start...@startcom.org
Blog:    http://blog.startcom.org/
Twitter: http://twitter.com/eddy_nigg