DigiNotar EV and Auditor Accountability

[Stephen Schultze]

Did DigiNotar have a current EV audit document at the time of the hack?
There is no audit listed in Kathleen's BuiltInCAs spreadsheet.

Was DigiNotar EV-enabled at the time of the hack? It seems that it was:
https://bugzilla.mozilla.org/show_bug.cgi?id=682927#c48

Presuming an updated WebTrust EV audit from PriceWaterhouseCoopers
exists and is essentially the same as the outdated one listed on the
DigiNotar site and found in the DigiNotar EV-enablement bug (369357),
will PriceWaterhouseCoopers be asked to answer how they mistakenly
reported that DigiNotar:

"maintained effective controls to provide reasonable assurance that:
- EV Subscriber information is properly collected, authenticated (for
the registration activities performed by DigiNotar) and verified;
- The integrity of keys and EV certificates it manages is established
and protected throughout their life cycles"

Will Mozilla continue to accept audit reports from PriceWaterhouseCoopers?

[Kathleen Wilson]

On 9/15/11 12:06 PM, Stephen Schultze wrote:
> Did DigiNotar have a current EV audit document at the time of the hack?
> There is no audit listed in Kathleen's BuiltInCAs spreadsheet.
>

Oops. Fixed -- I had removed DigiNotar from the spreadsheet, but it was
noted that I should keep it there, just marked differently. When I put
it back I forgot to copy the audit into both slots.

The ETSI certificate is for both...
http://www.diginotar.nl/LinkClick.aspx?fileticket=ARFojxrOqKY%3d&tabid=1259
"DigiNotar B.V. located at Beverwijk, The Netherlands, complies with the
requirements of:
- qualified certificate policy QCP+ specified in ETSI TS 101 456 (v. 1.4.3)
- normalized certificate polices NCP+, EV specified in ETSI TS 102 042
(v. 2.1.2)"

> Was DigiNotar EV-enabled at the time of the hack? It seems that it was:
> https://bugzilla.mozilla.org/show_bug.cgi?id=682927#c48
>

Yes.

> Presuming an updated WebTrust EV audit from PriceWaterhouseCoopers
> exists and is essentially the same as the outdated one listed on the
> DigiNotar site and found in the DigiNotar EV-enablement bug (369357),
> will PriceWaterhouseCoopers be asked to answer how they mistakenly
> reported that DigiNotar:
>
> "maintained effective controls to provide reasonable assurance that:
> - EV Subscriber information is properly collected, authenticated (for
> the registration activities performed by DigiNotar) and verified;
> - The integrity of keys and EV certificates it manages is established
> and protected throughout their life cycles"
>

The date of issuance of the new ETSI certificate is November 1, 2010.

> Will Mozilla continue to accept audit reports from PriceWaterhouseCoopers?

TBD


Kathleen

[Ian G]

On 16/09/11 7:38 AM, Kathleen Wilson wrote:
> On 9/15/11 12:06 PM, Stephen Schultze wrote:

>> Did DigiNotar have a current EV audit document at the time of the hack?
>> There is no audit listed in Kathleen's BuiltInCAs spreadsheet.
>>
>
>

> Oops. Fixed -- I had removed DigiNotar from the spreadsheet, but it was
> noted that I should keep it there, just marked differently. When I put
> it back I forgot to copy the audit into both slots.
>
> The ETSI certificate is for both...
> http://www.diginotar.nl/LinkClick.aspx?fileticket=ARFojxrOqKY%3d&tabid=1259
> "DigiNotar B.V. located at Beverwijk, The Netherlands, complies with the
> requirements of:
> - qualified certificate policy QCP+ specified in ETSI TS 101 456 (v. 1.4.3)
> - normalized certificate polices NCP+, EV specified in ETSI TS 102 042
> (v. 2.1.2)"

>> Was DigiNotar EV-enabled at the time of the hack? It seems that it was:
>> https://bugzilla.mozilla.org/show_bug.cgi?id=682927#c48
>>
>

> Yes.

My understanding was that in addition to the ETSI or standard WebTrust
there had to be an EV-specific audit conducted in order for EV to be
enabled. Which is this? Or is my understanding wrong?

That is, the two ETSIs above are not EV audits, right?

>> Presuming an updated WebTrust EV audit from PriceWaterhouseCoopers
>> exists and is essentially the same as the outdated one listed on the
>> DigiNotar site and found in the DigiNotar EV-enablement bug (369357),
>> will PriceWaterhouseCoopers be asked to answer how they mistakenly
>> reported that DigiNotar:
>>
>> "maintained effective controls to provide reasonable assurance that:
>> - EV Subscriber information is properly collected, authenticated (for
>> the registration activities performed by DigiNotar) and verified;
>> - The integrity of keys and EV certificates it manages is established
>> and protected throughout their life cycles"
>>
>
>

> The date of issuance of the new ETSI certificate is November 1, 2010.
>
>

>> Will Mozilla continue to accept audit reports from
>> PriceWaterhouseCoopers?
>
>

> TBD


Hmmm... Quis custodiet ipsos custodes? This issue has always been
present, and frequently commented upon.

Part of the issue is that we haven't a clear idea on what we expect the
audit to cover us for. Oh, yes, we say we must have an audit, but that
is customary, not goal-oriented. E.g., what statement can we make, now
that we have valid and current (i.e., in the last year) audits over
DigiNotar? And, how does that statement compare with our new
information that suggests that the integrity protections, while
effective at some level as evidenced by the audit(s), didn't work in the
event of the Iranian chap?

Another issue is that the audit is fundamentally risk-based, whereas our
thinking and model is fundamentally binary or absolutist. These clash,
and one of them must change. As audits will always be risk-based by
their very nature ... it's us that might have to change :)

I can see lots of ways for us to analyse this issue, but I can't see any
successful ways, ones that improve our situation, help our users. E.g.,
we can discuss it on this group, but we won't get anywhere. E.g.2.,
Mozilla could have some private discussions, but we've also seen that
this results in unclear results.

So, unhappily, I'd have to say that Steve's strawman proposal to drop
PriceWaterhouseCoopers has merit. We just don't have any other way of
credibility to respond.



iang

[Bernd Kirsig]

> On 16/09/11 7:38 AM, Kathleen Wilson wrote:
> > On 9/15/11 12:06 PM, Stephen Schultze wrote:

> >> Did DigiNotar have a current EV audit document at the time
> of the hack?
> >> There is no audit listed in Kathleen's BuiltInCAs spreadsheet.
> >>
> >
> >

> > Oops. Fixed -- I had removed DigiNotar from the spreadsheet, but it
> > was noted that I should keep it there, just marked
> differently. When I
> > put it back I forgot to copy the audit into both slots.
> >
> > The ETSI certificate is for both...
> >
> http://www.diginotar.nl/LinkClick.aspx?fileticket=ARFojxrOqKY%3d&tabid
> > =1259 "DigiNotar B.V. located at Beverwijk, The
> Netherlands, complies
> > with the requirements of:
> > - qualified certificate policy QCP+ specified in ETSI TS
> 101 456 (v.
> > 1.4.3)
> > - normalized certificate polices NCP+, EV specified in ETSI
> TS 102 042
> > (v. 2.1.2)"
>

> >> Was DigiNotar EV-enabled at the time of the hack? It seems
> that it was:
> >> https://bugzilla.mozilla.org/show_bug.cgi?id=682927#c48
> >>
> >

> > Yes.
>
> My understanding was that in addition to the ETSI or standard
> WebTrust there had to be an EV-specific audit conducted in
> order for EV to be enabled. Which is this? Or is my
> understanding wrong?
>
> That is, the two ETSIs above are not EV audits, right?

ETSI TS 102 042 (v. 2.1.2) is somewhat modular.
Options for EV are included.

Bernd

Bernd Kirsig Tel.: +49 (0)40 / 80 80 26-2 51
Sr. Princ. IT Security Officer Fax.: +49 (0)40 / 80 80 26-1 26
TC TrustCenter GmbH - Now part of Symantec
Sonninstraße 24-28 mailto: kir...@trustcenter.de
D-20097 Hamburg http://www.trustcenter.de
Geschaeftsfuehrung/Managing Directors:
Austin McCabe, Norman Osumi
AG Hamburg, HRB 96168

>
> >> Presuming an updated WebTrust EV audit from PriceWaterhouseCoopers
> >> exists and is essentially the same as the outdated one
> listed on the
> >> DigiNotar site and found in the DigiNotar EV-enablement
> bug (369357),
> >> will PriceWaterhouseCoopers be asked to answer how they mistakenly
> >> reported that DigiNotar:
> >>
> >> "maintained effective controls to provide reasonable
> assurance that:
> >> - EV Subscriber information is properly collected,
> authenticated (for
> >> the registration activities performed by DigiNotar) and verified;
> >> - The integrity of keys and EV certificates it manages is
> established
> >> and protected throughout their life cycles"
> >>
> >
> >

> > The date of issuance of the new ETSI certificate is
> November 1, 2010.
> >
> >

> >> Will Mozilla continue to accept audit reports from
> >> PriceWaterhouseCoopers?
> >
> >

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

[Jean-Marc Desperrier]

Ian G wrote:
> I'd have to say that Steve's strawman proposal to drop
> PriceWaterhouseCoopers has merit

The first step is to establish whether PWC has failed to realize a
proper audit or not.

There's basically three different scenario :

1) It's actually possible to have a perfectly successful audit with some
major failures in your architecture, and neither Diginotar nor PWC
actually cheated the audit => We'd need to identify what's missing in
the audit content to be sure to identify such failures, and then update
them so that it doesn't happen again

2) DigiNotar lied/hid some elements to PWC => The hardest case. If a
minimal level of honesty from the auditee can't be assumed, it's very
hard for the auditor to do a proper job. Solution : Hold DigiNotar
liable in penal for lying to PWC ? Make sure what at stake is strong
enough that the auditee is unlikely to hid something significant ?

3) PWC indeed did a bad job at the auditing. That's the case where they
are to held responsible. But Mozilla needs in a first step to talk with
them to establish where exactly they failed, and then yes one of
consequence could very well be to stop accepting CA's audits by PWC.

[Ian G]

On 16/09/11 11:30 PM, Jean-Marc Desperrier wrote:

> Ian G wrote:
>> I'd have to say that Steve's strawman proposal to drop

>> PriceWaterhouseCoopers has merit
>
> The first step is to establish whether PWC has failed to realize a
> proper audit or not.
>
> There's basically three different scenario :
>

> ...But Mozilla needs in a first step to talk with

> them to establish where exactly they failed,

Right. But they are the experts. We/Mozilla is not. So they will
simply tell us some story. We won't be able to crack that story [0].

So the first problem is that we won't even begin to find out what went
really wrong. The only way to do that is to do a full independent
review. An audit over audit, in effect.

Can we trust the result? Who's going to do that? This path has three
possibilities that I see:

1. by us, in which case we don't have the expertise (and not the budget).
2. By another audit firm, in which case we have a conflict of
interest, as the auditors will work together to reduce damage to the
industry.
3. By NL government, which has a much wider problem than we do, being
the collapsor of a QC supplier that covered a big proportion of its
government sites. But they also have different interests and
responsibities, so their audit might not be directly helpful either.



Which leaves us with a quality problem. This is actually the same
problem we've had all along: we can do an audit over a CA (and in this
case an audit over an audit). But, we can't interpret the results to
any easy or reliable degree. So we don't know whether we're wasting our
money or not.

The DigiNotar case (and the cases of the other 4 or so hacked CAs)
indicate that we may be wasting our money. In a sense, how much money
do you want to spend to discover we might be wasting all our money?



iang




[0] I can to some extent speculate what that story will be. The first
claim would be that the tests were done to a reasonable standard, and
the events fitted in with that. Second, the threat scenario changed
over the year, and next year's audit would have reflected that. Then,
in the audit process, there is some risk of a false statement being
made. This is just one of those events. Another, that some unique
factors caused something to be missed. That's fixed now, rest easy.
More, the criteria aren't specific enough. Plus, ... how long to I have
to do this before you'll stop asking questions?

[Stephen Schultze]

On 9/16/11 1:14 PM, Ian G wrote:
> On 16/09/11 11:30 PM, Jean-Marc Desperrier wrote:
>> Ian G wrote:
>>> I'd have to say that Steve's strawman proposal to drop
>>> PriceWaterhouseCoopers has merit
>>
>> The first step is to establish whether PWC has failed to realize a
>> proper audit or not.
>>
>

> Right. But they are the experts. We/Mozilla is not. So they will simply
> tell us some story. We won't be able to crack that story [0].

It makes a person wonder whether requiring disclosure of more of the
actual audit documents, rather than a one-page pro-forma statement,
might have been a good policy idea.

[Jean-Marc Desperrier]

Stephen Schultze wrote:
> It makes a person wonder whether requiring disclosure of more of the
> actual audit documents, rather than a one-page pro-forma statement,
> might have been a good policy idea.

I think we're here typically a situation where Mozilla can require more
complete information about what happened during the audit than it had
initially, and request disclosure of the actual audit documents.

[Stephen Schultze]

Yes, but "an ounce of prevention" and all that...

And, in any case, Mozilla "can require" anything it wishes.

[Jean-Marc Desperrier]

Stephen Schultze wrote:
> And, in any case, Mozilla "can require" anything it wishes.

And it can also set any rules it wants about what trusted auditors are
accepted for root CA inclusions request.

Including defining as a trusted auditor only an auditor that shows a
cooperating and constructive attitude when some specific circumstances
rise the need for more complete information about a specific audit, and
the disclosure of some documents that were not required initially.

[ianG]

On 17/09/11 05:25 AM, Stephen Schultze wrote:
> On 9/16/11 1:14 PM, Ian G wrote:

>> On 16/09/11 11:30 PM, Jean-Marc Desperrier wrote:
>>> Ian G wrote:
>>>> I'd have to say that Steve's strawman proposal to drop
>>>> PriceWaterhouseCoopers has merit
>>>
>>> The first step is to establish whether PWC has failed to realize a
>>> proper audit or not.
>>>
>>

>> Right. But they are the experts. We/Mozilla is not. So they will simply
>> tell us some story. We won't be able to crack that story [0].
>

> It makes a person wonder whether requiring disclosure of more of the
> actual audit documents, rather than a one-page pro-forma statement,
> might have been a good policy idea.

Having worked as an auditor, and having reviewed many of the files of
the AICPA as well as some small amount of academic criticism of the
field, I can suggest that auditors really do know what they are doing.
They have erected an impenetrable barrier for outsiders. (Which is to
stress my words above, they are the experts, we are not.)

In other words, no chance: auditors are wayyyyyyyy ahead of you in that
game.

The only people who have any chance at all of reviewing an/the audit are
auditors [1]. And they're all in the game. So that's protected too.

The only way a group like Mozilla would be able to crack that barrier
and influence or even understand what was going on in audit would be if
there were clear evidence of egregious behaviour. I'm thinking here
about the Madoff case or the Satyam case, not the DigiNotar case which
is unlikely to be egregious. In the case that an auditor didn't do a
good job, or simply missed something that collapsed spectacularly
(different things really), there is little chance for Mozilla to
influence that in advance, or even after the fact. You can collect all
the documents you want, do all the reviews you want, form more secret
committees and cartels with them, ... you won't get close.

You might feel better about it. You might even meet your legal needs
for due diligence or compliance [2]. You might impress the users, and
make CAs happy... But you'll never ever be able to achieve a positive
benefit of influence or quality by engaging directly with the process.

The only way to influence the result, IMHO, is using methods external to
the process. Think outside the box.

iang



[1] hence my suggestion that the NL government regulator should be
watched in this case. They may call in the government auditors to give
an accounting. If they do this, there is a chance that the government
audit body will be sufficiently separated from the commercial auditor
concerned, and the failure complete enough, such that an unbiased review
be conducted. If you wanted to influence events, this would be the way
in: contact the regulator of CAs in NL and ask for their government
auditors to go in. This would be like the GAO for americans.

[2] liability shifting, iow.

[Lucas Adamski]

There are enough experts to go around, and security auditing is not some sort of black magic. Problem is too many auditors (and security products) seek to dig just deep enough to provide the appearance of a thorough audit, without digging so deep as to cause serious "problems".

That said, there might be another approach here, which is to hold auditors accountable for the results of an audit. Which is to say if a CA passes an audit then gets compromised... we could stop accepting audits from said auditor. Maybe suspend them for some period of time (6 months) for the first incident, 12 for the second, life for third. Then maybe they will care about more than getting repeat business.

I do think we should receive a copy of the last audit for any compromised CA's.
Lucas.

On Sep 20, 2011, at 6:38 AM, ianG wrote:

> On 17/09/11 05:25 AM, Stephen Schultze wrote:
>> On 9/16/11 1:14 PM, Ian G wrote:

>>> On 16/09/11 11:30 PM, Jean-Marc Desperrier wrote:
>>>> Ian G wrote:
>>>>> I'd have to say that Steve's strawman proposal to drop
>>>>> PriceWaterhouseCoopers has merit
>>>>
>>>> The first step is to establish whether PWC has failed to realize a
>>>> proper audit or not.
>>>>
>>>

>>> Right. But they are the experts. We/Mozilla is not. So they will simply
>>> tell us some story. We won't be able to crack that story [0].
>>

>> It makes a person wonder whether requiring disclosure of more of the actual audit documents, rather than a one-page pro-forma statement, might have been a good policy idea.
>
> Having worked as an auditor, and having reviewed many of the files of the AICPA as well as some small amount of academic criticism of the field, I can suggest that auditors really do know what they are doing. They have erected an impenetrable barrier for outsiders. (Which is to stress my words above, they are the experts, we are not.)
>
> In other words, no chance: auditors are wayyyyyyyy ahead of you in that game.
>
> The only people who have any chance at all of reviewing an/the audit are auditors [1]. And they're all in the game. So that's protected too.
>
> The only way a group like Mozilla would be able to crack that barrier and influence or even understand what was going on in audit would be if there were clear evidence of egregious behaviour. I'm thinking here about the Madoff case or the Satyam case, not the DigiNotar case which is unlikely to be egregious. In the case that an auditor didn't do a good job, or simply missed something that collapsed spectacularly (different things really), there is little chance for Mozilla to influence that in advance, or even after the fact. You can collect all the documents you want, do all the reviews you want, form more secret committees and cartels with them, ... you won't get close.
>
> You might feel better about it. You might even meet your legal needs for due diligence or compliance [2]. You might impress the users, and make CAs happy... But you'll never ever be able to achieve a positive benefit of influence or quality by engaging directly with the process.
>
> The only way to influence the result, IMHO, is using methods external to the process. Think outside the box.
>
> iang
>
>
>
> [1] hence my suggestion that the NL government regulator should be watched in this case. They may call in the government auditors to give an accounting. If they do this, there is a chance that the government audit body will be sufficiently separated from the commercial auditor concerned, and the failure complete enough, such that an unbiased review be conducted. If you wanted to influence events, this would be the way in: contact the regulator of CAs in NL and ask for their government auditors to go in. This would be like the GAO for americans.
>
> [2] liability shifting, iow.

[Stephen Schultze]

On 9/20/11 9:38 AM, ianG wrote:
> On 17/09/11 05:25 AM, Stephen Schultze wrote:
>> On 9/16/11 1:14 PM, Ian G wrote:
>>> On 16/09/11 11:30 PM, Jean-Marc Desperrier wrote:
>>>> Ian G wrote:
>>>>> I'd have to say that Steve's strawman proposal to drop
>>>>> PriceWaterhouseCoopers has merit
>>>>
>>>> The first step is to establish whether PWC has failed to realize a
>>>> proper audit or not.
>>>>
>>>
>>> Right. But they are the experts. We/Mozilla is not. So they will simply
>>> tell us some story. We won't be able to crack that story [0].
>>
>> It makes a person wonder whether requiring disclosure of more of the
>> actual audit documents, rather than a one-page pro-forma statement,
>> might have been a good policy idea.
>

> The only people who have any chance at all of reviewing an/the audit are
> auditors [1]. And they're all in the game. So that's protected too.

This is a rather naive attitude in light of well-known historical
failures of security by obscurity and the corresponding success of
disclosure-based security. (Which is, incidentally, the principle that
Mozilla embraces.)

[ianG]

On 21/09/11 05:14 AM, Stephen Schultze wrote:
> On 9/20/11 9:38 AM, ianG wrote:

>> The only people who have any chance at all of reviewing an/the audit are
>> auditors [1]. And they're all in the game. So that's protected too.
>

> This is a rather naive attitude

Well, I suggest you try it. I have.

Here's an easy way forward. Find the audit report and look in there to
find the clues.

E.g., Comodo, about 2 years back there was a failure. And then we
looked in the Audit report. And it was written in black & white. But
nobody saw it before hand, and nobody did anything about it.

Does anyone have DigiNotar's report to hand?

> in light of well-known historical failures of security by obscurity
> and the corresponding success of disclosure-based security. (Which
> is, incidentally, the principle that Mozilla embraces.)

IMO, that's naive :) Skype is a counter-example of security by
obscurity that whips all competition. Security-by-openness is not well
correlated with security, although it is an intellectually appealing idea.

iang

[Stephen Schultze]

On Sep 20, 4:46 pm, ianG <i...@iang.org> wrote:
> On 21/09/11 05:14 AM, Stephen Schultze wrote:
>
> > On 9/20/11 9:38 AM, ianG wrote:
> >> The only people who have any chance at all of reviewing an/the audit are
> >> auditors [1].  And they're all in the game.  So that's protected too.
>
> > This is a rather naive attitude
>
> Well, I suggest you try it.  I have.
>
> Here's an easy way forward.  Find the audit report and look in there to
> find the clues.

Please do point us to one. I don't recall ever seeing one disclosed.

> E.g., Comodo, about 2 years back there was a failure.  And then we
> looked in the Audit report.  And it was written in black & white.  But
> nobody saw it before hand, and nobody did anything about it.

What was written in black & white?

Look, it's likely that there's not enough information in the audit
report for outsiders to catch all problems, but I don't buy the
premise that nobody but the auditors can understand the documents.

> Does anyone have DigiNotar's report to hand?

Huh? Who would other than DigiNotar and the auditor? That's the
point.

> > in light of well-known historical failures of security by obscurity
> > and the corresponding success of disclosure-based security.  (Which
> > is, incidentally, the principle that Mozilla embraces.)
>
> IMO, that's naive :)  Skype is a counter-example of security by
> obscurity that whips all competition.  Security-by-openness is not well
> correlated with security, although it is an intellectually appealing idea.

https://encrypted.google.com/search?q=tom-skype+surveillance
https://encrypted.google.com/search?q=skype+back+door

[ianG]

On 22/09/11 04:35 AM, Stephen Schultze wrote:
> On Sep 20, 4:46 pm, ianG<i...@iang.org> wrote:

>> On 21/09/11 05:14 AM, Stephen Schultze wrote:
>>
>>> On 9/20/11 9:38 AM, ianG wrote:
>>>> The only people who have any chance at all of reviewing an/the audit are
>>>> auditors [1]. And they're all in the game. So that's protected too.
>>> This is a rather naive attitude
>> Well, I suggest you try it. I have.
>>
>> Here's an easy way forward. Find the audit report and look in there to
>> find the clues.

> Please do point us to one. I don't recall ever seeing one disclosed.

http://www.mozilla.org/projects/security/certs/included/#Comodo

Curiously, DigiNotar's one isn't there. Kathleen, any ideas?

>> E.g., Comodo, about 2 years back there was a failure. And then we

>> looked in the Audit report. And it was written in black& white. But

>> nobody saw it before hand, and nobody did anything about it.

> What was written in black& white?

>
> Look, it's likely that there's not enough information in the audit
> report for outsiders to catch all problems, but I don't buy the
> premise that nobody but the auditors can understand the documents.

OK :)

>
>> Does anyone have DigiNotar's report to hand?

> Huh? Who would other than DigiNotar and the auditor? That's the
> point.

Are you thinking of an additional internal report? This is an optional
extra...

The Auditor provides an opinion on which some third party can rely.
That's the point of the Auditor, to provide an external attestation over
some statement to third party(s). In this case, it is customary for the
Auditor's opinion to be presented to vendors, and it is expected
(although I'm not sure it is stated) that relying parties can get it and
rely on it as well.

>>> in light of well-known historical failures of security by obscurity
>>> and the corresponding success of disclosure-based security. (Which
>>> is, incidentally, the principle that Mozilla embraces.)
>> IMO, that's naive :) Skype is a counter-example of security by
>> obscurity that whips all competition. Security-by-openness is not well
>> correlated with security, although it is an intellectually appealing idea.

> https://encrypted.google.com/search?q=tom-skype+surveillance
> https://encrypted.google.com/search?q=skype+back+door

Yeah. So, if you take *any* security system, and place a false or
perverted interface in front of the user, the security is then broken
totally. This is more a law of physics, in that the user cannot see
past the thing in front of him, and computers typically have no trusted
hardware to distinguish.

The existance of a special download for China is more or less a proof,
however perverse, that the Skype system is indeed secure. They had to
backdoor the client, they couldn't tap the traffic. Two confirming
evidence datums: the EADS interception kit for Skype was indeed exactly
that, replacing the client on the victim's machine; and the complaints
of the European intel agencies died down when the other intel agencies
told them how to do it.

As long as you're sure you have an untampered client, you're in good
condition. Now contrast that against any other system. Open source or
not :)

iang

[Stephen Schultze]

On 9/21/11 3:02 PM, ianG wrote:
>>> Here's an easy way forward. Find the audit report and look in there to
>>> find the clues.
>> Please do point us to one. I don't recall ever seeing one disclosed.
>
> http://www.mozilla.org/projects/security/certs/included/#Comodo
>
> Curiously, DigiNotar's one isn't there. Kathleen, any ideas?

Are you talking about the one or two-page auditor statement? If so,
look at the second message in this thread (by Kathleen).

>>> Does anyone have DigiNotar's report to hand?
>> Huh? Who would other than DigiNotar and the auditor? That's the
>> point.
>
> Are you thinking of an additional internal report? This is an optional
> extra...
>
> The Auditor provides an opinion on which some third party can rely.
> That's the point of the Auditor, to provide an external attestation over
> some statement to third party(s). In this case, it is customary for the
> Auditor's opinion to be presented to vendors, and it is expected
> (although I'm not sure it is stated) that relying parties can get it and
> rely on it as well.

I am talking about whatever report(s) exist other than the one or
two-page statement. That short statement communicates nothing other
than "this auditor confirmed that the CA has processes and procedures
set up consistent with their CPS". I don't know what the formal term is
for all of these different documents, but I am lead to believe that
there exists a more detailed report that is not customarily (or ever?)
disclosed to the public.

Perhaps you could give us a taxonomy of the standard documents.

>>>> in light of well-known historical failures of security by obscurity
>>>> and the corresponding success of disclosure-based security. (Which
>>>> is, incidentally, the principle that Mozilla embraces.)
>>> IMO, that's naive :) Skype is a counter-example of security by
>>> obscurity that whips all competition. Security-by-openness is not well
>>> correlated with security, although it is an intellectually appealing
>>> idea.
>> https://encrypted.google.com/search?q=tom-skype+surveillance
>> https://encrypted.google.com/search?q=skype+back+door
>
> Yeah. So, if you take *any* security system, and place a false or
> perverted interface in front of the user, the security is then broken
> totally. This is more a law of physics, in that the user cannot see past
> the thing in front of him, and computers typically have no trusted
> hardware to distinguish.
>
> The existance of a special download for China is more or less a proof,
> however perverse, that the Skype system is indeed secure. They had to
> backdoor the client, they couldn't tap the traffic. Two confirming
> evidence datums: the EADS interception kit for Skype was indeed exactly
> that, replacing the client on the victim's machine; and the complaints
> of the European intel agencies died down when the other intel agencies
> told them how to do it.
>
> As long as you're sure you have an untampered client, you're in good
> condition. Now contrast that against any other system. Open source or
> not :)

No, the lesson of this experience is that:
a) Non-disclosed systems are at most as trustworthy as the people who
control the systems (vendors, governments, attackers)
b) Non-disclosed systems are more vulnerable to longstanding 0-day
attacks because it is much harder to investigate them

I really don't want to re-hash the security-by-obscurity vs. disclosure
debate on this list, particularly because Mozilla itself has already
decided that disclosure (with very time-limited embargoes that are
consistent with the responsible disclosure ethos) is the way to go.

[Eddy Nigg]

On 09/21/2011 10:55 PM, From Stephen Schultze:

> I am talking about whatever report(s) exist other than the one or
> two-page statement.

Well, I haven't seen them either - ever. And such a nice report probably
doesn't exist, I think.

Rather auditors have the audit criterion and a bunch of published an
unpublished papers, computer files and internal procedures where they
note the compliance to the criteria, have it signed by the immediate
auditor, supervisors and so forth. As I know, there is lots of reviews
and rechecking going on behind the scenes. And I assume that those
documents we'll never see - neither me as a CA nor you as a relying
party. What we'll get in the end is the standard opinion letter.

Should have a criteria not be fulfilled, this is usually noted in that
report. I recall to have seen such a shortcoming in a audit report, but
I don't recall who it was.

--
Regards

Signer: Eddy Nigg, StartCom Ltd.
XMPP: star...@startcom.org
Blog: http://blog.startcom.org/
Twitter: http://twitter.com/eddy_nigg

[Bernd Kirsig]

> -----Ursprüngliche Nachricht-----
> Von:
> dev-security-policy-bounces+kirsig=trustce...@lists.mozill
> a.org
> [mailto:dev-security-policy-bounces+kirsig=trustcenter.de@list
> s.mozilla.org] Im Auftrag von Eddy Nigg
> Gesendet: Mittwoch, 21. September 2011 22:39
> An: mozilla-dev-s...@lists.mozilla.org
> Betreff: Re: DigiNotar EV and Auditor Accountability

>
> On 09/21/2011 10:55 PM, From Stephen Schultze:

> > I am talking about whatever report(s) exist other than the one or
> > two-page statement.
>

> Well, I haven't seen them either - ever. And such a nice
> report probably doesn't exist, I think.
>
> Rather auditors have the audit criterion and a bunch of
> published an unpublished papers, computer files and internal
> procedures where they note the compliance to the criteria,
> have it signed by the immediate auditor, supervisors and so
> forth. As I know, there is lots of reviews and rechecking
> going on behind the scenes. And I assume that those documents
> we'll never see - neither me as a CA nor you as a relying
> party. What we'll get in the end is the standard opinion letter.

For the ETSI audits performed at TC TrustCenter we get 50 to 80 pages audit
reports.
Seems to be different from WebTrust.

Bernd

Bernd Kirsig Tel.: +49 (0)40 / 80 80 26-2 51
Sr. Princ. IT Security Officer Fax.: +49 (0)40 / 80 80 26-1 26

TC TrustCenter GmbH – Now part of Symantec

Sonninstraße 24-28 mailto: kir...@trustcenter.de
D-20097 Hamburg http://www.trustcenter.de
Geschaeftsfuehrung/Managing Directors:
Austin McCabe, Norman Osumi
AG Hamburg, HRB 96168

>

> Should have a criteria not be fulfilled, this is usually
> noted in that report. I recall to have seen such a
> shortcoming in a audit report, but I don't recall who it was.
>
> --
> Regards
>
> Signer: Eddy Nigg, StartCom Ltd.
> XMPP: star...@startcom.org
> Blog: http://blog.startcom.org/
> Twitter: http://twitter.com/eddy_nigg
>

[ianG]

On 22/09/11 21:54 PM, Bernd Kirsig wrote:
> For the ETSI audits performed at TC TrustCenter we get 50 to 80 pages
> audit reports.

Ah, interesting. Are these sent to Mozilla? Are these published?

Presumably there is a DigiNotar one around somewhere... I wonder if
there is a measurable difference across the different ETSI reports?

> Seems to be different from WebTrust.

An understatement :)


iang

[Florian Weimer]

> The only people who have any chance at all of reviewing an/the audit
> are auditors [1]. And they're all in the game. So that's protected
> too.

Does this matter? Mozilla doesn't do anything about sub-CAs which do
not comply with its guidelines (in practice, they do, because they
violate their own CPS). As long as you ignore even formally documented
non-compliance, you don't really have to care about what auditors do or
don't.

It's also not very interesting to look at business processes. Mozilla
could demand that all certificates issued by CAs in the root program are
published in due course, just as domain names under gTLDs are published
and available in bulk form. This would help with detection of certain
process failures and is much easier to do than judging an audit. It
would also implicitly cover the RA side, which is currently completely
out of scope as far as I can tell based on the two-page audit reports.

--
Florian Weimer <fwe...@bfk.de>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99

[Eddy Nigg]

On 09/23/2011 10:07 AM, From Florian Weimer:

> Does this matter? Mozilla doesn't do anything about sub-CAs which do
> not comply with its guidelines (in practice, they do, because they
> violate their own CPS).

If you have such information, why don't you inform Mozilla or the list
about it?

> Mozilla
> could demand that all certificates issued by CAs in the root program are
> published in due course, just as domain names under gTLDs are published
> and available in bulk form.

Mozilla requested all cross-signed CAs and might also request all sub
CAs in the future. This is equal to what you refer as gTLD. However
requesting all issued certificates will probably not work, that's like
requesting all domains of a registrar.

[Florian Weimer]

* Eddy Nigg:

> On 09/23/2011 10:07 AM, From Florian Weimer:

>> Does this matter? Mozilla doesn't do anything about sub-CAs which do
>> not comply with its guidelines (in practice, they do, because they
>> violate their own CPS).
>

> If you have such information, why don't you inform Mozilla or the list
> about it?

I was told that sub-CAs were out of scope.

>> Mozilla could demand that all certificates issued by CAs in the root
>> program are published in due course, just as domain names under gTLDs
>> are published and available in bulk form.
>

> Mozilla requested all cross-signed CAs and might also request all sub
> CAs in the future. This is equal to what you refer as gTLD. However
> requesting all issued certificates will probably not work, that's like
> requesting all domains of a registrar.

ICANN's gTLD contracts require that the registry publishes the domain
names it creates (in bulk form), and that the registrar responsible is
readily identifiable over WHOIS. (Bulk WHOIS access is only a
theoretical option, only about one in ten registrars makes a convincing
effort at compliance, and decent coverage is neither administratively
nor financially feasible.)

Contrast this with the allegedly secure system of browser CAs, where the
CAs themselves operate under pseudonyms, do not publish the certificates
they issue, and do not identify the responsible registrar (which would
translate to the acting RA, I think).

If certificate data were available in bulk form, those who care
(operators of high-value service mostly, but also others) could download
it and analyze it for suspicious activity, just as they do with domain
names today.

[Eddy Nigg]

On 09/23/2011 12:22 PM, From Florian Weimer:

> If you have such information, why don't you inform Mozilla or the list
> about it?
> I was told that sub-CAs were out of scope.

Intermediate CA certificates are not included with NSS, but compliance
to the Mozilla CA Policy (and their own policies) applies in full
extended for the entire PKI. Therefore, if you have such information,
you should contact Kathleen or post to the list.

> (Bulk WHOIS access is only a
> theoretical option, only about one in ten registrars makes a convincing
> effort at compliance, and decent coverage is neither administratively
> nor financially feasible.)

Yeah, just wanted to ask about where I can retrieve all domains of a
particular TLD. So it's my understanding that this basically doesn't exist.

> Contrast this with the allegedly secure system of browser CAs, where the
> CAs themselves operate under pseudonyms

Which CA? I know some which have some real crap in their subject name,
but I believe this will be a thing of the past very soon when the basic
requirements guidelines will be adopted by the software vendors. Also
the Mozilla CA policy might have something to say about this.

> If certificate data were available in bulk form, those who care
> (operators of high-value service mostly, but also others) could download
> it and analyze it for suspicious activity, just as they do with domain
> names today.

You can check revocation of a certificate, similar in the way you can
check the WHOIS records if a particular domain if you know its name. But
you have not provided evidence that domain names can be obtained from
domain name registrars. Also WHOIS lookups are severally limited with
many registrars.

As such, domain name registrars and certificate authorities are
different animals, I'm not sure if it's correct to make the equation you
do. I believe that due to competition and privacy concerns, CA will
probably not agree to disclose their entire customer base.

It also shouldn't be your task to measure the performances of CAs I
guess in such a way, even though many different groups and entities are
searching the net for certificates for various purposes and from time to
time questionable certificates are discovered.

[Florian Weimer]

* Eddy Nigg:

> On 09/23/2011 12:22 PM, From Florian Weimer:

>> If you have such information, why don't you inform Mozilla or the
>> list about it?
>> I was told that sub-CAs were out of scope.
>

> Intermediate CA certificates are not included with NSS, but compliance
> to the Mozilla CA Policy (and their own policies) applies in full
> extended for the entire PKI. Therefore, if you have such information,
> you should contact Kathleen or post to the list.

Uhm. Current Webtrust audits only cover internal RAs and do not extend
to sub-CAs. If Mozilla no longer wants to ignore sub-CAs and external
RAs, the Webtrust audits do not provide sufficient value. If a cut-off
date has been set for new policies, I have missed it.

>> (Bulk WHOIS access is only a
>> theoretical option, only about one in ten registrars makes a convincing
>> effort at compliance, and decent coverage is neither administratively
>> nor financially feasible.)
>

> Yeah, just wanted to ask about where I can retrieve all domains of a
> particular TLD. So it's my understanding that this basically doesn't
> exist.

It's called "zone file access program" and exists for the majority of
gTLDs (the "g" is important, ccTLDs are different).

>> Contrast this with the allegedly secure system of browser CAs, where the
>> CAs themselves operate under pseudonyms
>

> Which CA?

"Equifax Secure CA", "AddTrust Low-Value Services Root", and a couple of
others.

>> If certificate data were available in bulk form, those who care
>> (operators of high-value service mostly, but also others) could download
>> it and analyze it for suspicious activity, just as they do with domain
>> names today.
>

> You can check revocation of a certificate, similar in the way you can
> check the WHOIS records if a particular domain if you know its
> name. But you have not provided evidence that domain names can be
> obtained from domain name registrars.

You participate in the zone file access program (which is offered by
most gTLDs, per ICANN requirements), which gives you the entire list.
For the subset you're interested in, you query the TLD WHOIS server and
potentially follow referrals.

> Also WHOIS lookups are severally limited with many registrars.

True, but at least *some* data is published.

> As such, domain name registrars and certificate authorities are
> different animals, I'm not sure if it's correct to make the equation
> you do. I believe that due to competition and privacy concerns, CA
> will probably not agree to disclose their entire customer base.

But they have to, otherwise it is impossible to detect fraudulent
certificates before much damage is done. Of course this will hurt
business, but that's inevitable.

[Eddy Nigg]

On 09/23/2011 01:43 PM, From Florian Weimer:

> If Mozilla no longer wants to ignore sub-CAs and external RAs, the
> Webtrust audits do not provide sufficient value.

Compliance to the Mozilla CA Policy doesn't stop at the CA root. Rest be
assured that we recognized this both here and at the CAB Forum.

> It's called "zone file access program" and exists for the majority of
> gTLDs (the "g" is important, ccTLDs are different).

There is always something to learn - can you teach me how to do that?

> "Equifax Secure CA", "AddTrust Low-Value Services Root", and a couple of
> others.

Yeah, but those are harmless... there is worse, but I don't want to
embarrass anybody.

> But they have to, otherwise it is impossible to detect fraudulent
> certificates before much damage is done.

Well, there shouldn't be any to start with and it would be an entirely
different approach to the compliance audits we have today. But who
knows, maybe things will change :-)

[ianG]

On 23/09/11 17:07 PM, Florian Weimer wrote:
>> The only people who have any chance at all of reviewing an/the audit
>> are auditors [1]. And they're all in the game. So that's protected
>> too.

> Does this matter? Mozilla doesn't do anything about sub-CAs which do
> not comply with its guidelines (in practice, they do, because they

> violate their own CPS). As long as you ignore even formally documented
> non-compliance, you don't really have to care about what auditors do or
> don't.

Well, right. That's an issue. There is a long standing loophole in the
procedures that allowed the sub CA to effectively be issued outside
policy/CPS/audit. It would appear that this is now out of favour [0].

But, sub CAs and RAs is only one of the huge truck-sized holes in
audit. The big point here is that we as Mozilla have relied upon audit
to do the job. It has done a job, but it hasn't done the job that we
think it has done, or the job we want it to do.

So, Steve & Jean-Marc and others want to find out why the PwC audit
failed to pick up the problem at DigiNotar? But, let's be clear here
about the alternate -- should PwC have picked up the problem? Can we
point a the causal line of checking that would have led to that?

Or, is Audit just another case of security by obscurity? Purchase thy
seal, and be happy?

> It's also not very interesting to look at business processes.

That is the job of audit. If they are not doing a good job of it, then
we have to ask why. And be prepared to take on some of the blame...

> Mozilla
> could demand that all certificates issued by CAs in the root program are
> published in due course, just as domain names under gTLDs are published

> and available in bulk form. This would help with detection of certain
> process failures and is much easier to do than judging an audit. It
> would also implicitly cover the RA side, which is currently completely
> out of scope as far as I can tell based on the two-page audit reports.

This would also kill the sense of certs being used for privacy and
confidentiality. Maybe that's a good thing, dunno?

iang



[0] Is this being tightened up? I don't know. I've seen many
mutterings, but I'm not sure if each of the different bodies has really
taken it on to task. We are told there is a lot of backroom dealing as
those CAs that made a lot of hay while the sun shone are assessing the
damage.

As far as Mozilla is concerned, policy covers it, IMHO. There is no
exception in there that says "if you issue a sub-CA, it's outside policy."

[Jean-Marc Desperrier]

ianG wrote:
> So, Steve & Jean-Marc and others want to find out why the PwC audit
> failed to pick up the problem at DigiNotar? But, let's be clear here
> about the alternate -- should PwC have picked up the problem? Can we
> point a the causal line of checking that would have led to that?

Yes, the conclusion might very well be that the problem was outside what
PwC was paid to verify. The important point is that it wasn't outside
what the public could legitimately expect from an audited and trusted
CA. So we need to follow the causal line until finding what failed,
which is not necessarily PwC.
I know that Mozilla is already on it's way to implement a number of
change in it's CA policy, but before jumping to implementing change,
it's much better to have a detailled understanding of what failed.

[ianG]

On 26/09/11 19:28 PM, Jean-Marc Desperrier wrote:

> ianG wrote:
>> So, Steve & Jean-Marc and others want to find out why the PwC audit
>> failed to pick up the problem at DigiNotar? But, let's be clear here
>> about the alternate -- should PwC have picked up the problem? Can we
>> point a the causal line of checking that would have led to that?
>

> Yes, the conclusion might very well be that the problem was outside
> what PwC was paid to verify. The important point is that it wasn't
> outside what the public could legitimately expect from an audited and
> trusted CA. So we need to follow the causal line until finding what
> failed, which is not necessarily PwC.
> I know that Mozilla is already on it's way to implement a number of
> change in it's CA policy, but before jumping to implementing change,
> it's much better to have a detailled understanding of what failed.

Yep. In full agreement.

And another call for full disclosure :) Uncomfortable as it will be for
CAs, we now have evidence of systemic flaws in the CA process [0]. As
we will not be able to rely on the CAs to clean their houses, we will
need to lean heavily on those independents outside the CAs to advise.
Which means they need to be able to see what really happened.

Telling Mozilla won't help this process.



iang


[0] by this I mean data that supports the long-known theoretical flaws.