Turktrust SubCA abuse and MITM'ing

5467 views
Skip to first unread message

Stephen Schultze

unread,
Jan 3, 2013, 2:08:14 PM1/3/13
to mozilla-dev-s...@lists.mozilla.org

Reed Loden

unread,
Jan 3, 2013, 2:20:38 PM1/3/13
to dev-secur...@lists.mozilla.org
On Thu, 03 Jan 2013 14:08:14 -0500
Stephen Schultze <sjschult...@gmail.com> wrote:

> Radio silence over here:
> https://blog.mozilla.org/security/

https://blog.mozilla.org/security/2013/01/03/revoking-trust-in-two-turktrust-certficates/

Just took a little bit longer to post.

:)

~reed

Phillip Hallam-Baker

unread,
Jan 3, 2013, 2:21:56 PM1/3/13
to Stephen Schultze, mozilla-dev-s...@lists.mozilla.org
Hmm, what is there to discuss? The nature of the breach seems clear enough.
There is a well established set of controls that could and should have been
deployed to mitigate it.

So the question I would like to have an answer to is why the auditor didn't
spot the lack of those controls. I would also like to know exactly how the
issue occurred and why the certs had the cert signing bit set.

Another question is if they tried to install the certs on a server and if
so why it accepted a cert signing cert as an end-entity cert. They are
meant to be mutually exclusive roles.


On Thu, Jan 3, 2013 at 2:08 PM, Stephen Schultze <
sjschult...@gmail.com> wrote:

> So, are we discussing this?
>
> http://googleonlinesecurity.**blogspot.com/2013/01/**
> enhancing-digital-certificate-**security.html<http://googleonlinesecurity.blogspot.com/2013/01/enhancing-digital-certificate-security.html>
>
> http://blogs.technet.com/b/**msrc/archive/2013/01/03/**
> security-advisory-2798897-**released-certificate-trust-**list-updated.aspx<http://blogs.technet.com/b/msrc/archive/2013/01/03/security-advisory-2798897-released-certificate-trust-list-updated.aspx>
>
> Radio silence over here:
> https://blog.mozilla.org/**security/ <https://blog.mozilla.org/security/>
> ______________________________**_________________
> dev-security-policy mailing list
> dev-security-policy@lists.**mozilla.org<dev-secur...@lists.mozilla.org>
> https://lists.mozilla.org/**listinfo/dev-security-policy<https://lists.mozilla.org/listinfo/dev-security-policy>
>



--
Website: http://hallambaker.com/

Stephen Schultze

unread,
Jan 3, 2013, 5:17:17 PM1/3/13
to mozilla-dev-s...@lists.mozilla.org
From the blog post:
"We have also suspended inclusion of the “TÜRKTRUST Bilgi İletişim ve
Bilişim Güvenliği Hizmetleri A.Ş. (c) Aralık 2007” root certificate,
pending further review."

I only see these two Turktrust root certs in NSS, neither of which
matches that name:
https://www.mozilla.org/projects/security/certs/included/#TURKTRUST

I see, however, that the "2007" CA cert is listed here as the new root
approved for EV:
https://bugzilla.mozilla.org/show_bug.cgi?id=433845

Evidently this has not even made its way into production yet?
https://bugzilla.mozilla.org/show_bug.cgi?id=795355

Was this the root that the SubCAs in question were issued from, or were
they issued from one of the two DV certs? What is the rationale for
revoking trust in the non-production "2007" EV CA cert but not the other
DV certs?

Is there a private bugzilla bug for tracking this issue that can now be
made public?

Steve

Kathleen Wilson

unread,
Jan 3, 2013, 5:30:56 PM1/3/13
to mozilla-dev-s...@lists.mozilla.org
On 1/3/13 2:17 PM, Stephen Schultze wrote:
>> https://blog.mozilla.org/security/2013/01/03/revoking-trust-in-two-turktrust-certficates/
>>
> From the blog post:
> "We have also suspended inclusion of the “TÜRKTRUST Bilgi İletişim ve
> Bilişim Güvenliği Hizmetleri A.Ş. (c) Aralık 2007” root certificate,
> pending further review."
>
> I only see these two Turktrust root certs in NSS, neither of which
> matches that name:
> https://www.mozilla.org/projects/security/certs/included/#TURKTRUST
>
> I see, however, that the "2007" CA cert is listed here as the new root
> approved for EV:
> https://bugzilla.mozilla.org/show_bug.cgi?id=433845
>

The newer root cert had been included in Firefox 18. However, while we
made the change to distrust the two mis-issued certs, we backed out the
code change for the new root cert inclusion.


> Evidently this has not even made its way into production yet?
> https://bugzilla.mozilla.org/show_bug.cgi?id=795355
>

This update had been in Firefox 18 beta for a while. The other root cert
changes in bug #795355 are still happening. But the code has been
updated to not include the newer TURKTRUST root cert.



> Was this the root that the SubCAs in question were issued from, or were
> they issued from one of the two DV certs? What is the rationale for
> revoking trust in the non-production "2007" EV CA cert but not the other
> DV certs?
>

No. The mis-issued certs were signed by one of the older TURKTRUST root
certs.

We want to discuss this case before adding trust to another TURKTRUST
root. There are various options to consider, including:
a) Add the new TURKTRUST root in a future FF release, and allow EV
treatment.
b) Add the new TURKTRUST root in a future FF release, but don't allow
the EV treatment.
c) Add a new TURKTRUST root that has been name constrained to *.tr and
*.ir in a future FF release.
d) Don't add a new TURKTRUST root cert to NSS. The old TURKTRUST root
certs expire in March 2015. (the old roots are not EV-enabled)
e) Remove all TURKTRUST root certs from NSS.
d) other?


> Is there a private bugzilla bug for tracking this issue that can now be
> made public?
>

Yes. I'll check.


Kathleen




mert....@gmail.com

unread,
Jan 3, 2013, 5:53:27 PM1/3/13
to mozilla.dev.s...@googlegroups.com, mozilla-dev-s...@lists.mozilla.org, Stephen Schultze
Please find some critical points below about the root cause of the instance:

• The case occurred in August 2011 during a software migration operation, before the first successful ETSI TS 102 042 audit which took place in November 2011. The sequence of events that led to the mistaken issuance of two certificates can be best summarized as follows:
• Prior to June 2011, the certificate profiles on the production system were exported to the test system, containing a particular number of certificate profiles.
• For the sake of testing purposes, 2 more profiles were added that contain CA extensions.
• In the meantime, the production system was also updated to meet the need of demand to contain 3 more SSL certificate profiles. Hence the production system and the test system appeared to have different number of profiles by one, and the two sets matched only in the profiles at the date of the first data migration.
• The tests were completed before June 30, 2011. It was the unfortunate event that the production system was patched with the profiles in the test system, which had happened to contain 2 wrong profiles and lacked 3 correct profiles.
• The wrong profiles were only used on August 8, 2011 to issue those two faulty certificates which were certainly not intended to be sub-CA certificates.
• A certificate request on the 10th of August had called the use of one of the missing profiles, which was drawn to attention by a thrown exception. In order to fix this the whole set of certificate profiles was this time replaced to contain all correct profiles. Therefore the problem had been fixed once and for all although the unfortunate event that the certificates were mistakenly issued remained hidden.
It is assured that, this clearly identifies the root cause that led to the generation of faulty certificates. All related data resides in archives, and the sequence was all traced back to understand what really had happened. The system had been finally updated on October 17, 2011 and went through a successful ETSI TS 102 042 audit by KPMG on November 2011.
Although the system is now immune to any such kind of errors, further preemptive measures are implemented as described below:
One is a post process control for the certificates issued; the other is a run time control checking the certificate profile for end users.
Via the post process control, the validity period, basic constraints, CRL distribution point, Authority Info Access (AIA) and the other profile details are checked after certificate generation according to the certificate requirements coming from respective certificate policies before the certificate is delivered to the end customer. The post process control is planned as a separate and independent service from the certificate generation module of the CA software.
Via the run time control, the basic constraints are restricted to the end user certificates.
Both controls have already been implemented.
All OCSP requests and CRL data have already been analyzed to detect any anomaly during the entire period. The data revealed no anomaly at all.

The following issues are also worth considering at the moment:
1) One of the mistakenly issued certificates has been revoked before putting into use upon the request of the customer.
2) The other certificate was reported to sign a fraudulent certificate (*.google.com) on December 6, 2012.
3) Before the December 6, 2012, the certificate was installed on an IIS as a web mail server.
4) On December 6, 2012, the cert (and the key) was exported to a Checkpoint firewall. It was the same day as the issuance of the fraudulent certificate (*.google.com).
5) The Checkpoint firewall was said to be configured as MITM. It appears that the Checkpoint automatically generates MITM certificates once a CA cert was installed (http://www.gilgil.net/communities/19714)
6) The second certificate was immediately revoked as soon as it was brought to TURKTRUST’s attention by Google on December 26, 2012.
7) The available data strongly suggests that the *.google.com cert was not issued for dishonest purposes or has not been used for such a purpose.
8) There is certainly not a bit of data to show an evidence of a security breach on TURKTRUST systems.



Message has been deleted

Ned Ulbricht

unread,
Jan 3, 2013, 9:24:27 PM1/3/13
to dev-secur...@lists.mozilla.org
On 01/03/2013 02:21 PM, Phillip Hallam-Baker wrote:

> Another question is if they tried to install the certs on a server and if
> so why it accepted a cert signing cert as an end-entity cert. They are
> meant to be mutually exclusive roles.

Naturally, I appreciate the few technical details provided to this list
by Mert Özarar. In relation to your question, I note with interest, his
statement:

> 3) Before the December 6, 2012, the certificate was installed on
> an IIS as a web mail server.

Do all versions of IIS accept cert signing certs as end-entity certs?

I'm afraid that I have no experience at all in operating IIS.

--
Ned Ulbricht
mailto:ne...@netscape.net

Stephen Schultze

unread,
Jan 3, 2013, 9:26:43 PM1/3/13
to mozilla-dev-s...@lists.mozilla.org
Why would a certificate intended for end-entity SSL use (albeit actually
enabled for CA use) be installed on a "firewall"? What was the system
administrator's intent? What is the Checkpoint device model that it was
installed on? The link that you provided shows a video in which MITM is
a feature of Checkpoint devices that must be explicitly enabled.

On what network was this Checkpoint device installed, and what set of
users were being MITM'ed? Specific IP blocks would be helpful.

I presume that the SubCA cert in question is the one for *.EGO.GOV.TR,
so you can answer these questions with respect to what the sysadmin for
the government of Turkey was intending to do.

Did your OCSP requests show no requests at all for gmail.com? If so,
this is troubling, given the evidence that at least one request for
gmail.com by a Chrome browser was MITM'ed. What happened to the OCSP
request? How many OCSP requests appear in your logs for domain names for
which Turktrust has not issued certificates? None? If more than none,
which domains and how many requests?

Where is your KPMG audit memorialized? Kathleen has your auditor listed
in her spreadsheet as BTK (the Turkish government), and I see only BTK
audits attached to your bugzilla bugs.

To be clear, I'm not making accusations, but a bit of professional
paranoia is in order.

Phillip Hallam-Baker

unread,
Jan 3, 2013, 9:45:30 PM1/3/13
to Stephen Schultze, mozilla-dev-s...@lists.mozilla.org
Thinking off the top of my head, wouldn't an SSL VPN use an end entity SSL
cert?

I can think of endless reasons for putting SSL certs on things. In fact I
don't think I have ever come across a device with a 32 bit CPU that could
not benefit from at least one in some way.

What strikes me as bizarre is the idea that you feed it a CA cert and it
starts playing Big Brother. Isn't that a rather serious legal liability?
Not to mention a security vulnerability.

On Thu, Jan 3, 2013 at 9:26 PM, Stephen Schultze <
sjschult...@gmail.com> wrote:

> Why would a certificate intended for end-entity SSL use (albeit actually
> enabled for CA use) be installed on a "firewall"? What was the system
> administrator's intent? What is the Checkpoint device model that it was
> installed on? The link that you provided shows a video in which MITM is a
> feature of Checkpoint devices that must be explicitly enabled.
>
> On what network was this Checkpoint device installed, and what set of
> users were being MITM'ed? Specific IP blocks would be helpful.
>
> I presume that the SubCA cert in question is the one for *.EGO.GOV.TR, so
> you can answer these questions with respect to what the sysadmin for the
> government of Turkey was intending to do.
>
> Did your OCSP requests show no requests at all for gmail.com? If so, this
> is troubling, given the evidence that at least one request for gmail.comby a Chrome browser was MITM'ed. What happened to the OCSP request? How
> many OCSP requests appear in your logs for domain names for which Turktrust
> has not issued certificates? None? If more than none, which domains and how
> many requests?
>
> Where is your KPMG audit memorialized? Kathleen has your auditor listed in
> her spreadsheet as BTK (the Turkish government), and I see only BTK audits
> attached to your bugzilla bugs.
>
> To be clear, I'm not making accusations, but a bit of professional
> paranoia is in order.
>
>
> On 1/3/13 5:53 PM, mert....@gmail.com wrote:
>
>> a CA cert was installed (http://www.gilgil.net/**communities/19714<http://www.gilgil.net/communities/19714>
>> )
>> 6) The second certificate was immediately revoked as soon as it was
>> brought to TURKTRUST’s attention by Google on December 26, 2012.
>> 7) The available data strongly suggests that the *.google.com cert
>> was not issued for dishonest purposes or has not been used for such a
>> purpose.
>> 8) There is certainly not a bit of data to show an evidence of a
>> security breach on TURKTRUST systems.
>>
>
>

Stephen Schultze

unread,
Jan 3, 2013, 10:01:24 PM1/3/13
to mozilla-dev-s...@lists.mozilla.org
On 1/3/13 9:45 PM, Phillip Hallam-Baker wrote:
> What strikes me as bizarre is the idea that you feed it a CA cert and it
> starts playing Big Brother. Isn't that a rather serious legal liability?
> Not to mention a security vulnerability.

Indeed, it seems hard to imagine that a device would be designed to do
this. Furthermore, the Checkpoint product materials do not seem to
indicate that this is this case.

Stephen Schultze

unread,
Jan 3, 2013, 11:51:24 PM1/3/13
to mozilla-dev-s...@lists.mozilla.org
My commentary on what we know so far:
https://freedom-to-tinker.com/blog/sjs/turktrust-certificate-authority-errors-demonstrate-the-risk-of-subordinate-certificates/

As a bonus, I link to the openssl output of the SubCA cert that issued
the gmail.com cert. I don't think that this has been published publicly
so far.

Rob Stradling

unread,
Jan 4, 2013, 3:04:02 AM1/4/13
to Stephen Schultze, mozilla-dev-s...@lists.mozilla.org
On 04/01/13 02:26, Stephen Schultze wrote:
<snip>
> Did your OCSP requests show no requests at all for gmail.com? If so,
> this is troubling, given the evidence that at least one request for
> gmail.com by a Chrome browser was MITM'ed. What happened to the OCSP
> request?

Steve,

Firstly, OCSP requests don't contain domain names.

Secondly, although I haven't seen the fake gmail.com cert, I would be
very surprised if it states http://ocsp.turktrust.com.tr as the
AIA->OCSP URL, given that it was generated by a Checkpoint firewall
rather than issued by Turktrust.

--
Rob Stradling
Senior Research & Development Scientist
COMODO - Creating Trust Online

Ned Ulbricht

unread,
Jan 4, 2013, 6:08:12 AM1/4/13
to dev-secur...@lists.mozilla.org
On 01/03/2013 11:51 PM, Stephen Schultze wrote:

> As a bonus, I link to the openssl output of the SubCA cert that issued
> the gmail.com cert. I don't think that this has been published publicly
> so far.

Thank you.

Of course when I first heard the news, in my own personal browser, I
immediately removed the trust bits from Turktrust's root certificates.

Which again leaves the vexing question, what action should I take to
protect my users?

Madell, William

unread,
Jan 4, 2013, 6:55:06 AM1/4/13
to Rob Stradling, Stephen Schultze, mozilla-dev-s...@lists.mozilla.org
Rob's right - no AIA extension (or CRLDP, for that matter) in the bad EE cert:

0:d=0 hl=4 l=1455 cons: SEQUENCE
4:d=1 hl=4 l=1175 cons: SEQUENCE
8:d=2 hl=2 l= 3 cons: cont [ 0 ]
10:d=3 hl=2 l= 1 prim: INTEGER :02
13:d=2 hl=2 l= 16 prim: INTEGER :0A889040CE126E6557AEC2427B4AC1FB
31:d=2 hl=2 l= 13 cons: SEQUENCE
33:d=3 hl=2 l= 9 prim: OBJECT :sha1WithRSAEncryption
44:d=3 hl=2 l= 0 prim: NULL
46:d=2 hl=2 l= 110 cons: SEQUENCE
48:d=3 hl=2 l= 11 cons: SET
50:d=4 hl=2 l= 9 cons: SEQUENCE
52:d=5 hl=2 l= 3 prim: OBJECT :countryName
57:d=5 hl=2 l= 2 prim: PRINTABLESTRING :TR
61:d=3 hl=2 l= 15 cons: SET
63:d=4 hl=2 l= 13 cons: SEQUENCE
65:d=5 hl=2 l= 3 prim: OBJECT :stateOrProvinceName
70:d=5 hl=2 l= 6 prim: UTF8STRING :ANKARA
78:d=3 hl=2 l= 15 cons: SET
80:d=4 hl=2 l= 13 cons: SEQUENCE
82:d=5 hl=2 l= 3 prim: OBJECT :localityName
87:d=5 hl=2 l= 6 prim: UTF8STRING :ANKARA
95:d=3 hl=2 l= 12 cons: SET
97:d=4 hl=2 l= 10 cons: SEQUENCE
99:d=5 hl=2 l= 3 prim: OBJECT :organizationName
104:d=5 hl=2 l= 3 prim: UTF8STRING :EGO
109:d=3 hl=2 l= 24 cons: SET
111:d=4 hl=2 l= 22 cons: SEQUENCE
113:d=5 hl=2 l= 3 prim: OBJECT :organizationalUnitName
118:d=5 hl=2 l= 15 prim: UTF8STRING :EGO BILGI ISLEM
135:d=3 hl=2 l= 21 cons: SET
137:d=4 hl=2 l= 19 cons: SEQUENCE
139:d=5 hl=2 l= 3 prim: OBJECT :commonName
144:d=5 hl=2 l= 12 prim: UTF8STRING :*.EGO.GOV.TR
158:d=2 hl=2 l= 30 cons: SEQUENCE
160:d=3 hl=2 l= 13 prim: UTCTIME :121206085515Z
175:d=3 hl=2 l= 13 prim: UTCTIME :130607194327Z
190:d=2 hl=2 l= 102 cons: SEQUENCE
192:d=3 hl=2 l= 11 cons: SET
194:d=4 hl=2 l= 9 cons: SEQUENCE
196:d=5 hl=2 l= 3 prim: OBJECT :countryName
201:d=5 hl=2 l= 2 prim: PRINTABLESTRING :US
205:d=3 hl=2 l= 19 cons: SET
207:d=4 hl=2 l= 17 cons: SEQUENCE
209:d=5 hl=2 l= 3 prim: OBJECT :stateOrProvinceName
214:d=5 hl=2 l= 10 prim: PRINTABLESTRING :California
226:d=3 hl=2 l= 22 cons: SET
228:d=4 hl=2 l= 20 cons: SEQUENCE
230:d=5 hl=2 l= 3 prim: OBJECT :localityName
235:d=5 hl=2 l= 13 prim: PRINTABLESTRING :Mountain View
250:d=3 hl=2 l= 19 cons: SET
252:d=4 hl=2 l= 17 cons: SEQUENCE
254:d=5 hl=2 l= 3 prim: OBJECT :organizationName
259:d=5 hl=2 l= 10 prim: PRINTABLESTRING :Google Inc
271:d=3 hl=2 l= 21 cons: SET
273:d=4 hl=2 l= 19 cons: SEQUENCE
275:d=5 hl=2 l= 3 prim: OBJECT :commonName
280:d=5 hl=2 l= 12 prim: T61STRING :*.google.com
294:d=2 hl=3 l= 159 cons: SEQUENCE
297:d=3 hl=2 l= 13 cons: SEQUENCE
299:d=4 hl=2 l= 9 prim: OBJECT :rsaEncryption
310:d=4 hl=2 l= 0 prim: NULL
312:d=3 hl=3 l= 141 prim: BIT STRING
456:d=2 hl=4 l= 723 cons: cont [ 3 ]
460:d=3 hl=4 l= 719 cons: SEQUENCE
464:d=4 hl=2 l= 32 cons: SEQUENCE
466:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Extended Key Usage
471:d=5 hl=2 l= 1 prim: BOOLEAN :0
474:d=5 hl=2 l= 22 prim: OCTET STRING [HEX DUMP]:301406082B0601050507030106082B06010505070302
498:d=4 hl=4 l= 664 cons: SEQUENCE
502:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Subject Alternative Name
507:d=5 hl=2 l= 1 prim: BOOLEAN :0
510:d=5 hl=4 l= 652 prim: OCTET STRING [HEX DUMP]:30820288820C2A2E6
76F6F676C652E636F6D820D2A2E616E64726F69642E636F6D82162A2E617070656E67696E652E676
F6F676C652E636F6D82122A2E636C6F75642E676F6F676C652E636F6D82162A2E676F6F676C652D6
16E616C79746963732E636F6D820B2A2E676F6F676C652E6361820B2A2E676F6F676C652E636C820
E2A2E676F6F676C652E636F2E696E820E2A2E676F6F676C652E636F2E6A70820E2A2E676F6F676C6
52E636F2E756B820F2A2E676F6F676C652E636F6D2E6172820F2A2E676F6F676C652E636F6D2E617
5820F2A2E676F6F676C652E636F6D2E6272820F2A2E676F6F676C652E636F6D2E636F820F2A2E676
F6F676C652E636F6D2E6D78820F2A2E676F6F676C652E636F6D2E7472820F2A2E676F6F676C652E6
36F6D2E766E820B2A2E676F6F676C652E6465820B2A2E676F6F676C652E6573820B2A2E676F6F676
C652E6672820B2A2E676F6F676C652E6875820B2A2E676F6F676C652E6974820B2A2E676F6F676C6
52E6E6C820B2A2E676F6F676C652E706C820B2A2E676F6F676C652E7074820F2A2E676F6F676C656
17069732E636E82142A2E676F6F676C65636F6D6D657263652E636F6D820D2A2E677374617469632
E636F6D820C2A2E75726368696E2E636F6D82102A2E75726C2E676F6F676C652E636F6D82162A2E7
96F75747562652D6E6F636F6F6B69652E636F6D820D2A2E796F75747562652E636F6D820B2A2E797
4696D672E636F6D820B616E64726F69642E636F6D8204672E636F8206676F6F2E676C8214676F6F6
76C652D616E616C79746963732E636F6D820A676F6F676C652E636F6D8212676F6F676C65636F6D6
D657263652E636F6D820A75726368696E2E636F6D8208796F7574752E6265820B796F75747562652
E636F6D
1166:d=4 hl=2 l= 15 cons: SEQUENCE
1168:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Basic Constraints
1173:d=5 hl=2 l= 1 prim: BOOLEAN :255
1176:d=5 hl=2 l= 5 prim: OCTET STRING [HEX DUMP]:3003010100
1183:d=1 hl=2 l= 13 cons: SEQUENCE
1185:d=2 hl=2 l= 9 prim: OBJECT :sha1WithRSAEncryption
1196:d=2 hl=2 l= 0 prim: NULL
1198:d=1 hl=4 l= 257 prim: BIT STRING


Here's the PEM of the cert, itself, for those interested:

-----BEGIN CERTIFICATE-----
MIIFrzCCBJegAwIBAgIQCoiQQM4SbmVXrsJCe0rB+zANBgkqhkiG9w0BAQUFADBu
MQswCQYDVQQGEwJUUjEPMA0GA1UECAwGQU5LQVJBMQ8wDQYDVQQHDAZBTktBUkEx
DDAKBgNVBAoMA0VHTzEYMBYGA1UECwwPRUdPIEJJTEdJIElTTEVNMRUwEwYDVQQD
DAwqLkVHTy5HT1YuVFIwHhcNMTIxMjA2MDg1NTE1WhcNMTMwNjA3MTk0MzI3WjBm
MQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTEWMBQGA1UEBxMNTW91
bnRhaW4gVmlldzETMBEGA1UEChMKR29vZ2xlIEluYzEVMBMGA1UEAxQMKi5nb29n
bGUuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCs+WnJyJfzSM1n9Chn
cEgWZkAGH/qtEtKcROmSxvaD+dfvYGYghEFkgAzdA76MMDw7vWHjOZfFNhXxAkXj
6sw/CkUB7SqmCKUHxy2p/QPfG41P5ErAnAsCoAfpv0CPewwyv6zJHwEAUWxHmb94
LC9eWoJUa7hKifDg/io+R5yy/wIDAQABo4IC0zCCAs8wIAYDVR0lAQEABBYwFAYI
KwYBBQUHAwEGCCsGAQUFBwMCMIICmAYDVR0RAQEABIICjDCCAoiCDCouZ29vZ2xl
LmNvbYINKi5hbmRyb2lkLmNvbYIWKi5hcHBlbmdpbmUuZ29vZ2xlLmNvbYISKi5j
bG91ZC5nb29nbGUuY29tghYqLmdvb2dsZS1hbmFseXRpY3MuY29tggsqLmdvb2ds
ZS5jYYILKi5nb29nbGUuY2yCDiouZ29vZ2xlLmNvLmlugg4qLmdvb2dsZS5jby5q
cIIOKi5nb29nbGUuY28udWuCDyouZ29vZ2xlLmNvbS5hcoIPKi5nb29nbGUuY29t
LmF1gg8qLmdvb2dsZS5jb20uYnKCDyouZ29vZ2xlLmNvbS5jb4IPKi5nb29nbGUu
Y29tLm14gg8qLmdvb2dsZS5jb20udHKCDyouZ29vZ2xlLmNvbS52boILKi5nb29n
bGUuZGWCCyouZ29vZ2xlLmVzggsqLmdvb2dsZS5mcoILKi5nb29nbGUuaHWCCyou
Z29vZ2xlLml0ggsqLmdvb2dsZS5ubIILKi5nb29nbGUucGyCCyouZ29vZ2xlLnB0
gg8qLmdvb2dsZWFwaXMuY26CFCouZ29vZ2xlY29tbWVyY2UuY29tgg0qLmdzdGF0
aWMuY29tggwqLnVyY2hpbi5jb22CECoudXJsLmdvb2dsZS5jb22CFioueW91dHVi
ZS1ub2Nvb2tpZS5jb22CDSoueW91dHViZS5jb22CCyoueXRpbWcuY29tggthbmRy
b2lkLmNvbYIEZy5jb4IGZ29vLmdsghRnb29nbGUtYW5hbHl0aWNzLmNvbYIKZ29v
Z2xlLmNvbYISZ29vZ2xlY29tbWVyY2UuY29tggp1cmNoaW4uY29tggh5b3V0dS5i
ZYILeW91dHViZS5jb20wDwYDVR0TAQH/BAUwAwEBADANBgkqhkiG9w0BAQUFAAOC
AQEAE9hy/EupOVhpJvcDLGkfFdWGUDw93eLzebKguOuaVSJ5xIxaiNU3MWO4ciJq
e8EZ9rPGGhiFfqEAPuEd0Svzn8zL523OHgwfQZ0zXHPDhN/KZDI3ohQH6t0EbA3k
hXMo9oeHWRfCbzEQcZTZ3TtC04D0MJGThjfx1jmq5WHvmWKwtujfYNSBLAyETnlq
lbEZFH4LDdaYayZk3VnRw4SQx7ykbDSOjbbCsUoBSV04sFEcCWNvU6IhyEMAPCRy
MlDakG1cc9G3U0yeGXM7SAvqPs90TDlgBvDtgTfzLv9vZTmYw3ziDSORUBE55kGH
/gF9p4SAOb3230ExWmWAayoyZg==
-----END CERTIFICATE-----

Cheers,
Bill
_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Erwann Abalea

unread,
Jan 4, 2013, 7:08:10 AM1/4/13
to mozilla-dev-s...@lists.mozilla.org
Le vendredi 4 janvier 2013 12:55:06 UTC+1, Madell, William a écrit :
> Rob's right - no AIA extension (or CRLDP, for that matter) in the bad EE cert:

Accessible here: http://pastebin.com/0kchRaYp

I think the original certificate were to be used on https://posta.ego.gov.tr.
The long list of dnsNames seems to be the same as what is found on the legitimate certificate for encrypted.google.com.

Erwann Abalea

unread,
Jan 4, 2013, 7:28:49 AM1/4/13
to mozilla-dev-s...@lists.mozilla.org
There's also no basicConstraints:pathlen on the issuing CA, which could have mitigated the failure (on decent validation path implementations).
That's probably not a new idea, but could Mozilla (or any other browser vendor) ask for basicConstraints:pathlen=0 for CAs intended to issue end-entities?

Carsten.D...@external.t-systems.com

unread,
Jan 4, 2013, 7:42:52 AM1/4/13
to kwi...@mozilla.com, mozilla-dev-s...@lists.mozilla.org
Hi Kathleen,

<snip>
> We want to discuss this case before adding trust to another TURKTRUST
> root. There are various options to consider, including:

For me personally it's very unclear, how a certificate can be _mistakenly_ issued as CA instead of EE certificate (as the google blog post is stating [1]). Does it means, someone enabled the appropriate extension within the profile used to issue certificates?
In this case I wonder, why there are only 2 of those certs. Wouldn't that mean that someone discovered the mistake and change the profile back as it should be? Next question coming to my mind is, why didn't someone check for mis-issued certs when they changed the profile?

Best regards,
Carsten

[1] http://googleonlinesecurity.blogspot.de/2013/01/enhancing-digital-certificate-security.html


Carsten Dahlenkamp
T-Systems International GmbH
Trust Center Applications
Untere Industriestraße 20, 57250 Netphen, Germany
+49 271 708-1643 (Tel.)
E-Mail: carsten.d...@external.t-systems.com
http://www.t-systems.com, http://www.telesec.de


Message has been deleted

TURKTRUST

unread,
Jan 4, 2013, 10:25:20 AM1/4/13
to mozilla.dev.s...@googlegroups.com, mozilla-dev-s...@lists.mozilla.org
* We try to answer related questions while representing TURKTRUST.

On Friday, January 4, 2013 4:26:43 AM UTC+2, Stephen Schultze wrote:
> Why would a certificate intended for end-entity SSL use (albeit actually
>
> enabled for CA use) be installed on a "firewall"? What was the system
>
> administrator's intent? What is the Checkpoint device model that it was
>
> installed on? The link that you provided shows a video in which MITM is
>
> a feature of Checkpoint devices that must be explicitly enabled.
>
>
>
> On what network was this Checkpoint device installed, and what set of
>
> users were being MITM'ed? Specific IP blocks would be helpful.
>

* It was installed on the customer side. We are certainly not in position to answer these questions to full extent. However, it is almost apparent that the agency has wanted to configure the firewall as a MITM proxy for their internal users. Our thorough OCSP analysis has also supported this in the sense that almost 96% of OCSP requests stemmed from the EGO domain.


> I presume that the SubCA cert in question is the one for *.EGO.GOV.TR,
>
> so you can answer these questions with respect to what the sysadmin for
>
> the government of Turkey was intending to do.
>

* There was a misunderstanding. There is nothing related directly to Turkish Government. EGO is an agency working for the Municipality of Ankara to give services about electricity, gas and transportation. They have
changed their firewall vendor/model and this case happened. Odds are too low that there is a malicious intent on their side. There is also no evidence at all any other malicious use of the faulty cert.

>
> Did your OCSP requests show no requests at all for gmail.com? If so,
>
> this is troubling, given the evidence that at least one request for
>
> gmail.com by a Chrome browser was MITM'ed. What happened to the OCSP
>
> request? How many OCSP requests appear in your logs for domain names for
>
> which Turktrust has not issued certificates? None? If more than none,
>
> which domains and how many requests?
>

* The faulty certificate has neither OCSP nor CRL pointers. Hence OCSP and CRL logs are followed from the misissued certified that havesigned the faulty one. As we have just said, almost all requests stemmed from the EGO domain. There is no OCSP request from any domain that we have not issued a certificate.

>
> Where is your KPMG audit memorialized? Kathleen has your auditor listed
>
> in her spreadsheet as BTK (the Turkish government), and I see only BTK
>
> audits attached to your bugzilla bugs.
>

* https://www.turktrust.com.tr/files/bsi.pdf is our BSI-certificate. We may also share with you the lead auditor's coordinates of the KPMG of the Netherlands if you are interested in contacting with him.

Florian Weimer

unread,
Jan 4, 2013, 2:36:58 PM1/4/13
to TURKTRUST, mozilla-dev-s...@lists.mozilla.org
* TURKTRUST:

> * It was installed on the customer side. We are certainly not in
> position to answer these questions to full extent. However, it is
> almost apparent that the agency has wanted to configure the firewall
> as a MITM proxy for their internal users. Our thorough OCSP analysis
> has also supported this in the sense that almost 96% of OCSP
> requests stemmed from the EGO domain.

Does this mean that the certificate was actually used for interception
purposes?

> * https://www.turktrust.com.tr/files/bsi.pdf is our
> BSI-certificate. We may also share with you the lead auditor's
> coordinates of the KPMG of the Netherlands if you are interested in
> contacting with him.

Can we blacklist BSI Group The Netherlands B.V. as auditors because
they didn't catch this?

Kathleen, I think in

<https://docs.google.com/spreadsheet/pub?key=0Ah-tHXMAwqU3dGx0cGFObG9QM192NFM4UWNBMlBaekE&single=true&gid=1&output=html>

the audit links for TÜRKTRUST are wrong, they refer to qualified
electronic signatures which aren't relevant for web usage.

Jan Schejbal

unread,
Jan 4, 2013, 4:07:30 PM1/4/13
to mozilla-dev-s...@lists.mozilla.org
Am 2013-01-04 20:36, schrieb Florian Weimer:
> Does this mean that the certificate was actually used for interception
> purposes?

Yes - a false certificate issued for a google domain was how this entire
issue was discovered:
http://googleonlinesecurity.blogspot.de/2013/01/enhancing-digital-certificate-security.html

Kind regards,
Jan

--
Please avoid sending mails, use the group instead.
If you really need to send me an e-mail, mention "FROM NG"
in the subject line, otherwise my spam filter will delete your mail.
Sorry for the inconvenience, thank the spammers...
Message has been deleted
Message has been deleted

Ned Ulbricht

unread,
Jan 4, 2013, 5:37:01 PM1/4/13
to dev-secur...@lists.mozilla.org
On 01/04/2013 10:25 AM, TURKTRUST wrote:

> * It was installed on the customer side. We are certainly not in
> position to answer these questions to full extent. However, it is
> almost apparent that the agency has wanted to configure the firewall
> as a MITM proxy for their internal users.

I can certainly appreciate that you are not in position to fully answer
questions about your customer's operations.

>From the "HTTPS Inspection FAQ" at the Checkpoint Security Support
Center, I'm reading:

| Can I replace the gateway's CA with a different CA?
|
| A: Yes, you can import any CA certificate to be used for HTTPS
| inspection.
|
| To import a CA certificate:
|
| 1. Open SmartDashboard.
| 2. Right-click a gateway object and select Edit.
| 3. Click HTTPS Inspection > Import HTTPS Inspection.
|
| Or, from the HTTPS Inspection > Gateways pane of a supported blade,
| click the arrow next to Create Certificate and select Import
| certificate from file.

Without actually using this UI myself, I am still inferring from this
that your customer knew that they were in possession of a subCA
certificate and intended to install that CA cert for "HTTPS Inspection".

You seem to be telling us that you've reached that same conclusion
regarding your customer's level of knowledge and intent--at the time
they installed the certificate on the Checkpoint firewall.

Again, I really appreciate Turktrust's responsiveness and disclosure
regarding this issue.

Ned Ulbricht

unread,
Jan 5, 2013, 8:32:50 AM1/5/13
to dev-secur...@lists.mozilla.org
On 01/04/2013 04:07 PM, Jan Schejbal wrote:

> Yes - a false certificate issued for a google domain was how this entire
> issue was discovered:

It occurs to me --perhaps I am slow immediately after the holidays--
that while MitM appliances may not be able to prevent Google's Chrome
browser from detecting a MitM CA, the vendors will nevertheless develop
countermeasures. Even if Chrome detects a MitM attack, a MitM appliance
is position to block Chrome's report of that attack to Google. An arms
race between Google and the MitM appliance vendors may bloom.

Sitting here on the sidelines as we are here, that will bring up an
interesting set of policy questions. We should perhaps all think about
those in advance.

Ned Ulbricht

unread,
Jan 5, 2013, 9:09:37 AM1/5/13
to dev-secur...@lists.mozilla.org
On 01/03/2013 05:53 PM, mert....@gmail.com wrote:
> 4) On December 6, 2012, the cert (and the key) was exported to a
> Checkpoint firewall. It was the same day as the issuance of the
> fraudulent certificate (*.google.com).

This indicates a two-and-a-half week delay from the Dec 6, 2012
generation of the MitM CA certificate until Google's detection of the
attack on Dec 24, 2012.

I am presuming that this delay is due to the use of browsers other than
Google's Chrome on the intercepted network. The cert may have been
generated on the MitM appliance due to browsing activity using Microsoft
IE, Apple Safari, or using Mozilla Firefox.

mert....@gmail.com

unread,
Jan 5, 2013, 10:30:44 AM1/5/13
to mozilla-dev-s...@lists.mozilla.org, mozilla-dev-s...@lists.mozilla.org
We have verified that EGO started the firewall set-up on the 6th of December. MITM proxy feature service started on the 21st of December. Chrome browsers in the LAN had started access errors to Google site on the very same day. This information is based on the declaration of the firewall vendor and EGO. We believe that it is most likely that the faulty certificate was automatically generated by the firewall on the 21st of December (cloning all attribues but few, worth reminding that the faulty certificate differ from the true Google certificate only in a few fields). The likelihood of a malicious usage is indeed barely above zero.

ps. We have put it to our web site, as well.

TURKTRUST Inc.


On Friday, January 4, 2013 11:19:46 PM UTC+2, TURKTRUST wrote:
> Dear All,
>
>
>
> The press release and especially technical details are being posted on our official web site (available in Turkish at http://turktrust.com.tr/kamuoyu-aciklamasi.html and in English at http://turktrust.com.tr/en/kamuoyu-aciklamasi-en.html). Some links we believe that would help understand the situation better are also made available.
>
>
>
> As of now, the current data strongly suggests that the faulty certificate has not been used maliciously. Please check the technical details document for a discussion of MITM certs on a firewall and Google's "public key pinning" implementation.
>
>
>
> The second document and the links shall be kept updated as new information become available. We'll let you know when the site is updated.
>
>
>
> Best Regards,
>
>
>
> TURKTRUST Inc.

mert....@gmail.com

unread,
Jan 5, 2013, 10:30:44 AM1/5/13
to mozilla.dev.s...@googlegroups.com, mozilla-dev-s...@lists.mozilla.org

Phillip Hallam-Baker

unread,
Jan 5, 2013, 10:34:55 AM1/5/13
to Ned Ulbricht, dev-secur...@lists.mozilla.org
Well lets just forget the particular incident here and instead focus on the
fact that this is now the second time that one of these issues has come to
light because of the policy enforcement in Google Chrome.

I think that what we should be urgently considering right now is how to
extend that capability so that it covers more domains and so that it is
more robust. I proposed a scheme of that sort with Ben Laurie at Google in
the first version of CAA but those parts got stripped out because another
group had a turf war.

I think there will be no shortage of people piling on to heap criticism on
Turktrust, I think we should also consider ways we can reinforce the system
with technical measures.



On Sat, Jan 5, 2013 at 8:32 AM, Ned Ulbricht <ne...@netscape.net> wrote:

> On 01/04/2013 04:07 PM, Jan Schejbal wrote:
>
> > Yes - a false certificate issued for a google domain was how this entire
> > issue was discovered:
>
> It occurs to me --perhaps I am slow immediately after the holidays--
> that while MitM appliances may not be able to prevent Google's Chrome
> browser from detecting a MitM CA, the vendors will nevertheless develop
> countermeasures. Even if Chrome detects a MitM attack, a MitM appliance
> is position to block Chrome's report of that attack to Google. An arms
> race between Google and the MitM appliance vendors may bloom.
>

--
Website: http://hallambaker.com/

Phillip Hallam-Baker

unread,
Jan 5, 2013, 10:35:33 AM1/5/13
to Ned Ulbricht, dev-secur...@lists.mozilla.org
"Well lets just forget the particular incident here" FOR A MOMENT

oops, left out an important caveat.
--
Website: http://hallambaker.com/

TURKTRUST

unread,
Jan 5, 2013, 10:37:46 AM1/5/13
to mozilla-dev-s...@lists.mozilla.org, mozilla-dev-s...@lists.mozilla.org
On Friday, January 4, 2013 9:36:58 PM UTC+2, Florian Weimer wrote:
> * TURKTRUST:
>
>
>
> > * It was installed on the customer side. We are certainly not in
>
> > position to answer these questions to full extent. However, it is
>
> > almost apparent that the agency has wanted to configure the firewall
>
> > as a MITM proxy for their internal users. Our thorough OCSP analysis
>
> > has also supported this in the sense that almost 96% of OCSP
>
> > requests stemmed from the EGO domain.
>
>
>
> Does this mean that the certificate was actually used for interception
>
> purposes?
>

Well again, that cannot be said with full confidence. We tend to interpret the OCSP data from a different perspective: Since the faulty firewall certificate had neither OCSP address nor CRL Distribution Point, the requests came for the mistakenly issued certificate. Having a very large majority of requests stemming from their domain is a strong indication that the mistakenly cert was used only by EGO.

TURKTRUST

unread,
Jan 5, 2013, 10:37:46 AM1/5/13
to mozilla.dev.s...@googlegroups.com, mozilla-dev-s...@lists.mozilla.org
On Friday, January 4, 2013 9:36:58 PM UTC+2, Florian Weimer wrote:
> * TURKTRUST:
>
>
>
> > * It was installed on the customer side. We are certainly not in
>
> > position to answer these questions to full extent. However, it is
>
> > almost apparent that the agency has wanted to configure the firewall
>
> > as a MITM proxy for their internal users. Our thorough OCSP analysis
>
> > has also supported this in the sense that almost 96% of OCSP
>
> > requests stemmed from the EGO domain.
>
>
>
> Does this mean that the certificate was actually used for interception
>
> purposes?
>

Well again, that cannot be said with full confidence. We tend to interpret the OCSP data from a different perspective: Since the faulty firewall certificate had neither OCSP address nor CRL Distribution Point, the requests came for the mistakenly issued certificate. Having a very large majority of requests stemming from their domain is a strong indication that the mistakenly cert was used only by EGO.

>
>

David E. Ross

unread,
Jan 5, 2013, 11:27:19 AM1/5/13
to mozilla-dev-s...@lists.mozilla.org
Please stop sending your messages to both the E-mail mailing list
<mozilla-dev-s...@lists.mozilla.org> and the newsgroup
[mozilla.dev.security.policy]. Also stop sending to the same E-mail
address as both To: and Cc:. Your messages are appearing multiple times
since the mailing list and newsgroup feed into each other.

Erwann Abalea

unread,
Jan 5, 2013, 12:41:15 PM1/5/13
to mozilla-dev-s...@lists.mozilla.org
Le samedi 5 janvier 2013 16:30:44 UTC+1, mert....@gmail.com a écrit :
> We have verified that EGO started the firewall set-up on the 6th of December. MITM proxy feature service started on the 21st of December. Chrome browsers in the LAN had started access errors to Google site on the very same day. This information is based on the declaration of the firewall vendor and EGO. We believe that it is most likely that the faulty certificate was automatically generated by the firewall on the 21st of December (cloning all attribues but few, worth reminding that the faulty certificate differ from the true Google certificate only in a few fields). The likelihood of a malicious usage is indeed barely above zero.

If the certificate was installed on Dec 6th but the MITM feature was only activated on Dec 21st, it means that this activation wasn't automatic, as opposed to what you wrote on Jan 3rd. You then also wrote that the *.google.com certificate was generated on Dec 6th, and now on Dec 21st. Which date is the good one?

If this activation isn't automatic, then it's manual (obviously), and it can be considered a malicious usage. Next question is "who did something malicious?".

If this activation is automatic and started on Dec 6th, then other services and users have been MITMed, undetected. Next question is "was it a malicious action, or an error?".

Stephen Schultze

unread,
Jan 5, 2013, 1:15:20 PM1/5/13
to mozilla-dev-s...@lists.mozilla.org
On 1/5/13 10:34 AM, Phillip Hallam-Baker wrote:
> I think that what we should be urgently considering right now is how to
> extend that capability so that it covers more domains and so that it is
> more robust. I proposed a scheme of that sort with Ben Laurie at Google in
> the first version of CAA but those parts got stripped out because another
> group had a turf war.

This work is already well underway, and I agree that we should support
it. Whatever your interpretation of how the pinning proposal in CAA got
stripped out, DNS-based pinning is now clearly defined in DANE (RFC
6698), which is an IETF Proposed Standard Protocol. So, for the DNS
path, that would seem to be a clear choice.

There are also trust-on-first-use proposals that are in advanced stages:

- draft-ietf-websec-key-pinning is on Standards Track, and work is
underway to implement it in NSS:
https://tools.ietf.org/html/draft-ietf-websec-key-pinning
https://wiki.mozilla.org/Security/Features/CA_pinning_functionality
https://bugzilla.mozilla.org/show_bug.cgi?id=744204

- See also Moxie's TACK, also Standards Track:
https://tools.ietf.org/html/draft-perrin-tls-tack

So, yeah, these should be supported and implemented. Of course, getting
to the bottom of this Turktrust issue can continue in parallel.

TURKTRUST

unread,
Jan 5, 2013, 2:12:40 PM1/5/13
to mozilla-dev-s...@lists.mozilla.org
* First a clarification: The modification of dates or any information concerning the firewall is only due to the new information that is made available to us by Turkish representative of the firewall and their client EGO. Both parties have been very cooperative until now to discuss the issue and help us understand what had actually happened.

* What follows is the series of events according to the available information at the present:
- On 6th of December, the new firewall was installed. MITM (SSL Inspection) was certainly not enabled. Therefore the faulty certificate generation was not a question on the 6th of December.
- MITM was enabled manually on the 21st of December by making the relavant configuration. The mistakenly issued sub-CA cert was certainly being installed on the same day and was checked for SSL inspection. This is confirmed once more via a phone call with the firewall representative.
- Hence the firewall (on the fly) generated the faulty google cert.
- We are certainly not in a position to tell wheter they were really aware of what they had been actually doing. Our feeling, however, is that their action was most probably not intended to generate a faulty certificate.

* We do not like to comment on whether configuring a MITM server can be considered a malicious usage.

John Wilander

unread,
Jan 5, 2013, 2:42:05 PM1/5/13
to Stephen Schultze, mozilla-dev-s...@lists.mozilla.org
I'd just like to add my support for two questions that have been asked but are still not answered afaik:

1) Turktrust, why did you not immediately check the config of all certs issued during the fairly small time window with faulty profiles from the test system? It seems an obvious thing to do for a professional CA discovering a serious misconfig.

2) Mozilla, the effectiveness of Chrome's cert pinning is proving itself. Recently Firefox started using Chrome's whitelist of pre-configured HSTS domains. Why not start cert pinning in Firefox following the same pattern? With the popularity of Google's services we would get a good increase in detection capability _and_ we would get more transparency.

Regards, John


--
My music http://www.johnwilander.com
Twitter https://twitter.com/johnwilander
CV or Résumé http://johnwilander.se

5 jan 2013 kl. 19:15 skrev Stephen Schultze <sjschult...@gmail.com>:

> On 1/5/13 10:34 AM, Phillip Hallam-Baker wrote:
>> I think that what we should be urgently considering right now is how to
>> extend that capability so that it covers more domains and so that it is
>> more robust. I proposed a scheme of that sort with Ben Laurie at Google in
>> the first version of CAA but those parts got stripped out because another
>> group had a turf war.
>
> This work is already well underway, and I agree that we should support it. Whatever your interpretation of how the pinning proposal in CAA got stripped out, DNS-based pinning is now clearly defined in DANE (RFC 6698), which is an IETF Proposed Standard Protocol. So, for the DNS path, that would seem to be a clear choice.
>
> There are also trust-on-first-use proposals that are in advanced stages:
>
> - draft-ietf-websec-key-pinning is on Standards Track, and work is underway to implement it in NSS:
> https://tools.ietf.org/html/draft-ietf-websec-key-pinning
> https://wiki.mozilla.org/Security/Features/CA_pinning_functionality
> https://bugzilla.mozilla.org/show_bug.cgi?id=744204
>
> - See also Moxie's TACK, also Standards Track:
> https://tools.ietf.org/html/draft-perrin-tls-tack
>
> So, yeah, these should be supported and implemented. Of course, getting to the bottom of this Turktrust issue can continue in parallel.

Erwann Abalea

unread,
Jan 5, 2013, 2:57:15 PM1/5/13
to mozilla-dev-s...@lists.mozilla.org
Le samedi 5 janvier 2013 20:12:40 UTC+1, TURKTRUST a écrit :
> * First a clarification: The modification of dates or any information concerning the firewall is only due to the new information that is made available to us by Turkish representative of the firewall and their client EGO. Both parties have been very cooperative until now to discuss the issue and help us understand what had actually happened.

I understand that.

> * What follows is the series of events according to the available information at the present:
>
> - On 6th of December, the new firewall was installed. MITM (SSL Inspection) was certainly not enabled. Therefore the faulty certificate generation was not a question on the 6th of December.
>
> - MITM was enabled manually on the 21st of December by making the relavant configuration. The mistakenly issued sub-CA cert was certainly being installed on the same day and was checked for SSL inspection. This is confirmed once more via a phone call with the firewall representative.
>
> - Hence the firewall (on the fly) generated the faulty google cert.

But based on evidence found here: http://pastebin.com/0kchRaYp
the faulty Google certificate was generated on Dec 6th. And this certificate correctly chains up to your "TURKTRUST_Elektronik_Sunucu_Sertifikasi_Hizmetleri_s2" root CA included in Mozilla.
Is that Dec 6th date a bogus one? Or does it correspond to anything else?

> - We are certainly not in a position to tell wheter they were really aware of what they had been actually doing. Our feeling, however, is that their action was most probably not intended to generate a faulty certificate.

If MITM was manually activated, it means they were aware of what they were doing. It also means the generation of faulty certificates was deliberate (that cannot be separated from the MITM goal).

> * We do not like to comment on whether configuring a MITM server can be considered a malicious usage.

Under a private root, it's not malicious. Under a public one, it's malicious. Easy.

Erwann Abalea

unread,
Jan 5, 2013, 3:03:46 PM1/5/13
to mozilla-dev-s...@lists.mozilla.org, mozilla-dev-s...@lists.mozilla.org
Le samedi 5 janvier 2013 20:57:15 UTC+1, Erwann Abalea a écrit :
> But based on evidence found here: http://pastebin.com/0kchRaYp
> the faulty Google certificate was generated on Dec 6th. And this certificate correctly chains up to your "TURKTRUST_Elektronik_Sunucu_Sertifikasi_Hizmetleri_s2" root CA included in Mozilla.

Sorry, copied/pasted the wrong name. The root this MITM chains up to is named
"CN=TÜRKTRUST Elektronik Sertifika Hizmet Sağlayıcısı, C=TR, L=Ankara, O=TÜRKTRUST Bilgi İletişim ve Bilişim Güvenliği Hizmetleri A.Ş. (c) Kasım 2005"
(UTF8 combined with completely foreign names is hard to deal with)
Message has been deleted

TURKTRUST

unread,
Jan 6, 2013, 10:16:43 AM1/6/13
to mozilla-dev-s...@lists.mozilla.org
It appears that Dec 6th date is a bogus one. It seems that the firewall had replicated the true google cert at https://encrypted.google.com/ as soon as the MITM was activated on 21st of December.

Not really wanted to be a part of a discussion about whether their action was delibarate or not to issue a faulty cert, but let us share with you what they (EGO and the firewall representative) have confirmed to us yesterday: They had first tried to use the internal CA on the firewall. The internal clients (obviously) had given trust warning, so they had decided to export the trusted cert on the web mail server. They should, of course, have chosen to install trust for the internal CA into their clients in the domain.

Jan Schejbal

unread,
Jan 6, 2013, 10:25:55 AM1/6/13
to mozilla-dev-s...@lists.mozilla.org
Am 2013-01-05 20:57, schrieb Erwann Abalea:
> But based on evidence found here: http://pastebin.com/0kchRaYp
> the faulty Google certificate was generated on Dec 6th.

The certificate bears a notBefore validity date of Dec 6th. This does
not necessarily mean it was generated on that day. I have no experience
with Checkpoint MitM, but they might simply copy the validity date of
the original certificate.

Erwann Abalea

unread,
Jan 6, 2013, 10:46:49 AM1/6/13
to mozilla-dev-s...@lists.mozilla.org
Le dimanche 6 janvier 2013 16:25:55 UTC+1, Jan Schejbal a écrit :
> Am 2013-01-05 20:57, schrieb Erwann Abalea:
>
> > But based on evidence found here: http://pastebin.com/0kchRaYp
> > the faulty Google certificate was generated on Dec 6th.
>
> The certificate bears a notBefore validity date of Dec 6th. This does
> not necessarily mean it was generated on that day. I have no experience
> with Checkpoint MitM, but they might simply copy the validity date of
> the original certificate.

That's plausible. Start and end dates are identical on the real and the fake certificates.
So this Dec 6th is a bogus date.

Ned Ulbricht

unread,
Jan 6, 2013, 10:57:13 AM1/6/13
to dev-secur...@lists.mozilla.org
On 01/06/2013 10:16 AM, TURKTRUST wrote:

> It appears that Dec 6th date is a bogus one. It seems that the
> firewall had replicated the true google cert at
> https://encrypted.google.com/ as soon as the MITM was activated on
> 21st of December.

The 21st of December, 2012 was a Friday.

I'm unfamiliar with typical working practices in Turkey. Would it have
been usual for IT maintainance activities to take place on Friday?

Most of the other workers would have had the Friday off?


> Not really wanted to be a part of a discussion about whether their
> action was delibarate or not to issue a faulty cert

You have consistently told us that the initial events of August 2011
resulted from simple configuration mistakes. To be crudely blunt, I am
not sure that configuration mistakes are more trivial than --say-- for
wild hyperbolic effect, "Penetration by Iranian- or Syrian- sponsored
intelligence operations." If you play against nation-state level
adversaries, then sometimes you might lose. It happens. Insert coin
and play again. But if you just have sloppy testing procedures....

OTOH, I sincerely appreciate that simple mistakes in testing protocols
occur, and are much more likely to bite people and organizations than
wild, implausible Hollywood scenarios which lack any shred of supporting
evidence. Mistakes happen, simply and more frequently than anyone would
like.

mert....@gmail.com

unread,
Jan 6, 2013, 10:58:15 AM1/6/13
to mozilla-dev-s...@lists.mozilla.org
The team had got stuck with a missing profile, they had not considered the possibility of having any wrong profiles at all. After all, a mistake is a mistake. Being professional and making mistakes, we do not really want to comment on that.

Erwann Abalea

unread,
Jan 6, 2013, 11:01:33 AM1/6/13
to mozilla-dev-s...@lists.mozilla.org
Le dimanche 6 janvier 2013 16:16:43 UTC+1, TURKTRUST a écrit :
> Not really wanted to be a part of a discussion about whether their action was delibarate or not to issue a faulty cert, but let us share with you what they (EGO and the firewall representative) have confirmed to us yesterday: They had first tried to use the internal CA on the firewall. The internal clients (obviously) had given trust warning, so they had decided to export the trusted cert on the web mail server. They should, of course, have chosen to install trust for the internal CA into their clients in the domain.

I don't think we can avoid that discussion. You're a public CA, this has been made public, so the whole resolution has to be made public. If it appears that at the end, a succession of errors were made and nobody can be considered malicious, right, but it still has to be discussed first.
It can be discussed on the public CABForum mailing list (openly, between CAs and Browsers), or here (because Mozilla and CAs communicated exactly one year ago on the MiTM subject). Either one seems fine.

Let me rephrase what you wrote. EGO had an internal CA, and used it for MiTM purpose on the CheckPoint HTTPS inspection stack. That raised warnings on their internal clients. On the same day, they discovered they had a public CA certificate instead of a classic server certificate, exported it from their server and imported it on the firewall to stop the warnings while doing HTTPS inspection. Is that right?

Mert Özarar (TÜRKTRUST)

unread,
Jan 6, 2013, 12:05:29 PM1/6/13
to mozilla-dev-s...@lists.mozilla.org
Rephrasing is right.

As regards to the discussion, we are certainly open to provide as much
information as possible as we did in the last couple of days. We'll
continue to cooperate that way until the issue is fully cleared. The
sooner the outcome is reached, the better condition we'll be in.
Nevertheless, what we meant in the last communication was that
inferring the purpose of their action was just difficult (i.e. EGO and
their service provider, the firewall representative). To help the
discussion, we will continue to supply any available information
coming to us from EGO.

TURKTRUST

unread,
Jan 6, 2013, 12:17:02 PM1/6/13
to mozilla-dev-s...@lists.mozilla.org
Monday through Friday are the working days of the week in Turkey. Well, practising a firewall maintenance on a working day is, of course, up to the action taker.

Madell, William

unread,
Jan 6, 2013, 6:35:21 PM1/6/13
to mozilla-dev-s...@lists.mozilla.org
Erwann,

>> Let me rephrase what you wrote. EGO had an internal CA, and used it for MiTM purpose on the CheckPoint HTTPS inspection stack. That raised warnings on their internal clients. On the same day, they discovered they had a public CA certificate instead of a classic server certificate, exported it from their server and imported it on the firewall to stop the warnings while doing HTTPS inspection. Is that right?<<

That's not how I interpret Mert's statement regarding the Checkpoint.

I believe he said that the initial SSL cert for the firewall was generated by the firewall's own internal CA - this resulted in an untrusted EE cert from the viewpoint of the browser clients. So, the thinking on EGO's part, appears to be: that SSL cert is untrusted, so let's import an SSL cert onto the firewall that we know is publicly trusted (i.e. the one we've already got on a web server). Now, not knowing that this "SSL" cert was actually a "CA" cert, by installing it on the firewall they inadvertently initialised an 'automated' process by which the firewall says, "hey look, a new CA cert has just been imported....let me automatically issue an SSL cert from this one in order to enable the HTTPS MitM proxy service".

Of course, I'm only assuming that's how the Checkpoint reacted to the import of a new CA cert based on what Mert has shared about that product....others may understand the features of this firewall better than me.

Cheers,
Bill

phar...@rim.com

unread,
Jan 4, 2013, 5:54:45 AM1/4/13
to mozilla-dev-s...@lists.mozilla.org, mozilla-dev-s...@lists.mozilla.org
Am Freitag, 4. Januar 2013 05:51:24 UTC+1 schrieb Stephen Schultze:
> My commentary on what we know so far:
>
> https://freedom-to-tinker.com/blog/sjs/turktrust-certificate-authority-errors-demonstrate-the-risk-of-subordinate-certificates/
>
>
>
> As a bonus, I link to the openssl output of the SubCA cert that issued
>
> the gmail.com cert. I don't think that this has been published publicly
>
> so far.

Thanks, nice article. Are the two mis-issued intermediate certificates publicly available somewhere?

za...@zachlipton.com

unread,
Jan 4, 2013, 5:52:20 PM1/4/13
to mozilla-dev-s...@lists.mozilla.org, mozilla-dev-s...@lists.mozilla.org
As I noted in my comments to Stephen Schultze's blog post last night (https://freedom-to-tinker.com/blog/sjs/turktrust-certificate-authority-errors-demonstrate-the-risk-of-subordinate-certificate) and as discussed in Turktrust's latest documents, it appears likely that the sub-CA and MITM in question were discovered by Google as a result of their unique (to my knowledge) public key pinning mechanism (http://www.imperialviolet.org/2011/05/04/pinning.html) and Chrome's diagnostic/metrics system, which is capable of reporting pinning failures back to Google. This system only reports failures when users attempt to connect to a site within a whitelist of Google-owned domains, so it is unclear what other certificates may have been issued by the sub-CA.

On the brighter side, if Google does have key pinning failure logs that were used to identify this situation, it stands to reason that Google will be shed some light on the scope of the sub-CA abuse based on the logged data. I'm curious whether anyone from Google can share more about how this was discovered. This may help provide some independent confirmation as to whether any interception attempts occurred outside of the ego.gov.tr network if Google associates user IP addresses with these logs.

Note that a really clever MITM attack would block Chrome's diagnostic reporting back to Google and/or avoid interfering with SSL traffic to the whitelisted domains.

I also have some confusion about Turktrust's account of this incident. According to Turktrust:
* Turktrust's trusted root CA mistakenly created two sub-CAs on August 8, 2011, both were issued to customers.
* One was revoked. The other was installed on a web server to facilitate SSL access to a corporate webmail system.
* Sometime after December 6, 2012, the sub-CA was copied from the web server to a Checkpoint "firewall" (perhaps an inappropriate label for a device that is intentionally performing active manipulation of the data stream, but in any event...), intending to configure a MITM proxy.

For the web server application, Turktrust's customer would have wanted a standard run-of-the-mill end entity SSL certificate. However, for the firewall/MITM use, the network administrator must have known that he needed to upload a CA, typically a corporate controlled self-signed CA, to the device, as the device needs a CA (with corresponding private key) to issue certificates. Why then would he try to install the certificate from the web server as a CA on the Checkpoint device unless he knew that he was actually in possession of a trusted sub-CA? It would seem that the customer knew exactly what it was doing here.

Cheers,

--zach

za...@zachlipton.com

unread,
Jan 4, 2013, 5:52:20 PM1/4/13
to mozilla.dev.s...@googlegroups.com, mozilla-dev-s...@lists.mozilla.org

Chris Palmer

unread,
Jan 7, 2013, 2:05:40 PM1/7/13
to Stephen Schultze, mozilla-dev-s...@lists.mozilla.org
On Sat, Jan 5, 2013 at 10:15 AM, Stephen Schultze
<sjschult...@gmail.com> wrote:

> There are also trust-on-first-use proposals that are in advanced stages:

I don't get why people call HPKP and TACK "TOFU". It is more accurate
to call them "trust fewer after first use". In case it isn't clear,
http://tools.ietf.org/html/draft-ietf-websec-key-pinning-04#section-2.5
means that the Host MUST serve a certificate chain that chains up to a
trust anchor the UA already trusts (i.e. the cert chain must be valid
according to the current rules for certificate validation).

> - draft-ietf-websec-key-pinning is on Standards Track, and work is underway
> to implement it in NSS:
> https://tools.ietf.org/html/draft-ietf-websec-key-pinning

See in particular
http://tools.ietf.org/html/draft-ietf-websec-key-pinning-04#section-3.
We intended reporting mode to be a way for site operators to learn how
tightly they can set their pin sets ("Gee, who all does issue for our
sites?"), but it could also serve as a fraudulent certificate
reporting feature as we have in Chrome for Google sites.

Phillip Hallam-Baker

unread,
Jan 7, 2013, 2:37:48 PM1/7/13
to Chris Palmer, mozilla-dev-s...@lists.mozilla.org, Stephen Schultze
What I have been saying about all these proposals from the start is that
90% of the value comes from reporting and 95% of the problems are due to
blocking on false positives.

So using these schemes in a report only mode has a lot of value with little
downside.

Which I why I gave up on a certain IETF proposal after the WG insisted that
they were going to do yet another crypto-perfectionist approach that will
fall flat on its face when dealing with real world administrators.
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>



--
Website: http://hallambaker.com/

Kathleen Wilson

unread,
Jan 7, 2013, 3:09:42 PM1/7/13
to mozilla-dev-s...@lists.mozilla.org

>> Is there a private bugzilla bug for tracking this issue that can now be
>> made public?
>>
>
> Yes. I'll check.


https://bugzilla.mozilla.org/show_bug.cgi?id=825022


za...@zachlipton.com

unread,
Jan 7, 2013, 6:26:38 PM1/7/13
to mozilla-dev-s...@lists.mozilla.org
On Thursday, January 3, 2013 6:24:27 PM UTC-8, Ned Ulbricht wrote:
> On 01/03/2013 02:21 PM, Phillip Hallam-Baker wrote:
> > 3) Before the December 6, 2012, the certificate was installed on
>
> > an IIS as a web mail server.
>
>
>
> Do all versions of IIS accept cert signing certs as end-entity certs?
>
>
>
> I'm afraid that I have no experience at all in operating IIS.
>

I have no experience with IIS either, but I just ran a simple experiment to try this on IIS 7 in a Windows 7 VM I had handy. I generated a self-signed root CA on my Mac (using OS X's Certificate Assistant in fact) and created a sub-CA off that root. I did not create any actual end-entity certificates from either CA. I was able to successfully import the sub-CA to the Windows certificate store and to configure IIS to serve https using the sub-CA. IIS and IE 9 didn't seem to care that I was using a CA certificate instead of an end-entity certificate.

Based on this test, it doesn't seem implausible that someone could be using a cert signing cert as an end-entity cert under IIS 7 without realizing it. Of course, it appears that someone likely did realize exactly what he had when he setup the Checkpoint MITM device last month.

Interestingly, Firefox did throw up a 'sec_error_inadequate_key_usage' when accessing the IIS test server (with the root certificate trusted, naturally). I suspect that I could have avoided this error if I spent more time twiddling the key usage extension bits, but I didn't go down that road.

My biggest question is what Google observed in its diagnostic logs that presumably led to the identification of this situation. That information may be able to provide some degree of independent confirmation as to the number of users impacted and what network(s) they were using at the time.

--zach

Gervase Markham

unread,
Jan 7, 2013, 6:45:44 PM1/7/13
to Erwann Abalea
On 04/01/13 12:28, Erwann Abalea wrote:
> There's also no basicConstraints:pathlen on the issuing CA, which
> could have mitigated the failure (on decent validation path
> implementations). That's probably not a new idea, but could Mozilla
> (or any other browser vendor) ask for basicConstraints:pathlen=0 for
> CAs intended to issue end-entities?

This is currently a SHOULD in the Baseline Requirements. There is
discussion about making it a MUST.

Gerv


Erwann Abalea

unread,
Jan 8, 2013, 5:14:36 AM1/8/13
to mozilla-dev-s...@lists.mozilla.org, Erwann Abalea
BR 1.1, Appendix B, Subordinate CA Certificate
-----
D. basicConstraints
This extension MUST be present and MUST be marked critical. The cA field MUST be set true. The pathLenConstraint field MAY be present.
-----

The field MAY be present, and nothing is recommended about its value.

Gervase Markham

unread,
Jan 8, 2013, 7:48:50 AM1/8/13
to mozilla-dev-s...@lists.mozilla.org
On 08/01/13 10:14, Erwann Abalea wrote:
> BR 1.1, Appendix B, Subordinate CA Certificate ----- D.
> basicConstraints This extension MUST be present and MUST be marked
> critical. The cA field MUST be set true. The pathLenConstraint
> field MAY be present. -----
>
> The field MAY be present, and nothing is recommended about its
> value.

I'm sorry; I misread the HTML-converted-to-plain-text patch someone sent
to the mailing list. One proposed new wording is this:

"This extension MUST be present and MUST be marked critical. The cA
field MUST be set true. The pathLenConstraint field SHOULD be present.
For Subordinate CAs which are used only to sign Subscriber Certificates,
OCSP certificates or responses, and CRLs, it is RECOMMENDED that the
pathLenConstraint field be present and set to zero."

Gerv


Phillip Hallam-Baker

unread,
Jan 8, 2013, 2:23:08 PM1/8/13
to Stephen Schultze, mozilla-dev-s...@lists.mozilla.org
If the browser providers would support hard fail on CRLs, most of the
problems we are discussing here would either not be problems or be a lot
easier to solve.

So given the refusal to support hard fail on CRLs, I find it rather
unlikely that the same browser providers will support hard fail on DANE
policy advice. Particularly since that involves an experimental
infrastructure (DNSSEC) that can be subject to MITM denial of service
attack by anyone who has MITM capability on TLS.

Take out the mandates and the insistence on making DANE a 100%
crypto-perfectionist solution and I could support it. But right now its a
fragile proposal that solves one problem by creating another.

Solving the reporting problem might eventually lead to a mechanism that
permits client enforcement. But client enforcement has much higher demands.


Now I have to take a look at the DANE TLS for SMTP proposal...


On Sat, Jan 5, 2013 at 1:15 PM, Stephen Schultze <
sjschult...@gmail.com> wrote:

> On 1/5/13 10:34 AM, Phillip Hallam-Baker wrote:
>
>> I think that what we should be urgently considering right now is how to
>> extend that capability so that it covers more domains and so that it is
>> more robust. I proposed a scheme of that sort with Ben Laurie at Google in
>> the first version of CAA but those parts got stripped out because another
>> group had a turf war.
>>
>
> This work is already well underway, and I agree that we should support it.
> Whatever your interpretation of how the pinning proposal in CAA got
> stripped out, DNS-based pinning is now clearly defined in DANE (RFC 6698),
> which is an IETF Proposed Standard Protocol. So, for the DNS path, that
> would seem to be a clear choice.
>
> There are also trust-on-first-use proposals that are in advanced stages:
>
> - draft-ietf-websec-key-pinning is on Standards Track, and work is
> underway to implement it in NSS:
> https://tools.ietf.org/html/**draft-ietf-websec-key-pinning<https://tools.ietf.org/html/draft-ietf-websec-key-pinning>
> https://wiki.mozilla.org/**Security/Features/CA_pinning_**
> functionality<https://wiki.mozilla.org/Security/Features/CA_pinning_functionality>
> https://bugzilla.mozilla.org/**show_bug.cgi?id=744204<https://bugzilla.mozilla.org/show_bug.cgi?id=744204>
>
> - See also Moxie's TACK, also Standards Track:
> https://tools.ietf.org/html/**draft-perrin-tls-tack<https://tools.ietf.org/html/draft-perrin-tls-tack>
>
> So, yeah, these should be supported and implemented. Of course, getting to
> the bottom of this Turktrust issue can continue in parallel.
> ______________________________**_________________
> dev-security-policy mailing list
> dev-security-policy@lists.**mozilla.org<dev-secur...@lists.mozilla.org>
> https://lists.mozilla.org/**listinfo/dev-security-policy<https://lists.mozilla.org/listinfo/dev-security-policy>
>



--
Website: http://hallambaker.com/

Erwann Abalea

unread,
Jan 8, 2013, 3:49:32 PM1/8/13