Incidents involving the CA WoSign

22460 views
Skip to first unread message

Gervase Markham

unread,
Aug 24, 2016, 9:09:02 AM8/24/16
to mozilla-dev-s...@lists.mozilla.org, Richard Wang
Dear m.d.s.policy,

Several incidents have come to our attention involving the CA "WoSign".
Mozilla is considering what action it should take in response to these
incidents. This email sets out our understanding of the situation.

Before we begin, we note that Section 1 of the Mozilla CA Certificate
Enforcement Policy[0] says: "When a serious security concern is noticed,
such as a major root compromise, it should be treated as a
security-sensitive bug, and the Mozilla Policy for Handling Security
Bugs should be followed." It is clear to us, and appears to be clear to
other CAs based on their actions, that misissuances where domain control
checks have failed fall into the category of "serious security concern".

Incident 0
----------

On or around April 23rd, 2015, WoSign's certificate issuance system for
their free certificates allowed the applicant to choose any port for
validation. Once validation had been completed, WoSign would
issue certificates for that domain. A researcher was able to obtain a
certificate for a university by opening a high-numbered port (>50,000)
and getting WoSign to use that port for validation of control.

This problem was reported to Google, and thence to WoSign and resolved.
Mozilla only became aware of it recently.

* Before the recent passage of Ballot 169 in the CAB Forum, which limits
the ports and paths which can be used, the Baseline Requirements said
that one acceptable method of domain validation was "Having the
Applicant demonstrate practical control over the FQDN by making an
agreed‐upon change to information found on an online Web page identified
by a uniform resource identifier containing the FQDN". This method
therefore did not violate the letter of the BRs. However, Mozilla
considers the basic security knowledge that ports over 1024 are
unprivileged should have led all CAs not to accept validations of domain
control on such ports, even when not documented in the BRs.

* The misissuance incident was not reported to Mozilla by WoSign as it
should have been (see above).

* This misissuance incident did not turn up on WoSign's subsequent BR
audit[1].

Incident 1
----------

In June 2015, an applicant found a problem with WoSign's free
certificate service, which allowed them to get a certificate for the
base domain if they were able to prove control of a subdomain.

The reporter proved the problem in two ways. They accidentally
discovered it when trying to get a certificate for med.ucf.edu and
mistakenly also applied for www.ucf.edu, which was approved. They then
confirmed the problem by using their control of
theiraccount.github.com/theiraccount.github.io to get a cert for
github.com, github.io, and www.github.io.

They reported this to WoSign, giving only the Github certificate as an
example. That cert was revoked and the vulnerability was fixed. However
recently, they got in touch with Google to note that the ucf.edu cert
still had not been revoked almost a year later.

* The lack of revocation of the ucf.edu certificate (still unrevoked at
time of writing, although it may have been by time of posting) strongly
suggests that WoSign either did not or could not search their issuance
databases for other occurrences of the same problem. Mozilla considers
such a search a basic part of the response to disclosure of a
vulnerability which causes misissuance, and expects CAs to keep records
detailed enough to make it possible.

* This misissuance incident was not reported to Mozilla by WoSign as it
should have been (see above).

* This misissuance incident did not turn up on WoSign's subsequent BR
audit[1].

Incident 2
----------

In July 2016, it became clear that there was some problems with the
StartEncrypt automatic issuance service recently deployed by the CA
StartCom. As well as other problems it had, which are outside the scope
of this discussion, changing a simple API parameter in the POST request
on the submission page changed the root certificate to which the
resulting certificate chained up. The value "2" made a certificate
signed by "StartCom Class 1 DV Server CA", "1" selected "WoSign CA Free
SSL Certificate G2" and "0" selected "CA 沃通根证书", another root
certificate owned by WoSign and trusted by Firefox.

Using the value "1" led to a certificate which had a notBefore date
(usage start date) of 20th December 2015, and which was signed using the
SHA-1 checksum algorithm.

* The issuance of certificates using SHA-1 has been banned by the
Baseline Requirements since January 1st, 2016. Browsers, including
Firefox, planned to enforce this[2] by not trusting certs with a
notBefore date after that date, but in the case of Firefox the fix had
to be backed out due to web compatibility issues. However, we are
considering how/when to reintroduce it, and CAs presumably know this.

* The issuance of backdated certificates is not forbidden, but is listed
in Mozilla's list of Problematic Practices[3]. It says "Minor tweaking
for technical compatibility reasons is accepted, but backdating
certificates in order to avoid some deadline or code-enforced
restriction is not."

* WoSign deny that their code backdated the certificates in order to
avoid browser-based restrictions - they say "this date is the day we
stop to use this code"[4]. If that is true, it is not clear to us how
StartCom came to deploy WoSign code that WoSign itself had abandoned.

* It seems clear from publicly available information that StartCom's
issuance systems are linked to WoSign's issuance systems in some way.
Nevertheless, it should not have been possible for an application for a
cert from StartCom to produce a cert signed by WoSign.

* This misissuance incident was not reported to Mozilla by WoSign as it
should have been.


Taking into account all these incidents and the actions of this CA,
Mozilla is considering what action to take. Your input is welcomed.

Gerv, Kathleen and Richard


[0]
https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/enforcement/
[1] https://cert.webtrust.org/SealFile?seal=2019&file=pdf
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=942515
[3]
https://wiki.mozilla.org/CA:Problematic_Practices#Backdating_the_notBefore_date
[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1293366

Eric Mill

unread,
Aug 24, 2016, 12:09:39 PM8/24/16
to Gervase Markham, Richard Wang, mozilla-dev-s...@lists.mozilla.org
For the thread's reference, here's the crt.sh link for the misissued GitHub
certificate:

https://crt.sh/?id=29647048

Valid for 3 years, for github.com. It's not in OneCRL, CRLset, or
Microsoft's disallowedcert.stl.
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>



--
konklone.com | @konklone <https://twitter.com/konklone>

Gervase Markham

unread,
Aug 24, 2016, 12:31:26 PM8/24/16
to Jeremy Rowley, Richard Wang
Hi Jeremy,

On 24/08/16 17:12, Jeremy Rowley wrote:
> On incident 0, its unclear whether a cert was actually mis-issued.
> Although they used a higher level port, did the researcher
> successfully bypass WoSign's domain validation process? Is the only
> concern that WoSign permitted higher level ports?

The result of the incident was that a certificate was issued to someone
who did not, in the normally understood sense of the word, have control
of the domain in question. Mozilla feels that even without a specific
injunction in the BRs, CAs should have known that ports > 1024 are not
privileged and not done control checks using them.

The severity of the problem, of course, is a matter for discussion here.

> On incident 2, it sounds like they are both using the same
> auto-generation script.

It seems like a bit more than that, doesn't it? Let's presume that
WoSign did not ship a copy of their intermediate cert's private key to
StartCom. Therefore, this cert must have been issued on the back end by
some sort of WoSign system. So either WoSign's back-end issuing service
has some form of authentication and the StartCom system had those
credentials (why?), or the WoSign system does not have any form of
authentication (concerning).

> Giving WoSign the benefit of the doubt, it
> sounds like maybe it was a bug in their software that permitted SHA1
> certs not an intentional back-dating issue. Is there any clarity
> around how this worked?

Perhaps WoSign would like to provide some :-)

Gerv

Peter Bowen

unread,
Aug 24, 2016, 12:44:44 PM8/24/16
to Gervase Markham, Richard Wang, mozilla-dev-s...@lists.mozilla.org, Jeremy Rowley
On Wed, Aug 24, 2016 at 9:30 AM, Gervase Markham <ge...@mozilla.org> wrote:
> On 24/08/16 17:12, Jeremy Rowley wrote:
>> On incident 2, it sounds like they are both using the same
>> auto-generation script.
>
> It seems like a bit more than that, doesn't it? Let's presume that
> WoSign did not ship a copy of their intermediate cert's private key to
> StartCom. Therefore, this cert must have been issued on the back end by
> some sort of WoSign system. So either WoSign's back-end issuing service
> has some form of authentication and the StartCom system had those
> credentials (why?), or the WoSign system does not have any form of
> authentication (concerning).

I think you are missing the most likely option: CA hosting. My
understanding is that it is not uncommon that one CA operator
contracts with another CA operator to run a CA on behalf of the first
operator. I don't think it has been clear what disclosure of this
practice is required. Given that I believe this is widespread, I
assumed that all of the issuing CAs in this case were operated by the
same entity.

Thanks,
Peter

Ryan Sleevi

unread,
Aug 24, 2016, 4:19:41 PM8/24/16
to Jeremy Rowley, Richard Wang, mozilla-dev-s...@lists.mozilla.org, Gervase Markham, Peter Bowen
On Wed, Aug 24, 2016 at 12:40 PM, Jeremy Rowley
<jeremy...@digicert.com> wrote:
> However, the fact a researcher was able to obtain a cert without proper domain
> validation is pretty serious. I'd like to hear more details about how this was
> accomplished. Ports 8080 and 8443 aren't that uncommon so penalizing someone
> merely for port use seems harsh when there wasn't a policy against it.

There was no restriction on ports at all. Any client-specified port
was accepted, and any HTTP-like response it gave back was accepted.

s...@gmx.ch

unread,
Aug 24, 2016, 8:22:59 PM8/24/16
to dev-secur...@lists.mozilla.org
Of course, adding the affected certs to OneCRL should be done immediately.

WoSign also has to be transparent about all (mis) issued certs in the
past and have to provide this info in the future.
If they can't, I think we may consider if the current certs that are
valid for 3 years should be restricted to a shorter period.

Regards,
Jonas
signature.asc

Richard Wang

unread,
Aug 24, 2016, 10:58:06 PM8/24/16
to Eric Mill, Gervase Markham, mozilla-dev-s...@lists.mozilla.org
this cert is revoked in the same once it is issued.
Thanks for posting to CT.


Best Regards,

Richard

From: Eric Mill [mailto:er...@konklone.com]
Sent: Thursday, August 25, 2016 12:08 AM
To: Gervase Markham <ge...@mozilla.org>
Cc: mozilla-dev-s...@lists.mozilla.org; Richard Wang <ric...@wosign.com>
Subject: Re: Incidents involving the CA WoSign

For the thread's reference, here's the crt.sh link for the misissued GitHub certificate:

https://crt.sh/?id=29647048

Valid for 3 years, for github.com<http://github.com>. It's not in OneCRL, CRLset, or Microsoft's disallowedcert.stl.
discovered it when trying to get a certificate for med.ucf.edu<http://med.ucf.edu> and
mistakenly also applied for www.ucf.edu<http://www.ucf.edu>, which was approved. They then
confirmed the problem by using their control of
theiraccount.github.com/theiraccount.github.io<http://theiraccount.github.com/theiraccount.github.io> to get a cert for
github.com<http://github.com>, github.io<http://github.io>, and www.github.io<http://www.github.io>.

They reported this to WoSign, giving only the Github certificate as an
example. That cert was revoked and the vulnerability was fixed. However
recently, they got in touch with Google to note that the ucf.edu<http://ucf.edu> cert
still had not been revoked almost a year later.

* The lack of revocation of the ucf.edu<http://ucf.edu> certificate (still unrevoked at
Taking into account all these incidents and the actions of this CA,
Mozilla is considering what action to take. Your input is welcomed.

dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>
https://lists.mozilla.org/listinfo/dev-security-policy



--
konklone.com<https://konklone.com> | @konklone<https://twitter.com/konklone>

Richard Wang

unread,
Aug 24, 2016, 11:40:53 PM8/24/16
to Gervase Markham, Jeremy Rowley, mozilla-dev-s...@lists.mozilla.org
See below inline, thanks.





Best Regards,



Richard



-----Original Message-----

From: Gervase Markham [mailto:ge...@mozilla.org]

Sent: Thursday, August 25, 2016 12:31 AM

To: Jeremy Rowley <jeremy...@digicert.com>; mozilla-dev-s...@lists.mozilla.org

Cc: Richard Wang <ric...@wosign.com>

Subject: Re: Incidents involving the CA WoSign



Hi Jeremy,



On 24/08/16 17:12, Jeremy Rowley wrote:

> On incident 0, its unclear whether a cert was actually mis-issued.

> Although they used a higher level port, did the researcher

> successfully bypass WoSign's domain validation process? Is the only

> concern that WoSign permitted higher level ports?



The result of the incident was that a certificate was issued to someone who did not, in the normally understood sense of the word, have control of the domain in question. Mozilla feels that even without a specific injunction in the BRs, CAs should have known that ports > 1024 are not privileged and not done control checks using them.



The severity of the problem, of course, is a matter for discussion here.



R: We got report from Google, but don’t tell me if any mis-issued certificate happen, just tell us to forbidden this high level port.

We are searching our database to try to find if any mis-issued cert is issued. And will update to this mail list.





> On incident 2, it sounds like they are both using the same

> auto-generation script.



It seems like a bit more than that, doesn't it? Let's presume that WoSign did not ship a copy of their intermediate cert's private key to StartCom. Therefore, this cert must have been issued on the back end by some sort of WoSign system. So either WoSign's back-end issuing service has some form of authentication and the StartCom system had those credentials (why?), or the WoSign system does not have any form of authentication (concerning).



R: NOT this case you think. Due to root inclusion problem, WoSign root is cross signed by StartCom since 2011. And we shared some facility with StartCom like CRL and OCSP distribution etc. But not this case, as I declared in the previous email, this is a API parameter option that can post data to any server located in any place.





> Giving WoSign the benefit of the doubt, it sounds like maybe it was a

> bug in their software that permitted SHA1 certs not an intentional

> back-dating issue. Is there any clarity around how this worked?



Perhaps WoSign would like to provide some :-)



R: I think we said clearly in the related Bugzilla ☺





Gerv



Richard Wang

unread,
Aug 24, 2016, 11:42:44 PM8/24/16
to Peter Bowen, Gervase Markham, mozilla-dev-s...@lists.mozilla.org, Jeremy Rowley
Yes, correct.

Due to root inclusion problem, WoSign root is cross signed by StartCom since 2011. And we shared some facility with StartCom like CRL and OCSP distribution etc. But not this case, as I declared in the previous email, this is a API parameter option that can post data to any server located in any place.

Best Regards,

Richard

-----Original Message-----
From: Peter Bowen [mailto:pzb...@gmail.com]
Sent: Thursday, August 25, 2016 12:45 AM
To: Gervase Markham <ge...@mozilla.org>
Cc: Jeremy Rowley <jeremy...@digicert.com>; mozilla-dev-s...@lists.mozilla.org; Richard Wang <ric...@wosign.com>
Subject: Re: Incidents involving the CA WoSign

On Wed, Aug 24, 2016 at 9:30 AM, Gervase Markham <ge...@mozilla.org> wrote:
> On 24/08/16 17:12, Jeremy Rowley wrote:
>> On incident 2, it sounds like they are both using the same
>> auto-generation script.
>
> It seems like a bit more than that, doesn't it? Let's presume that
> WoSign did not ship a copy of their intermediate cert's private key to
> StartCom. Therefore, this cert must have been issued on the back end
> by some sort of WoSign system. So either WoSign's back-end issuing
> service has some form of authentication and the StartCom system had
> those credentials (why?), or the WoSign system does not have any form
> of authentication (concerning).

Richard Wang

unread,
Aug 25, 2016, 12:05:32 AM8/25/16
to s...@gmx.ch, dev-secur...@lists.mozilla.org
We revoked this certificate, and we know this certificate is for test only.

For transparency, WoSign announced full transparency for all SSL certificate from July 5th that post all issued SSL certificate to Google log server, browsers can distrust WoSign issued SSL certificate after that day if no SCT embedded data in the certificate.

And WoSign even plan to post the code signing certificate and client certificate to log server for full transparency for all certificates.

See this news if you missed: https://www.wosign.com/english/News/2016_wosign_CT.htm.

And we plan to setup an free alert service for worldwide users that if any SSL certificate for domain you care is issued from any CA, then you can get the alert email, this will benfit the ecosystem.


Best Regards,

Richard

-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of s...@gmx.ch
Sent: Thursday, August 25, 2016 8:18 AM
To: dev-secur...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign

Of course, adding the affected certs to OneCRL should be done immediately.

WoSign also has to be transparent about all (mis) issued certs in the past and have to provide this info in the future.
If they can't, I think we may consider if the current certs that are valid for 3 years should be restricted to a shorter period.

Regards,
Jonas


> For the thread's reference, here's the crt.sh link for the misissued
> GitHub
> certificate:
>
> https://crt.sh/?id=29647048
>
> Valid for 3 years, for github.com. It's not in OneCRL, CRLset, or

Matt Palmer

unread,
Aug 25, 2016, 2:48:30 AM8/25/16
to dev-secur...@lists.mozilla.org
On Thu, Aug 25, 2016 at 04:03:04AM +0000, Richard Wang wrote:
> For transparency, WoSign announced full transparency for all SSL
> certificate from July 5th that post all issued SSL certificate to Google
> log server, browsers can distrust WoSign issued SSL certificate after that
> day if no SCT embedded data in the certificate.

That would be slightly more reassuring if there wasn't a history of certs
being issued with seemingly misleading notBefore values...

Separately, do you have any thoughts on the reports that WoSign's BR auditor
did not note any of the misissuances? Also, what changes, exactly, has
WoSign implemented to its policies and procedures to ensure that all trust
programs in which WoSign is a participant are notified of future incidents,
in line with each program's requirements?

- Matt

--
"The user-friendly computer is a red herring. The user-friendliness of a
book just makes it easier to turn pages. There's nothing user-friendly about
learning to read."
-- Alan Kay

Richard Wang

unread,
Aug 25, 2016, 3:14:10 AM8/25/16
to Matt Palmer, dev-secur...@lists.mozilla.org
Thanks for your friendly reminder.

We can post all 2015 issued SSL certificate to CT log server if necessary.

For BR auditor, I think this issue is too technical that fewer auditor can find out this problem.

We will add the quality control system to PKI system before issuing the certificate, and will check the crt.sh or use the CABF lint and X590 Lint to check the certificate before and after the certificate is issued to prevent such case, if such case happen, we will notify all browsers instantly.


Best Regards,

Richard

-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Matt Palmer
Sent: Thursday, August 25, 2016 2:48 PM
To: dev-secur...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign

_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Ryan Sleevi

unread,
Aug 25, 2016, 3:09:59 PM8/25/16
to mozilla-dev-s...@lists.mozilla.org
On Thursday, August 25, 2016 at 12:14:10 AM UTC-7, Richard Wang wrote:
> We can post all 2015 issued SSL certificate to CT log server if necessary.

Is there any reason not to do that proactively?

> For BR auditor, I think this issue is too technical that fewer auditor can find out this problem.

The audit letter included an attestation from Management that, during the time of the audit, management believed that the CA complied with the Baseline Requirements.

Management was aware of non-compliance, by virtue of revocation and system and procedural changes to align with compliance.

Thus, do you believe it was faithful and accurate for Management to warrant that the CA was operated in compliance with the BRs, given that Management was aware of incidents of non-compliance?

Matt Palmer

unread,
Aug 25, 2016, 7:35:39 PM8/25/16
to dev-secur...@lists.mozilla.org
On Thu, Aug 25, 2016 at 07:11:18AM +0000, Richard Wang wrote:
> We can post all 2015 issued SSL certificate to CT log server if necessary.

That doesn't provide any assurance, in the face of misleading notBefore
values in certificates. Without strong assurances that whatever failure of
systems or processes allowed misleading notBefore values to end up in
certificates has been corrected, the only way to be sure that there are no
shenanigans going on is for *all* certificates issued by WoSign,
*regardless* of their purported issuance date, to carry SCTs from a
qualified log (or have them presented at TLS establishment), and have the
"effective date" for the purposes of compliance with relevant standards and
guidelines to be the date of the earliest SCT presented. This is because
the notBefore value in the certificate, being produced by a flawed process
which allows misleading values to be used, is useless for the purposes of
determining when the certificate was *actually* issued. Any TLS connection
using a WoSign-issued certificate which does not present SCTs would have to
be considered completely untrustworthy.

> For BR auditor, I think this issue is too technical that fewer auditor can
> find out this problem.

I believe Ryan has done an excellent job of outlining the concerns with this
statement.

> We will add the quality control system to PKI system before issuing the
> certificate, and will check the crt.sh or use the CABF lint and X590 Lint
> to check the certificate before and after the certificate is issued to
> prevent such case, if such case happen, we will notify all browsers
> instantly.

This neither addresses the question I posed, nor does it contain sufficient
specificity to be reassuring. I asked:

> what changes, exactly, has WoSign implemented to its policies and
> procedures to ensure that all trust programs in which WoSign is a
> participant are notified of future incidents, in line with each program's
> requirements?

I'm after the specifics of the changes to WoSign's policies and procedures
regarding *notification*, not quality control. What were WoSign's previous
policies and procedures regarding notification (obviously there was
something in place, since Google was notified), and what changes have been
made to improve those policies to ensure that all root programs are notified
in line with each program's requirements in the future?

- Matt

--
I told [my daughter] that if I see her digging a hole that she might not be
able to crawl out of, my job isn't to stand back and say "That's a *real*
nice hole you're digging there".
-- Paul Tomblin, ASR

Ryan Sleevi

unread,
Aug 25, 2016, 8:16:07 PM8/25/16
to mozilla-dev-s...@lists.mozilla.org
On Thursday, August 25, 2016 at 4:35:39 PM UTC-7, Matt Palmer wrote:
> I'm after the specifics of the changes to WoSign's policies and procedures
> regarding *notification*, not quality control. What were WoSign's previous
> policies and procedures regarding notification (obviously there was
> something in place, since Google was notified), and what changes have been
> made to improve those policies to ensure that all root programs are notified
> in line with each program's requirements in the future?

Clarification: In none of these incidents was Google notified proactively by WoSign. Instead, Google received communication from internal or external researchers regarding these issues, either prior to resolution or much later after the fact, and subsequently contacted WoSign regarding them.

It was only when Google found out recently that other programs were NOT notified, proactively, as had been expected, that Google shared the details it was aware of regarding various CA incidents, including those of WoSign, mentioned in this thread.

Matt Palmer

unread,
Aug 25, 2016, 10:03:50 PM8/25/16
to dev-secur...@lists.mozilla.org
On Thu, Aug 25, 2016 at 05:15:58PM -0700, Ryan Sleevi wrote:
> On Thursday, August 25, 2016 at 4:35:39 PM UTC-7, Matt Palmer wrote:
> > I'm after the specifics of the changes to WoSign's policies and procedures
> > regarding *notification*, not quality control. What were WoSign's previous
> > policies and procedures regarding notification (obviously there was
> > something in place, since Google was notified), and what changes have been
> > made to improve those policies to ensure that all root programs are notified
> > in line with each program's requirements in the future?
>
> Clarification: In none of these incidents was Google notified proactively
> by WoSign. Instead, Google received communication from internal or
> external researchers regarding these issues, either prior to resolution or
> much later after the fact, and subsequently contacted WoSign regarding
> them.

Oh, wow. I totally misread the initial report. That's almost certainly
much *worse*, then, because it's not simply a matter of some adjustments to
an existing process to improve it, but likely the development and deployment
of an entirely new process.

- Matt

--
The main advantages of Haynes and Chilton manuals are that they cost $15,
where the factory manuals cost $100 and up, and that they will tell you how
to use two hammers, a block of wood, and a meerkat to replace "special tool
no. 2-112-A" -- Matt Roberds in asr.

Richard Wang

unread,
Aug 25, 2016, 11:05:46 PM8/25/16
to Ryan Sleevi, mozilla-dev-s...@lists.mozilla.org
See below inline, thanks.

Best Regards,

Richard

-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Ryan Sleevi
Sent: Friday, August 26, 2016 3:10 AM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign

On Thursday, August 25, 2016 at 12:14:10 AM UTC-7, Richard Wang wrote:
> We can post all 2015 issued SSL certificate to CT log server if necessary.

Is there any reason not to do that proactively?

R: OK, we will post all 2015 issued SSL certificates to CT log server, but this take time since we issued 115K SSL certificate in 2015.
Now we are posting the (1) using higher level port validated orders related to incident 0, total 72 certificates. To be clear, those certificates are validated by website control validation method that using other port except 80 and 443; (2) Mis-issued certificate with un-validated subdomain related to incident 1, total 33 certificates. I will list all crt.sh URL to this mail thread.
Some certificates are revoked after getting report from subscriber, but some still valid, if any subscriber think it must be revoked and replaced new one, please contact us in the system, thanks.


> For BR auditor, I think this issue is too technical that fewer auditor can find out this problem.

The audit letter included an attestation from Management that, during the time of the audit, management believed that the CA complied with the Baseline Requirements.

Management was aware of non-compliance, by virtue of revocation and system and procedural changes to align with compliance.

Thus, do you believe it was faithful and accurate for Management to warrant that the CA was operated in compliance with the BRs, given that Management was aware of incidents of non-compliance?

R: This is my fault that I think it is not serious enough to state in the assertion letter, now I know and every related employee know how to do this in the future.

Richard Wang

unread,
Aug 25, 2016, 11:32:50 PM8/25/16
to Matt Palmer, dev-secur...@lists.mozilla.org
See below inline, thanks


Best Regards,

Richard

-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Matt Palmer
Sent: Friday, August 26, 2016 7:35 AM
To: dev-secur...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign

On Thu, Aug 25, 2016 at 07:11:18AM +0000, Richard Wang wrote:
> We can post all 2015 issued SSL certificate to CT log server if necessary.

That doesn't provide any assurance, in the face of misleading notBefore values in certificates. Without strong assurances that whatever failure of systems or processes allowed misleading notBefore values to end up in certificates has been corrected, the only way to be sure that there are no shenanigans going on is for *all* certificates issued by WoSign,
*regardless* of their purported issuance date, to carry SCTs from a qualified log (or have them presented at TLS establishment), and have the "effective date" for the purposes of compliance with relevant standards and guidelines to be the date of the earliest SCT presented. This is because the notBefore value in the certificate, being produced by a flawed process which allows misleading values to be used, is useless for the purposes of determining when the certificate was *actually* issued. Any TLS connection using a WoSign-issued certificate which does not present SCTs would have to be considered completely untrustworthy.

R: As I declared in the Bugzilla, only two certificate is the backdated certificate that not issued by normal system, it is a API bug that found and used by Computest to get the two backdates certificate, not WoSign normally issued this two cert. And this two test certificate are revoked after we got report, we stopped this API and delete the bug code.

As we stated in WoSign website, browsers can distrust WoSign issued certificate after July 5th that no SCT embedded data, this is for full transparency.


> For BR auditor, I think this issue is too technical that fewer auditor
> can find out this problem.

I believe Ryan has done an excellent job of outlining the concerns with this statement.

> We will add the quality control system to PKI system before issuing
> the certificate, and will check the crt.sh or use the CABF lint and
> X590 Lint to check the certificate before and after the certificate is
> issued to prevent such case, if such case happen, we will notify all
> browsers instantly.

This neither addresses the question I posed, nor does it contain sufficient specificity to be reassuring. I asked:

> what changes, exactly, has WoSign implemented to its policies and
> procedures to ensure that all trust programs in which WoSign is a
> participant are notified of future incidents, in line with each
> program's requirements?

I'm after the specifics of the changes to WoSign's policies and procedures regarding *notification*, not quality control. What were WoSign's previous policies and procedures regarding notification (obviously there was something in place, since Google was notified), and what changes have been made to improve those policies to ensure that all root programs are notified in line with each program's requirements in the future?

R: Sorry, I don't say it clear, please forgive my bad English since my native language is Chinese.
As I said this is my fault that we don't understand the Mozilla policy clearly that we don't think we need to report. But now we are clear that all mis-issued certificate case and any reported bug related system change also need to report. I and every related employee all clear now, then we can guarantee we will do it well in the future.
Why we log all SSL certificate from July 5th is for full transparency to let all related parties can report to us in the first time after the certificate is issued.


- Matt

--
I told [my daughter] that if I see her digging a hole that she might not be able to crawl out of, my job isn't to stand back and say "That's a *real* nice hole you're digging there".
-- Paul Tomblin, ASR

Richard Wang

unread,
Aug 25, 2016, 11:35:51 PM8/25/16
to Ryan Sleevi, mozilla-dev-s...@lists.mozilla.org
Yes, sorry for this.

As I admitted that this discussion gives us a big lesson that we know when we need to report incident to all browsers. We guarantee we will do it better.


Best Regards,

Richard

-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Ryan Sleevi
Sent: Friday, August 26, 2016 8:16 AM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign

On Thursday, August 25, 2016 at 4:35:39 PM UTC-7, Matt Palmer wrote:
> I'm after the specifics of the changes to WoSign's policies and
> procedures regarding *notification*, not quality control. What were
> WoSign's previous policies and procedures regarding notification
> (obviously there was something in place, since Google was notified),
> and what changes have been made to improve those policies to ensure
> that all root programs are notified in line with each program's requirements in the future?

Clarification: In none of these incidents was Google notified proactively by WoSign. Instead, Google received communication from internal or external researchers regarding these issues, either prior to resolution or much later after the fact, and subsequently contacted WoSign regarding them.

It was only when Google found out recently that other programs were NOT notified, proactively, as had been expected, that Google shared the details it was aware of regarding various CA incidents, including those of WoSign, mentioned in this thread.

Richard Wang

unread,
Aug 25, 2016, 11:40:23 PM8/25/16
to Matt Palmer, dev-secur...@lists.mozilla.org
We know how to do in the future, and believe me we will do this better.


Best Regards,

Richard

-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Matt Palmer
Sent: Friday, August 26, 2016 10:03 AM
To: dev-secur...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign

On Thu, Aug 25, 2016 at 05:15:58PM -0700, Ryan Sleevi wrote:
> On Thursday, August 25, 2016 at 4:35:39 PM UTC-7, Matt Palmer wrote:
> > I'm after the specifics of the changes to WoSign's policies and
> > procedures regarding *notification*, not quality control. What were
> > WoSign's previous policies and procedures regarding notification
> > (obviously there was something in place, since Google was notified),
> > and what changes have been made to improve those policies to ensure
> > that all root programs are notified in line with each program's requirements in the future?
>
> Clarification: In none of these incidents was Google notified
> proactively by WoSign. Instead, Google received communication from
> internal or external researchers regarding these issues, either prior
> to resolution or much later after the fact, and subsequently contacted
> WoSign regarding them.

Oh, wow. I totally misread the initial report. That's almost certainly much *worse*, then, because it's not simply a matter of some adjustments to an existing process to improve it, but likely the development and deployment of an entirely new process.

- Matt

--
The main advantages of Haynes and Chilton manuals are that they cost $15, where the factory manuals cost $100 and up, and that they will tell you how to use two hammers, a block of wood, and a meerkat to replace "special tool
no. 2-112-A" -- Matt Roberds in asr.

percy...@gmail.com

unread,
Aug 26, 2016, 4:05:14 AM8/26/16
to mozilla-dev-s...@lists.mozilla.org
The news about possible sanction against WoSign was reported by Solidot http://www.solidot.org/story?sid=49448
(the Chinese version of Slashdot). Out of 12 comments in total (at the time of writing), 8 of them call for revocation of WoSign, the rest talks about the general bad security practices in China.

A quick intro of myself.
I'm Percy Alpha and I broke the news on GFW's MITM attack on iCloud, Outlook and Yahoo in 2014 and was later the victim of Great Cannon attack in 2015.

percy...@gmail.com

unread,
Aug 26, 2016, 4:15:57 AM8/26/16
to mozilla-dev-s...@lists.mozilla.org
In most Chinese institutions, most checks and verifications are just formality. Contracting to the case of CNNIC CA, I'm not advocating for an outright removal of WoSign (even though I revoked the CA personally). But the incorrect notBefore date suggests that a mandatory inclusion of CT of all certs ever issued is needed. Of course, WoSign needs to address other issues raised by Matt and Ryan in addition to the CT requirement.

Richard Wang

unread,
Aug 26, 2016, 4:26:26 AM8/26/16
to percy...@gmail.com, mozilla-dev-s...@lists.mozilla.org
This is the standard way in China Internet, if a west company say something to China company, all will support the west company.

PLEASE don’t move this technical problem to political issue, thanks.

Best Regards,

Richard

-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of percy...@gmail.com
Sent: Friday, August 26, 2016 4:05 PM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign

233sec Team

unread,
Aug 26, 2016, 3:57:56 PM8/26/16
to mozilla-dev-s...@lists.mozilla.org
Wosign's Issue mechanism is high risking for large enterprise.
This is one prove:

https://gist.github.com/xiaohuilam/8589f2dfaac435bae4bf8dfe0984f69e

Alicdn.com is the cdn asset domain name of Taobao/tmall who belong to alibaba, which are Chinese biggest online shopping websites.
With the fake cert's middle man attack, password stealing, information leaking...

Jonathan Rudenberg

unread,
Aug 26, 2016, 4:24:44 PM8/26/16
to 233sec Team, mozilla-dev-s...@lists.mozilla.org
Here’s the crt.sh link for this certificate: https://crt.sh/?id=29884704

Can you provide more details about where this certificate came from? Did you issue it using one of the vulnerabilities discussed in this thread?

Richard Wang

unread,
Aug 29, 2016, 12:49:33 AM8/29/16
to Ryan Sleevi, mozilla-dev-s...@lists.mozilla.org
On Thursday, August 25, 2016 at 12:14:10 AM UTC-7, Richard Wang wrote:
> We can post all 2015 issued SSL certificate to CT log server if necessary.

Is there any reason not to do that proactively?

R: OK, we will post all 2015 issued SSL certificates to CT log server, but this take time since we issued 115K SSL certificate in 2015.

For incident 1 - mis-issued certificate with un-validated subdomain, total 33 certificates. We have posted to CT log server and listed in crt.sh, here is the URL. Some certificates are revoked after getting report from subscriber, but some still valid, if any subscriber think it must be revoked and replaced new one, please contact us in the system, thanks.
https://crt.sh/?id=7036355
https://crt.sh/?id=29805552
https://crt.sh/?id=7678955
https://crt.sh/?id=29805553
https://crt.sh/?id=29805554
https://crt.sh/?id=29805555
https://crt.sh/?id=29805556
https://crt.sh/?id=6798197
https://crt.sh/?id=29805558
https://crt.sh/?id=6798107
https://crt.sh/?id=29805559
https://crt.sh/?id=7728281
https://crt.sh/?id=29805560
https://crt.sh/?id=6639307
https://crt.sh/?id=29805561
https://crt.sh/?id=29805562
https://crt.sh/?id=6805650
https://crt.sh/?id=6739981
https://crt.sh/?id=7966229
https://crt.sh/?id=7094833
https://crt.sh/?id=29805563
https://crt.sh/?id=29805564
https://crt.sh/?id=29805565
https://crt.sh/?id=29805566
https://crt.sh/?id=29805567
https://crt.sh/?id=6843440
https://crt.sh/?id=29805568
https://crt.sh/?id=6999366
https://crt.sh/?id=29805569
https://crt.sh/?id=9534934
https://crt.sh/?id=29806448
https://crt.sh/?id=29813139
https://crt.sh/?id=29647048

For incident 0, the certificate issued related using higher level port validated, total 72 certificates. To be clear, those certificates are validated by website control validation method that using other port except 80 and 443. So we think those certificate no need to be revoked. The crt.sh link just for your reference.
https://crt.sh/?id=29805572
https://crt.sh/?id=7022909
https://crt.sh/?id=7564839
https://crt.sh/?id=29805573
https://crt.sh/?id=29805574
https://crt.sh/?id=29805575
https://crt.sh/?id=29805576
https://crt.sh/?id=29805577
https://crt.sh/?id=6969460
https://crt.sh/?id=29805578
https://crt.sh/?id=29805579
https://crt.sh/?id=29805580
https://crt.sh/?id=29805581
https://crt.sh/?id=29805582
https://crt.sh/?id=29805584
https://crt.sh/?id=29805585
https://crt.sh/?id=29805586
https://crt.sh/?id=9911443
https://crt.sh/?id=29805587
https://crt.sh/?id=7122803
https://crt.sh/?id=29805588
https://crt.sh/?id=29805589
https://crt.sh/?id=9985267
https://crt.sh/?id=29805590
https://crt.sh/?id=29805591
https://crt.sh/?id=29805592
https://crt.sh/?id=29805593
https://crt.sh/?id=29805594
https://crt.sh/?id=7224978
https://crt.sh/?id=10917791
https://crt.sh/?id=29805595
https://crt.sh/?id=29805596
https://crt.sh/?id=29805597
https://crt.sh/?id=6788465
https://crt.sh/?id=7224923
https://crt.sh/?id=9169568
https://crt.sh/?id=6836953
https://crt.sh/?id=29805598
https://crt.sh/?id=8172756
https://crt.sh/?id=29805599
https://crt.sh/?id=29805600
https://crt.sh/?id=7021184
https://crt.sh/?id=29805601
https://crt.sh/?id=29805602
https://crt.sh/?id=29805603
https://crt.sh/?id=29805604
https://crt.sh/?id=6927114
https://crt.sh/?id=6777468
https://crt.sh/?id=9793847
https://crt.sh/?id=29805605
https://crt.sh/?id=29805606
https://crt.sh/?id=29805607
https://crt.sh/?id=29805608
https://crt.sh/?id=9932344
https://crt.sh/?id=29805609
https://crt.sh/?id=29805610
https://crt.sh/?id=6880740
https://crt.sh/?id=29805611
https://crt.sh/?id=29805612
https://crt.sh/?id=7015627
https://crt.sh/?id=10008992
https://crt.sh/?id=29805613
https://crt.sh/?id=29805614
https://crt.sh/?id=29805615
https://crt.sh/?id=29805616
https://crt.sh/?id=7046181
https://crt.sh/?id=29805617
https://crt.sh/?id=29805618
https://crt.sh/?id=29805619
https://crt.sh/?id=7121749
https://crt.sh/?id=29805620
https://crt.sh/?id=6999366

Best Regards,

Richard

Gervase Markham

unread,
Aug 29, 2016, 4:27:59 AM8/29/16
to Peter Bowen, Richard Wang, Jeremy Rowley
On 24/08/16 17:44, Peter Bowen wrote:
> I think you are missing the most likely option: CA hosting. My
> understanding is that it is not uncommon that one CA operator
> contracts with another CA operator to run a CA on behalf of the first
> operator. I don't think it has been clear what disclosure of this
> practice is required. Given that I believe this is widespread, I
> assumed that all of the issuing CAs in this case were operated by the
> same entity.

If StartCom are hosting WoSign's infra (seems less likely), then it's
still a pretty severe mistake to accidentally issue a certificate from
one of your customer's roots rather than your own, although one might
say the mistake in this case would be StartCom's.

If WoSign are hosting StartCom's infra, it still leaves open the
question of why StartCom are deploying code that WoSign are no longer
using, and haven't for six months, and why WoSign permitted the StartCom
UI to issue WoSign certificates at all.

Gerv


Gervase Markham

unread,
Aug 29, 2016, 4:33:25 AM8/29/16
to Richard Wang, Jeremy Rowley
On 25/08/16 04:38, Richard Wang wrote:
> R: NOT this case you think. Due to root inclusion problem, WoSign
> root is cross signed by StartCom since 2011. And we shared some
> facility with StartCom like CRL and OCSP distribution etc. But not
> this case, as I declared in the previous email, this is a API
> parameter option that can post data to any server located in any
> place.

At what point in this process is the domain control validation done?

Gerv

Gervase Markham

unread,
Aug 29, 2016, 4:38:01 AM8/29/16
to Richard Wang, Ryan Sleevi
On 26/08/16 04:33, Richard Wang wrote:
> As I admitted that this discussion gives us a big lesson that we know
> when we need to report incident to all browsers. We guarantee we will
> do it better.

Richard,

You have been involved in this (Mozilla) discussion group and in the CAB
Forum for several years. In that time, you will have seen multiple CAs
disclose instances where certificates were misissued, and you will have
seen root programs take such disclosures seriously and consider them
important. Did it not occur to you that the same standard of disclosure
that everyone else was using should also apply to WoSign?

Gerv

Gervase Markham

unread,
Aug 29, 2016, 4:41:06 AM8/29/16
to Richard Wang, Ryan Sleevi
On 29/08/16 05:46, Richard Wang wrote:
> For incident 1 - mis-issued certificate with un-validated subdomain,
> total 33 certificates. We have posted to CT log server and listed in
> crt.sh, here is the URL. Some certificates are revoked after getting
> report from subscriber, but some still valid, if any subscriber think
> it must be revoked and replaced new one, please contact us in the
> system, thanks.

Er, no. If these certificates were issued with unvalidated parent
domains (e.g. with github.com when the person validation foo.github.com)
then they need to all be revoked. You should actively contact your
customers and issue them new certificates containing only validated
information, and then revoke these ones.

Gerv

Gervase Markham

unread,
Aug 29, 2016, 4:43:33 AM8/29/16
to 233sec Team
On 26/08/16 06:12, 233sec Team wrote:
> https://gist.github.com/xiaohuilam/8589f2dfaac435bae4bf8dfe0984f69e
>
> Alicdn.com is the cdn asset domain name of Taobao/tmall who belong to alibaba, which are Chinese biggest online shopping websites.
> With the fake cert's middle man attack, password stealing, information leaking...

Can you prove you have control of the private key associated with this
certificate?

Gerv

Patrick Figel

unread,
Aug 29, 2016, 9:55:50 AM8/29/16
to ric...@wosign.com, dev-secur...@lists.mozilla.org
Richard,

the problem with this approach is that the *subscriber* might not be
authorized to make this decision for the parent domain. To go back to
the GitHub case, the "owner" of a github.io subdomain telling you that
they are authorized to own a certificate that covers github.io is
irrelevant, as they have never demonstrated ownership of that domain.

The right approach would be to revoke the affected certificates
immediately and inform your subscribers that they will need to re-issue
their certificates (while also verifying ownership of the root domain).

Here's another similar case - cloudapp.net, which belongs to Microsoft
Azure. I'm fairly certain this certificate was not authorized by Microsoft:

https://crt.sh/?id=29805555

Thanks,

Patrick

On 29/08/16 11:30, Richard Wang wrote:
> Yes, we plan to revoke all after getting confirmation from
> subscriber. We are doing this.
>
> Regards,
>
> Richard
>
>> On 29 Aug 2016, at 16:38, Gervase Markham <ge...@mozilla.org>
>> wrote:
>>
>>> On 29/08/16 05:46, Richard Wang wrote: For incident 1 -
>>> mis-issued certificate with un-validated subdomain, total 33
>>> certificates. We have posted to CT log server and listed in
>>> crt.sh, here is the URL. Some certificates are revoked after
>>> getting report from subscriber, but some still valid, if any
>>> subscriber think it must be revoked and replaced new one, please
>>> contact us in the system, thanks.
>>
>> Er, no. If these certificates were issued with unvalidated parent
>> domains (e.g. with github.com when the person validation
>> foo.github.com) then they need to all be revoked. You should
>> actively contact your customers and issue them new certificates
>> containing only validated information, and then revoke these ones.
>>
>> Gerv
>>
>>

233sec Team

unread,
Aug 29, 2016, 12:08:03 PM8/29/16
to mozilla-dev-s...@lists.mozilla.org
Not vulnerabilities mentioned in this thread, but a Human-Audit weak process.
Detail you can see the reply content i send to Mr.Wang

在 2016年8月27日星期六 UTC+8上午4:24:44,Jonathan Rudenberg:

蓝小灰

unread,
Aug 29, 2016, 12:08:03 PM8/29/16
to Gervase Markham, mozilla-dev-s...@lists.mozilla.org
Of course I have private key of this certificate
Gervase Markham <ge...@mozilla.org>于2016年8月29日 周一16:42写道:

> On 26/08/16 06:12, 233sec Team wrote:
> > https://gist.github.com/xiaohuilam/8589f2dfaac435bae4bf8dfe0984f69e
> >
> > Alicdn.com is the cdn asset domain name of Taobao/tmall who belong to
> alibaba, which are Chinese biggest online shopping websites.
> > With the fake cert's middle man attack, password stealing, information
> leaking...
>

Tobias Sachs

unread,
Aug 29, 2016, 12:08:17 PM8/29/16
to mozilla-dev-s...@lists.mozilla.org
Is there any plan to revoke the certificate in OneCRL soon? And how we could speed this up? @ Kathleen Wilson

mar...@marcan.st

unread,
Aug 29, 2016, 12:08:36 PM8/29/16
to mozilla-dev-s...@lists.mozilla.org
Not only that, I see *two* certs issued for GitHub subdomains plus the parent domain:

https://crt.sh/?id=29647048
https://crt.sh/?id=29805567

Why wasn't this additional cert identified and disclosed prior to the aforementioned list being posted? It seems a no-brainer to manually audit the list for obvious cases of mis-issuance (i.e. not only the domain was not validated, but the customer clearly has no ability to validate it if asked).

xcrai...@gmail.com

unread,
Aug 29, 2016, 12:09:05 PM8/29/16
to mozilla-dev-s...@lists.mozilla.org
On Friday, August 26, 2016 at 4:26:26 PM UTC+8, Richard Wang wrote:
> This is the standard way in China Internet, if a west company say something to China company, all will support the west company.

-- especially when local CAs are losing credibility to end-users. Microsoft Azure's Chinese website[1] has migrated from CNNIC CA, to WoSign, and recently to DigiCert. CNNIC itself, ironically, also moved to DigiCert.

[1]: azure.cn

It is almost axiomatic that without a proper statement & fix (no more 233sec team exploits) made, WoSign will continue losing trust from end-users as well as webmasters.

> PLEASE don’t move this technical problem to political issue, thanks.

Very unfortunately WoSign's advertisements are seemingly doing the opposite. On this comparison between WoSign and foreign CAs[2], you made the following statements:

[2]: http://wayback.archive.org/web/20160828045112/https://www.wosign.com/about/WoSign_ForeignCA_compare.htm

* Security: handled by Chinese company itself, fully secure. (Foreign CA: System Security should not be a problem, but risks of random revokes and/or access failures exist.)
* Compliance with Chinese Law: Yes (Foreign CA: No, legal risks exist.)

Gervase Markham

unread,
Aug 29, 2016, 1:26:20 PM8/29/16
to mozilla-dev-s...@lists.mozilla.org
On 29/08/16 09:48, 蓝小灰 wrote:
> Of course I have private key of this certificate

I have asked 蓝小灰 for cryptographic proof of this.

Gerv


Percy

unread,
Aug 29, 2016, 5:53:25 PM8/29/16
to mozilla-dev-s...@lists.mozilla.org
Gerv, I've notified the security team in Alibaba about this possible fake cert and ask them to confirm that they have not applied a cert.
It's unlikely that Alibaba will use a free cert from WoSign. As a commercial site, they usually use Verisign or globalSign

Percy

unread,
Aug 29, 2016, 7:03:57 PM8/29/16
to mozilla-dev-s...@lists.mozilla.org
"Some certificates are revoked after getting report from subscriber, but some still valid, if any subscriber think it must be revoked and replaced new one, please contact us in the system, thanks"

WoSign seems to lack the basic understanding of how a certificate is used in authentication, consequently unfit to be a CA, I call for revocation of WoSign from all root programs immediately. http://www.percya.com/2016/08/chinese-ca-wosign-faces-revocation.html

dion...@gmail.com

unread,
Aug 30, 2016, 11:18:56 AM8/30/16
to mozilla-dev-s...@lists.mozilla.org
If I understand correctly, these 105 certificates are all mis-issued using the incorrect policies of either (0) website control validation with higher port numbers, or (1) parent domain-name verification by demonstrating control of a subdomain.

It is unclear why, given the fact that incorrect validation was done for these certificates, they are not already revoked. I don't understand why you are expecting a report from your respective subscriber to do that, as they have not proven control of the domain names in any case.

These certificates must all be revoked immediately.

On Monday, August 29, 2016 at 7:49:33 AM UTC+3, Richard Wang wrote:
>
> For incident 1 - mis-issued certificate with un-validated subdomain, total 33 certificates. We have posted to CT log server and listed in crt.sh, here is the URL. Some certificates are revoked after getting report from subscriber, but some still valid, if any subscriber think it must be revoked and replaced new one, please contact us in the system, thanks.
>

dymu...@gmail.com

unread,
Aug 30, 2016, 11:19:18 AM8/30/16
to mozilla-dev-s...@lists.mozilla.org
Both of those certificates were generated by me. The 'motorstoiclathe' cert was a pseudonym made up to do more testing. Note that it's only valid for github.io because the 'schrauger' account was grandfathered into an old DNS redirect, and the new account didn't have that rule. Hence, it wasn't able to be created for the .com domain.

It is interesting that WoSign followed the redirect. I suppose it is assumed that with a 301 permanent redirect that the new domain is controlled by the same person, but that seems a bit sketchy.

I had forgotten about the second github.io certificate, but looking back on my emails, it was also revoked the day after it was created.

Percy

unread,
Aug 30, 2016, 1:46:04 PM8/30/16
to mozilla-dev-s...@lists.mozilla.org
https://crt.sh is down. Maybe someone can check with comodo to see whether they got DDOSed?

Here are the Google CT for the possibly mis-issued certs mentioned in this thread. It would be a lot harder to take down the Google CT.

Possible fake cert for Github
https://www.google.com/transparencyreport/https/ct/#domain=github.io&incl_exp=false&incl_sub=false&issuer=lPrsb9Gbn4s%3D


Possible fake cert for Alibaba, the largest commercial site in China
https://www.google.com/transparencyreport/https/ct/#domain=alicdn.com&incl_exp=false&incl_sub=false&issuer=lPrsb9Gbn4s%3D

Possible fake cert for Microsoft
https://www.google.com/transparencyreport/https/ct/#domain=cloudapp.net&incl_exp=false&incl_sub=false&issuer=lPrsb9Gbn4s%3D

Rob Stradling

unread,
Aug 30, 2016, 4:43:10 PM8/30/16
to Percy, mozilla-dev-s...@lists.mozilla.org
On 30/08/16 18:45, Percy wrote:
> https://crt.sh is down. Maybe someone can check with comodo to see whether they got DDOSed?

Sorry about that. crt.sh is back up now.

It wasn't a DDOS attack.

Every so often something goes awry with the database replication
(between crt.sh's master database and front-end slave databases), which
causes all of the front-end databases to crash. Somebody (usually me,
but I've been out for most of today) is normally around to restart the
crashed databases. I don't know why our NOC team didn't fix this
several hours ago, but I intend to find out. Perhaps there are some
improvements we need to make to our internal monitoring systems.

> Here are the Google CT for the possibly mis-issued certs mentioned in this thread. It would be a lot harder to take down the Google CT.

I can't disagree with that statement. :-)

That said, I'll see what I can do to improve crt.sh's uptime. I already
have one offer of help...
https://twitter.com/FiloSottile/status/770642205304352768
--
Rob Stradling
Senior Research & Development Scientist
COMODO - Creating Trust Online

Nick Lamb

unread,
Aug 30, 2016, 9:19:46 PM8/30/16
to mozilla-dev-s...@lists.mozilla.org
On Tuesday, 30 August 2016 16:19:18 UTC+1, dymu...@gmail.com wrote:
> It is interesting that WoSign followed the redirect. I suppose it is assumed that with a 301 permanent redirect that the new domain is controlled by the same person, but that seems a bit sketchy.

Hmm. I think that if there's a 301 redirect in place for every URL then in practice the effect is that the target site does control the content of the origin site. Such circumstances already exist without redirects.

Suppose the site http://theclown.example.net/ functions as a thin proxy of another site, say http://famous-cloud-provider.example.com/ and simply copies the contents of equivalent resources at this other site, but replaces all instances of the word "cloud" in human-readable text with "clown". An amusing prank.

We can expect that the legitimate owner of famous-cloud-provider.example.com would be able to obtain a certificate for theclown.example.net by obeying either the typical ad hoc validation scheme of the sort which you experienced with WoSign, or a Ballot 169-compliant scheme like ACME. This is because it is very unlikely that the validation content would include the exact word "cloud" and so it would pass through the proxy unaltered and be readable by the CA's validation system when checking theclown.example.net

But this seems like no great loss, since in fact the contents of theclown.example.net are almost exactly controlled by famous-cloud-provider.example.com, the certificate's implications about who controls theclown.example.net are functionally true.

If the legitimate owner of theclown.example.net doesn't want this to be possible under prior ad hoc validation systems it's a bit tricky to prevent, but under Ballot 169 it's simply a matter of making /.well-known/ exempt from the proxy rule, which is anyway already a very good idea.

NB. If my musing inspires substantial further debate it should happen in another thread, not this one about WoSign.

Peter Bowen

unread,
Aug 30, 2016, 10:44:45 PM8/30/16
to Gervase Markham, Richard Wang, mozilla-dev-s...@lists.mozilla.org
On Wed, Aug 24, 2016 at 6:08 AM, Gervase Markham <ge...@mozilla.org> wrote:
> Dear m.d.s.policy,
>
> Several incidents have come to our attention involving the CA "WoSign".
> Mozilla is considering what action it should take in response to these
> incidents. This email sets out our understanding of the situation.
>
> Before we begin, we note that Section 1 of the Mozilla CA Certificate
> Enforcement Policy[0] says: "When a serious security concern is noticed,
> such as a major root compromise, it should be treated as a
> security-sensitive bug, and the Mozilla Policy for Handling Security
> Bugs should be followed." It is clear to us, and appears to be clear to
> other CAs based on their actions, that misissuances where domain control
> checks have failed fall into the category of "serious security concern".

I have run into another bug that appears to be fixed in WoSign's
infrastructure but is worth noting.

In April 2015, two different WoSign CAs issued multiple certificates
to distinct subjects using the same serial number. The CT logs have
picked up two instances of this occuring:

https://crt.sh/?serial=0D3BBDC3A0175E38F9D0070CD050986A shows eight
distinct certificates with the same serial number, all with notBefore
dates of 2015-04-14.

https://crt.sh/?serial=056D1570DA645BF6B44C0A7077CC6769 shows dozens
of distinct certificates with the same serial number, with notBefore
dates between 2015-04-10 and 2015-04-14.

I have not examined their management assertions to see if this was
documented and I do not know if this was reported to Mozilla at the
time. These certificates do not appear to meet RFC 5280's
requirements, which say:

"The serial number MUST be a positive integer assigned by the CA to
each certificate. It MUST be unique for each certificate issued by a
given CA (i.e., the issuer name and serial number identify a unique
certificate)"
(https://tools.ietf.org/html/rfc5280#section-4.1.2.2)

Was Mozilla advised of this issue?

Thanks,
Peter
Message has been deleted

Richard Wang

unread,
Aug 30, 2016, 11:07:49 PM8/30/16
to Peter Bowen, Gervase Markham, mozilla-dev-s...@lists.mozilla.org
This case is in the BR report: https://cert.webtrust.org/SealFile?seal=2019&file=pdf

Thanks.

Best Regards,

Richard

-----Original Message-----
From: Peter Bowen [mailto:pzb...@gmail.com]
Sent: Wednesday, August 31, 2016 10:45 AM
To: Gervase Markham <ge...@mozilla.org>
Cc: mozilla-dev-s...@lists.mozilla.org; Richard Wang <ric...@wosign.com>
Subject: Re: Incidents involving the CA WoSign

Percy

unread,
Aug 31, 2016, 4:49:23 AM8/31/16
to mozilla-dev-s...@lists.mozilla.org
On Tuesday, August 30, 2016 at 7:47:43 PM UTC-7, itk9...@gmail.com wrote:
> Wosign indirectly bought StartSSL, https://www.letsphish.org


Ha! It makes so much sense now why StartEncrypt is such a catastrophe(https://www.google.com/search?q=StartEncrypt). I've revoked all StarCom certs in my OS. Thanks for the heads up!

Peter Gutmann

unread,
Aug 31, 2016, 5:17:08 AM8/31/16
to itk9...@gmail.com, mozilla-dev-s...@lists.mozilla.org
itk9...@gmail.com <itk9...@gmail.com> writes:

>Wosign indirectly bought StartSSL, https://www.letsphish.org

Has there been any independent investigation into this? We know that CAs are
bought and sold like baseball trading cards, but it's usually done publicly
and freely acknowledged, whereas this one seems to have been done in a
somewhat underhanded manner...

Peter.

Gervase Markham

unread,
Aug 31, 2016, 6:16:29 AM8/31/16
to Percy
On 29/08/16 22:53, Percy wrote:
> Gerv, I've notified the security team in Alibaba about this possible fake cert and ask them to confirm that they have not applied a cert.
> It's unlikely that Alibaba will use a free cert from WoSign. As a commercial site, they usually use Verisign or globalSign

That might also help; thank you. Please ask them to contact me directly
to confirm this cert was not requested by them.

Gerv

s...@samspin.net

unread,
Aug 31, 2016, 8:42:45 AM8/31/16
to mozilla-dev-s...@lists.mozilla.org
To the policymakers at Mozilla, my name is Samuel Pinder.
I consider myself an computer network analyst and have a degree in Web Systems Development. I also host a small number of websites on a technical level. I have used Startcom's services for a number of years. I only recently came across WoSign when I discovered that they offer 3-year valid certificates for free for up to 5 hostnames. That seemed quite an excessively long time in my opinion for a free certificate, almost too good to be true. As a customer of effectively both WoSign and Startcom, I have been having gradually growing concerns for some time about their recent practices and this recent news has not helped. I am currently looking into using other CA's for my needs but will watch what happens next closely before making a decision. I firmly believe WoSign have almost 'purchased' Startcom as I noticed that the nameserver domain for startssl.com (namely startcomca.net) would bring up WoSign's website for a time when accessed via a web browser as www.startcomca.net. I personally emailed Eddy Nigg (Startcom founder) about this, and this was eventually changed to redirect to startssl.com. So it is obvious that WoSign is hosting/sharing Startcom's infrastucture on it's own CA hardware/software, which Eddie confirmed in his email reply. The SSL certificate at the time for www.startcom.org had expired for over a week and still does have many broken links. Using the 'Live Chat' on www.startssl.com had a very confused employee tell me that startcom.org is no longer the website for Startcom, but startssl.com is. I stressed that this was confusing to people and that they should redirect the latter if that's the case. However... startcom.org remains using Israeli-based nameservers. Just how closely related Startcom is to WoSign is unknown, but it does mean that almost anything that renders WoSign vulnerable, also applies to Startcom. I would hope that Startcom re-evaluate their relationship with WoSign following this mess at the very least.
Regarding WoSign in particular, the SHA-1 miss-issued certificates does look like a genuine mistake, being issued off of an SHA-2 intermediate makes their existence rather pointless in the case of backward compatibility, as old operating systems need a whole SHA-1 chain to validate properly. So there is no obvious motive for this API functionality to be left there on purpose, unless.... a rogue employee/subcontractor left it in place for when a SHA-1 hash collision becomes possible? What is likely here is that SHA-1 used to be the default signature choice, and was indeed legacy functionality left behind. It is very concerning however that despite WoSign having an external audit (which all CA's are meant to have) it appears nobody has independently performed a code audit of their API backend to prevent this being possible in a production environment. With regards to the mis-issued certificates applying to whole base domains validated only sub-domain, that prospect is horrifying. With regards to WoSign's API following 301 redirects from one domain to another though, I have noticed that Let's Encrypt's API (another CA at letsencrypt.org) also has this behaviour. This may be standard behaviour of the ACME protocol. Perhaps it would be prudent to decide in overall policy whether 301 redirects of verification URLs are acceptable. There are legitimate reasons why this would happen, for example if a customer owned www.example1.com and www.example2.net, but wanted the latter to always redirect to the other (because for example of a name change, but had many hardcoded https:// links), they would otherwise have to temporarily disable the redirect to get an SSL certificate covering both domains when validating with this method. With regards to ports other than 80 and 443 and 8080 being used for validation, I think the motive of this is because China has a mandatory media licencing requirement (ICP licence) for all servers that operate on either ports. It may not be practical for an individual in China to get one of these licences just to be able to gain an SSL certificate with this validation method. Of course allowing higher numbered ports does not give much excuse. I tested WoSign's website today however and can confirm that they have removed URL-based file verification, and now only allow DNS or email (from WHOIS) validation. All that's well and good, but the fact remains that there could be many other mis-issued certificates already in existence and the scope remains unknown at this point just how many. We only have WoSign's word that all of the mis-issued certificates are known and/or revoked. If it cannot be certain that WoSign have revoked all incorrectly-validated certificates, then the obvious choice is to add the "WoSign CA Free SSL Certificate G2" intermediate to OneCRL, and possibly their DV intermediate too, as these intermediates are known to have mis-issued certificates whilst their other intermediates have always had tighter validation requirements for OV and EV validation. That approach would certainly be painful for existing users and would require WoSign to immediately contact current and past customers to have new certificates issued from a new intermediate, once it is certain that all the bugs in the validation procedures have been resolved. Would it be appropriate to give customers time before such action is taken, given that it would break many websites, or does the risk of data leaks from unknown mis-issued certificates outweigh the need to give customers time? The other option of simply preventing any newly-issued certificates from being accepted in browers does not seem a very good idea given that WoSign have been known to backdate certificates. In fact, I would even suggest that WoSign cease issuing free certificates (and most possibly paid DV certificates) and in the meantime have an independent audit of their backend to ensure all vulnerablities are closed, with the results made public. Only then should
they recommence issuance- at least on an automated scale. GlobalSign did something similar to this when there was speculation that the hacker responsible for compromising Diginotar had also breached them, at great risk to their business but ultimately gained respect in the process. Given the severity of these incidents and the fact they went unreported though I would not be surprised if WoSign's whole roots are blacklisted eventually if WoSign do not get their act together. The promise to completely commit to Certificate Transparency is quite a step, but it does not change what has already happened. WoSign's recent announcement gives me the impression they expect others to manually scour the CT logs once they upload their 2015 certificate data for mis-issuances that need revoking rather than do this themselves. Whilst it may *technically* be 'transparent' with CT, they don't seem in the spirit of things being completely transparent with their practices- in particular the relationship between Startcom and WoSign. At least one of these incidents actually occurred during research into Startcom, and it turned out the API was being shared by both CA's. Given the circumstances, I think it is appropriate to ask just what the relationship is between them. For me, it appears they are now one and the same, while the original 'Startcom' infratructure I once trusted is gathering dust and has been abandoned. I think I'll be taking my needs to another CA very soon if things don't look up!
Samuel Pinder

On Wednesday, 24 August 2016 14:09:02 UTC+1, Gervase Markham wrote:
> Dear m.d.s.policy,
>
> Several incidents have come to our attention involving the CA "WoSign".
> Mozilla is considering what action it should take in response to these
> incidents. This email sets out our understanding of the situation.
>
> Before we begin, we note that Section 1 of the Mozilla CA Certificate
> Enforcement Policy[0] says: "When a serious security concern is noticed,
> such as a major root compromise, it should be treated as a
> security-sensitive bug, and the Mozilla Policy for Handling Security
> Bugs should be followed." It is clear to us, and appears to be clear to
> other CAs based on their actions, that misissuances where domain control
> checks have failed fall into the category of "serious security concern".
>
> Incident 0
> ----------
>
> On or around April 23rd, 2015, WoSign's certificate issuance system for
> their free certificates allowed the applicant to choose any port for
> validation. Once validation had been completed, WoSign would
> issue certificates for that domain. A researcher was able to obtain a
> certificate for a university by opening a high-numbered port (>50,000)
> and getting WoSign to use that port for validation of control.
>
> This problem was reported to Google, and thence to WoSign and resolved.
> Mozilla only became aware of it recently.
>
> * Before the recent passage of Ballot 169 in the CAB Forum, which limits
> the ports and paths which can be used, the Baseline Requirements said
> that one acceptable method of domain validation was "Having the
> Applicant demonstrate practical control over the FQDN by making an
> agreed‐upon change to information found on an online Web page identified
> by a uniform resource identifier containing the FQDN". This method
> therefore did not violate the letter of the BRs. However, Mozilla
> considers the basic security knowledge that ports over 1024 are
> unprivileged should have led all CAs not to accept validations of domain
> control on such ports, even when not documented in the BRs.
>
> * The misissuance incident was not reported to Mozilla by WoSign as it
> should have been (see above).
>
> * This misissuance incident did not turn up on WoSign's subsequent BR
> audit[1].
>
> Incident 1
> ----------
>
> In June 2015, an applicant found a problem with WoSign's free
> certificate service, which allowed them to get a certificate for the
> base domain if they were able to prove control of a subdomain.
>
> The reporter proved the problem in two ways. They accidentally
> discovered it when trying to get a certificate for med.ucf.edu and
> mistakenly also applied for www.ucf.edu, which was approved. They then
> confirmed the problem by using their control of
> theiraccount.github.com/theiraccount.github.io to get a cert for
> github.com, github.io, and www.github.io.
>
> They reported this to WoSign, giving only the Github certificate as an
> example. That cert was revoked and the vulnerability was fixed. However
> recently, they got in touch with Google to note that the ucf.edu cert
> still had not been revoked almost a year later.
>
> * The lack of revocation of the ucf.edu certificate (still unrevoked at
> time of writing, although it may have been by time of posting) strongly
> suggests that WoSign either did not or could not search their issuance
> databases for other occurrences of the same problem. Mozilla considers
> such a search a basic part of the response to disclosure of a
> vulnerability which causes misissuance, and expects CAs to keep records
> detailed enough to make it possible.
>
> * This misissuance incident was not reported to Mozilla by WoSign as it
> should have been (see above).
>
> * This misissuance incident did not turn up on WoSign's subsequent BR
> audit[1].
>
> Incident 2
> ----------
>
> In July 2016, it became clear that there was some problems with the
> StartEncrypt automatic issuance service recently deployed by the CA
> StartCom. As well as other problems it had, which are outside the scope
> of this discussion, changing a simple API parameter in the POST request
> on the submission page changed the root certificate to which the
> resulting certificate chained up. The value "2" made a certificate
> signed by "StartCom Class 1 DV Server CA", "1" selected "WoSign CA Free
> SSL Certificate G2" and "0" selected "CA 沃通根证书", another root
> certificate owned by WoSign and trusted by Firefox.
>
> Using the value "1" led to a certificate which had a notBefore date
> (usage start date) of 20th December 2015, and which was signed using the
> SHA-1 checksum algorithm.
>
> * The issuance of certificates using SHA-1 has been banned by the
> Baseline Requirements since January 1st, 2016. Browsers, including
> Firefox, planned to enforce this[2] by not trusting certs with a
> notBefore date after that date, but in the case of Firefox the fix had
> to be backed out due to web compatibility issues. However, we are
> considering how/when to reintroduce it, and CAs presumably know this.
>
> * The issuance of backdated certificates is not forbidden, but is listed
> in Mozilla's list of Problematic Practices[3]. It says "Minor tweaking
> for technical compatibility reasons is accepted, but backdating
> certificates in order to avoid some deadline or code-enforced
> restriction is not."
>
> * WoSign deny that their code backdated the certificates in order to
> avoid browser-based restrictions - they say "this date is the day we
> stop to use this code"[4]. If that is true, it is not clear to us how
> StartCom came to deploy WoSign code that WoSign itself had abandoned.
>
> * It seems clear from publicly available information that StartCom's
> issuance systems are linked to WoSign's issuance systems in some way.
> Nevertheless, it should not have been possible for an application for a
> cert from StartCom to produce a cert signed by WoSign.
>
> * This misissuance incident was not reported to Mozilla by WoSign as it
> should have been.
>
>
> Taking into account all these incidents and the actions of this CA,
> Mozilla is considering what action to take. Your input is welcomed.
>
> Gerv, Kathleen and Richard
>
>
> [0]
> https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/enforcement/
> [1] https://cert.webtrust.org/SealFile?seal=2019&file=pdf
> [2] https://bugzilla.mozilla.org/show_bug.cgi?id=942515
> [3]
> https://wiki.mozilla.org/CA:Problematic_Practices#Backdating_the_notBefore_date
> [4] https://bugzilla.mozilla.org/show_bug.cgi?id=1293366

Gervase Markham

unread,
Aug 31, 2016, 9:45:09 AM8/31/16
to mozilla-dev-s...@lists.mozilla.org
On 24/08/16 14:08, Gervase Markham wrote:
> * The issuance of certificates using SHA-1 has been banned by the
> Baseline Requirements since January 1st, 2016. Browsers, including
> Firefox, planned to enforce this[2] by not trusting certs with a
> notBefore date after that date, but in the case of Firefox the fix had
> to be backed out due to web compatibility issues.

Just as a note, this information is incomplete - the enforcement
returned for publicly-trusted CAs in bug
https://bugzilla.mozilla.org/show_bug.cgi?id=1254667 , since Firefox 48.

Gerv

ian....@gmail.com

unread,
Aug 31, 2016, 1:06:45 PM8/31/16