info
Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Incidents involving the CA WoSign
23,759 views
Skip to first unread message
Gervase Markham
Aug 24, 2016, 9:09:02 AM8/24/16
to mozilla-dev-s...@lists.mozilla.org, Richard Wang
Dear m.d.s.policy,
Several incidents have come to our attention involving the CA "WoSign".
Mozilla is considering what action it should take in response to these
incidents. This email sets out our understanding of the situation.
Before we begin, we note that Section 1 of the Mozilla CA Certificate
Enforcement Policy[0] says: "When a serious security concern is noticed,
such as a major root compromise, it should be treated as a
security-sensitive bug, and the Mozilla Policy for Handling Security
Bugs should be followed." It is clear to us, and appears to be clear to
other CAs based on their actions, that misissuances where domain control
checks have failed fall into the category of "serious security concern".
Incident 0
----------
On or around April 23rd, 2015, WoSign's certificate issuance system for
their free certificates allowed the applicant to choose any port for
validation. Once validation had been completed, WoSign would
issue certificates for that domain. A researcher was able to obtain a
certificate for a university by opening a high-numbered port (>50,000)
and getting WoSign to use that port for validation of control.
This problem was reported to Google, and thence to WoSign and resolved.
Mozilla only became aware of it recently.
* Before the recent passage of Ballot 169 in the CAB Forum, which limits
the ports and paths which can be used, the Baseline Requirements said
that one acceptable method of domain validation was "Having the
Applicant demonstrate practical control over the FQDN by making an
agreed‐upon change to information found on an online Web page identified
by a uniform resource identifier containing the FQDN". This method
therefore did not violate the letter of the BRs. However, Mozilla
considers the basic security knowledge that ports over 1024 are
unprivileged should have led all CAs not to accept validations of domain
control on such ports, even when not documented in the BRs.
* The misissuance incident was not reported to Mozilla by WoSign as it
should have been (see above).
* This misissuance incident did not turn up on WoSign's subsequent BR
audit[1].
Incident 1
----------
In June 2015, an applicant found a problem with WoSign's free
certificate service, which allowed them to get a certificate for the
base domain if they were able to prove control of a subdomain.
The reporter proved the problem in two ways. They accidentally
discovered it when trying to get a certificate for med.ucf.edu and
mistakenly also applied for www.ucf.edu, which was approved. They then
confirmed the problem by using their control of
theiraccount.github.com/theiraccount.github.io to get a cert for
github.com, github.io, and www.github.io.
They reported this to WoSign, giving only the Github certificate as an
example. That cert was revoked and the vulnerability was fixed. However
recently, they got in touch with Google to note that the ucf.edu cert
still had not been revoked almost a year later.
* The lack of revocation of the ucf.edu certificate (still unrevoked at
time of writing, although it may have been by time of posting) strongly
suggests that WoSign either did not or could not search their issuance
databases for other occurrences of the same problem. Mozilla considers
such a search a basic part of the response to disclosure of a
vulnerability which causes misissuance, and expects CAs to keep records
detailed enough to make it possible.
* This misissuance incident was not reported to Mozilla by WoSign as it
should have been (see above).
* This misissuance incident did not turn up on WoSign's subsequent BR
audit[1].
Incident 2
----------
In July 2016, it became clear that there was some problems with the
StartEncrypt automatic issuance service recently deployed by the CA
StartCom. As well as other problems it had, which are outside the scope
of this discussion, changing a simple API parameter in the POST request
on the submission page changed the root certificate to which the
resulting certificate chained up. The value "2" made a certificate
signed by "StartCom Class 1 DV Server CA", "1" selected "WoSign CA Free
SSL Certificate G2" and "0" selected "CA 沃通根证书", another root
certificate owned by WoSign and trusted by Firefox.
Using the value "1" led to a certificate which had a notBefore date
(usage start date) of 20th December 2015, and which was signed using the
SHA-1 checksum algorithm.
* The issuance of certificates using SHA-1 has been banned by the
Baseline Requirements since January 1st, 2016. Browsers, including
Firefox, planned to enforce this[2] by not trusting certs with a
notBefore date after that date, but in the case of Firefox the fix had
to be backed out due to web compatibility issues. However, we are
considering how/when to reintroduce it, and CAs presumably know this.
* The issuance of backdated certificates is not forbidden, but is listed
in Mozilla's list of Problematic Practices[3]. It says "Minor tweaking
for technical compatibility reasons is accepted, but backdating
certificates in order to avoid some deadline or code-enforced
restriction is not."
* WoSign deny that their code backdated the certificates in order to
avoid browser-based restrictions - they say "this date is the day we
stop to use this code"[4]. If that is true, it is not clear to us how
StartCom came to deploy WoSign code that WoSign itself had abandoned.
* It seems clear from publicly available information that StartCom's
issuance systems are linked to WoSign's issuance systems in some way.
Nevertheless, it should not have been possible for an application for a
cert from StartCom to produce a cert signed by WoSign.
* This misissuance incident was not reported to Mozilla by WoSign as it
should have been.
Taking into account all these incidents and the actions of this CA,
Mozilla is considering what action to take. Your input is welcomed.
Gerv, Kathleen and Richard
[0]
https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/enforcement/
[1] https://cert.webtrust.org/SealFile?seal=2019&file=pdf
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=942515
[3]
https://wiki.mozilla.org/CA:Problematic_Practices#Backdating_the_notBefore_date
[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1293366
Several incidents have come to our attention involving the CA "WoSign".
Mozilla is considering what action it should take in response to these
incidents. This email sets out our understanding of the situation.
Before we begin, we note that Section 1 of the Mozilla CA Certificate
Enforcement Policy[0] says: "When a serious security concern is noticed,
such as a major root compromise, it should be treated as a
security-sensitive bug, and the Mozilla Policy for Handling Security
Bugs should be followed." It is clear to us, and appears to be clear to
other CAs based on their actions, that misissuances where domain control
checks have failed fall into the category of "serious security concern".
Incident 0
----------
On or around April 23rd, 2015, WoSign's certificate issuance system for
their free certificates allowed the applicant to choose any port for
validation. Once validation had been completed, WoSign would
issue certificates for that domain. A researcher was able to obtain a
certificate for a university by opening a high-numbered port (>50,000)
and getting WoSign to use that port for validation of control.
This problem was reported to Google, and thence to WoSign and resolved.
Mozilla only became aware of it recently.
* Before the recent passage of Ballot 169 in the CAB Forum, which limits
the ports and paths which can be used, the Baseline Requirements said
that one acceptable method of domain validation was "Having the
Applicant demonstrate practical control over the FQDN by making an
agreed‐upon change to information found on an online Web page identified
by a uniform resource identifier containing the FQDN". This method
therefore did not violate the letter of the BRs. However, Mozilla
considers the basic security knowledge that ports over 1024 are
unprivileged should have led all CAs not to accept validations of domain
control on such ports, even when not documented in the BRs.
* The misissuance incident was not reported to Mozilla by WoSign as it
should have been (see above).
* This misissuance incident did not turn up on WoSign's subsequent BR
audit[1].
Incident 1
----------
In June 2015, an applicant found a problem with WoSign's free
certificate service, which allowed them to get a certificate for the
base domain if they were able to prove control of a subdomain.
The reporter proved the problem in two ways. They accidentally
discovered it when trying to get a certificate for med.ucf.edu and
mistakenly also applied for www.ucf.edu, which was approved. They then
confirmed the problem by using their control of
theiraccount.github.com/theiraccount.github.io to get a cert for
github.com, github.io, and www.github.io.
They reported this to WoSign, giving only the Github certificate as an
example. That cert was revoked and the vulnerability was fixed. However
recently, they got in touch with Google to note that the ucf.edu cert
still had not been revoked almost a year later.
* The lack of revocation of the ucf.edu certificate (still unrevoked at
time of writing, although it may have been by time of posting) strongly
suggests that WoSign either did not or could not search their issuance
databases for other occurrences of the same problem. Mozilla considers
such a search a basic part of the response to disclosure of a
vulnerability which causes misissuance, and expects CAs to keep records
detailed enough to make it possible.
* This misissuance incident was not reported to Mozilla by WoSign as it
should have been (see above).
* This misissuance incident did not turn up on WoSign's subsequent BR
audit[1].
Incident 2
----------
In July 2016, it became clear that there was some problems with the
StartEncrypt automatic issuance service recently deployed by the CA
StartCom. As well as other problems it had, which are outside the scope
of this discussion, changing a simple API parameter in the POST request
on the submission page changed the root certificate to which the
resulting certificate chained up. The value "2" made a certificate
signed by "StartCom Class 1 DV Server CA", "1" selected "WoSign CA Free
SSL Certificate G2" and "0" selected "CA 沃通根证书", another root
certificate owned by WoSign and trusted by Firefox.
Using the value "1" led to a certificate which had a notBefore date
(usage start date) of 20th December 2015, and which was signed using the
SHA-1 checksum algorithm.
* The issuance of certificates using SHA-1 has been banned by the
Baseline Requirements since January 1st, 2016. Browsers, including
Firefox, planned to enforce this[2] by not trusting certs with a
notBefore date after that date, but in the case of Firefox the fix had
to be backed out due to web compatibility issues. However, we are
considering how/when to reintroduce it, and CAs presumably know this.
* The issuance of backdated certificates is not forbidden, but is listed
in Mozilla's list of Problematic Practices[3]. It says "Minor tweaking
for technical compatibility reasons is accepted, but backdating
certificates in order to avoid some deadline or code-enforced
restriction is not."
* WoSign deny that their code backdated the certificates in order to
avoid browser-based restrictions - they say "this date is the day we
stop to use this code"[4]. If that is true, it is not clear to us how
StartCom came to deploy WoSign code that WoSign itself had abandoned.
* It seems clear from publicly available information that StartCom's
issuance systems are linked to WoSign's issuance systems in some way.
Nevertheless, it should not have been possible for an application for a
cert from StartCom to produce a cert signed by WoSign.
* This misissuance incident was not reported to Mozilla by WoSign as it
should have been.
Taking into account all these incidents and the actions of this CA,
Mozilla is considering what action to take. Your input is welcomed.
Gerv, Kathleen and Richard
[0]
https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/enforcement/
[1] https://cert.webtrust.org/SealFile?seal=2019&file=pdf
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=942515
[3]
https://wiki.mozilla.org/CA:Problematic_Practices#Backdating_the_notBefore_date
[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1293366
Eric Mill
Aug 24, 2016, 12:09:39 PM8/24/16
to Gervase Markham, Richard Wang, mozilla-dev-s...@lists.mozilla.org
For the thread's reference, here's the crt.sh link for the misissued GitHub
certificate:
https://crt.sh/?id=29647048
Valid for 3 years, for github.com. It's not in OneCRL, CRLset, or
Microsoft's disallowedcert.stl.
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>
--
konklone.com | @konklone <https://twitter.com/konklone>
certificate:
https://crt.sh/?id=29647048
Valid for 3 years, for github.com. It's not in OneCRL, CRLset, or
Microsoft's disallowedcert.stl.
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>
--
konklone.com | @konklone <https://twitter.com/konklone>
Gervase Markham
Aug 24, 2016, 12:31:26 PM8/24/16
to Jeremy Rowley, Richard Wang
Hi Jeremy,
On 24/08/16 17:12, Jeremy Rowley wrote:
> On incident 0, its unclear whether a cert was actually mis-issued.
> Although they used a higher level port, did the researcher
> successfully bypass WoSign's domain validation process? Is the only
> concern that WoSign permitted higher level ports?
The result of the incident was that a certificate was issued to someone
who did not, in the normally understood sense of the word, have control
of the domain in question. Mozilla feels that even without a specific
injunction in the BRs, CAs should have known that ports > 1024 are not
privileged and not done control checks using them.
The severity of the problem, of course, is a matter for discussion here.
> On incident 2, it sounds like they are both using the same
> auto-generation script.
It seems like a bit more than that, doesn't it? Let's presume that
WoSign did not ship a copy of their intermediate cert's private key to
StartCom. Therefore, this cert must have been issued on the back end by
some sort of WoSign system. So either WoSign's back-end issuing service
has some form of authentication and the StartCom system had those
credentials (why?), or the WoSign system does not have any form of
authentication (concerning).
> Giving WoSign the benefit of the doubt, it
> sounds like maybe it was a bug in their software that permitted SHA1
> certs not an intentional back-dating issue. Is there any clarity
> around how this worked?
Perhaps WoSign would like to provide some :-)
Gerv
On 24/08/16 17:12, Jeremy Rowley wrote:
> On incident 0, its unclear whether a cert was actually mis-issued.
> Although they used a higher level port, did the researcher
> successfully bypass WoSign's domain validation process? Is the only
> concern that WoSign permitted higher level ports?
The result of the incident was that a certificate was issued to someone
who did not, in the normally understood sense of the word, have control
of the domain in question. Mozilla feels that even without a specific
injunction in the BRs, CAs should have known that ports > 1024 are not
privileged and not done control checks using them.
The severity of the problem, of course, is a matter for discussion here.
> On incident 2, it sounds like they are both using the same
> auto-generation script.
It seems like a bit more than that, doesn't it? Let's presume that
WoSign did not ship a copy of their intermediate cert's private key to
StartCom. Therefore, this cert must have been issued on the back end by
some sort of WoSign system. So either WoSign's back-end issuing service
has some form of authentication and the StartCom system had those
credentials (why?), or the WoSign system does not have any form of
authentication (concerning).
> Giving WoSign the benefit of the doubt, it
> sounds like maybe it was a bug in their software that permitted SHA1
> certs not an intentional back-dating issue. Is there any clarity
> around how this worked?
Perhaps WoSign would like to provide some :-)
Gerv
Peter Bowen
Aug 24, 2016, 12:44:44 PM8/24/16
to Gervase Markham, Richard Wang, mozilla-dev-s...@lists.mozilla.org, Jeremy Rowley
On Wed, Aug 24, 2016 at 9:30 AM, Gervase Markham <ge...@mozilla.org> wrote:
> On 24/08/16 17:12, Jeremy Rowley wrote:
>> On incident 2, it sounds like they are both using the same
>> auto-generation script.
>
> It seems like a bit more than that, doesn't it? Let's presume that
> WoSign did not ship a copy of their intermediate cert's private key to
> StartCom. Therefore, this cert must have been issued on the back end by
> some sort of WoSign system. So either WoSign's back-end issuing service
> has some form of authentication and the StartCom system had those
> credentials (why?), or the WoSign system does not have any form of
> authentication (concerning).
I think you are missing the most likely option: CA hosting. My
> On 24/08/16 17:12, Jeremy Rowley wrote:
>> On incident 2, it sounds like they are both using the same
>> auto-generation script.
>
> It seems like a bit more than that, doesn't it? Let's presume that
> WoSign did not ship a copy of their intermediate cert's private key to
> StartCom. Therefore, this cert must have been issued on the back end by
> some sort of WoSign system. So either WoSign's back-end issuing service
> has some form of authentication and the StartCom system had those
> credentials (why?), or the WoSign system does not have any form of
> authentication (concerning).
understanding is that it is not uncommon that one CA operator
contracts with another CA operator to run a CA on behalf of the first
operator. I don't think it has been clear what disclosure of this
practice is required. Given that I believe this is widespread, I
assumed that all of the issuing CAs in this case were operated by the
same entity.
Thanks,
Peter
Ryan Sleevi
Aug 24, 2016, 4:19:41 PM8/24/16
to Jeremy Rowley, Richard Wang, mozilla-dev-s...@lists.mozilla.org, Gervase Markham, Peter Bowen
On Wed, Aug 24, 2016 at 12:40 PM, Jeremy Rowley
<jeremy...@digicert.com> wrote:
> However, the fact a researcher was able to obtain a cert without proper domain
> validation is pretty serious. I'd like to hear more details about how this was
> accomplished. Ports 8080 and 8443 aren't that uncommon so penalizing someone
> merely for port use seems harsh when there wasn't a policy against it.
There was no restriction on ports at all. Any client-specified port
was accepted, and any HTTP-like response it gave back was accepted.
<jeremy...@digicert.com> wrote:
> However, the fact a researcher was able to obtain a cert without proper domain
> validation is pretty serious. I'd like to hear more details about how this was
> accomplished. Ports 8080 and 8443 aren't that uncommon so penalizing someone
> merely for port use seems harsh when there wasn't a policy against it.
There was no restriction on ports at all. Any client-specified port
was accepted, and any HTTP-like response it gave back was accepted.
s...@gmx.ch
Aug 24, 2016, 8:22:59 PM8/24/16
to dev-secur...@lists.mozilla.org
Of course, adding the affected certs to OneCRL should be done immediately.
WoSign also has to be transparent about all (mis) issued certs in the
past and have to provide this info in the future.
If they can't, I think we may consider if the current certs that are
valid for 3 years should be restricted to a shorter period.
Regards,
Jonas
WoSign also has to be transparent about all (mis) issued certs in the
past and have to provide this info in the future.
If they can't, I think we may consider if the current certs that are
valid for 3 years should be restricted to a shorter period.
Regards,
Jonas
Richard Wang
Aug 24, 2016, 10:58:06 PM8/24/16
to Eric Mill, Gervase Markham, mozilla-dev-s...@lists.mozilla.org
this cert is revoked in the same once it is issued.
Thanks for posting to CT.
Best Regards,
Richard
From: Eric Mill [mailto:er...@konklone.com]
Sent: Thursday, August 25, 2016 12:08 AM
To: Gervase Markham <ge...@mozilla.org>
Cc: mozilla-dev-s...@lists.mozilla.org; Richard Wang <ric...@wosign.com>
Subject: Re: Incidents involving the CA WoSign
For the thread's reference, here's the crt.sh link for the misissued GitHub certificate:
https://crt.sh/?id=29647048
Valid for 3 years, for github.com<http://github.com>. It's not in OneCRL, CRLset, or Microsoft's disallowedcert.stl.
discovered it when trying to get a certificate for med.ucf.edu<http://med.ucf.edu> and
mistakenly also applied for www.ucf.edu<http://www.ucf.edu>, which was approved. They then
github.com<http://github.com>, github.io<http://github.io>, and www.github.io<http://www.github.io>.
They reported this to WoSign, giving only the Github certificate as an
example. That cert was revoked and the vulnerability was fixed. However
recently, they got in touch with Google to note that the ucf.edu<http://ucf.edu> cert
https://lists.mozilla.org/listinfo/dev-security-policy
--
konklone.com<https://konklone.com> | @konklone<https://twitter.com/konklone>
Thanks for posting to CT.
Best Regards,
Richard
From: Eric Mill [mailto:er...@konklone.com]
Sent: Thursday, August 25, 2016 12:08 AM
To: Gervase Markham <ge...@mozilla.org>
Cc: mozilla-dev-s...@lists.mozilla.org; Richard Wang <ric...@wosign.com>
Subject: Re: Incidents involving the CA WoSign
For the thread's reference, here's the crt.sh link for the misissued GitHub certificate:
https://crt.sh/?id=29647048
mistakenly also applied for www.ucf.edu<http://www.ucf.edu>, which was approved. They then
confirmed the problem by using their control of
theiraccount.github.com/theiraccount.github.io<http://theiraccount.github.com/theiraccount.github.io> to get a cert for
github.com<http://github.com>, github.io<http://github.io>, and www.github.io<http://www.github.io>.
They reported this to WoSign, giving only the Github certificate as an
example. That cert was revoked and the vulnerability was fixed. However
still had not been revoked almost a year later.
* The lack of revocation of the ucf.edu<http://ucf.edu> certificate (still unrevoked at
Taking into account all these incidents and the actions of this CA,
Mozilla is considering what action to take. Your input is welcomed.
Mozilla is considering what action to take. Your input is welcomed.
Gerv, Kathleen and Richard
[0]
https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/enforcement/
[1] https://cert.webtrust.org/SealFile?seal=2019&file=pdf
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=942515
[3]
https://wiki.mozilla.org/CA:Problematic_Practices#Backdating_the_notBefore_date
[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1293366
_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>
[0]
https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/enforcement/
[1] https://cert.webtrust.org/SealFile?seal=2019&file=pdf
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=942515
[3]
https://wiki.mozilla.org/CA:Problematic_Practices#Backdating_the_notBefore_date
[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1293366
_______________________________________________
dev-security-policy mailing list
https://lists.mozilla.org/listinfo/dev-security-policy
--
konklone.com<https://konklone.com> | @konklone<https://twitter.com/konklone>
Richard Wang
Aug 24, 2016, 11:40:53 PM8/24/16
to Gervase Markham, Jeremy Rowley, mozilla-dev-s...@lists.mozilla.org
See below inline, thanks.
Best Regards,
Richard
-----Original Message-----
From: Gervase Markham [mailto:ge...@mozilla.org]
Sent: Thursday, August 25, 2016 12:31 AM
To: Jeremy Rowley <jeremy...@digicert.com>; mozilla-dev-s...@lists.mozilla.org
Cc: Richard Wang <ric...@wosign.com>
Subject: Re: Incidents involving the CA WoSign
Hi Jeremy,
On 24/08/16 17:12, Jeremy Rowley wrote:
We are searching our database to try to find if any mis-issued cert is issued. And will update to this mail list.
> On incident 2, it sounds like they are both using the same
> auto-generation script.
It seems like a bit more than that, doesn't it? Let's presume that WoSign did not ship a copy of their intermediate cert's private key to StartCom. Therefore, this cert must have been issued on the back end by some sort of WoSign system. So either WoSign's back-end issuing service has some form of authentication and the StartCom system had those credentials (why?), or the WoSign system does not have any form of authentication (concerning).
R: NOT this case you think. Due to root inclusion problem, WoSign root is cross signed by StartCom since 2011. And we shared some facility with StartCom like CRL and OCSP distribution etc. But not this case, as I declared in the previous email, this is a API parameter option that can post data to any server located in any place.
> Giving WoSign the benefit of the doubt, it sounds like maybe it was a
> bug in their software that permitted SHA1 certs not an intentional
> back-dating issue. Is there any clarity around how this worked?
Perhaps WoSign would like to provide some :-)
R: I think we said clearly in the related Bugzilla ☺
Gerv
Best Regards,
Richard
-----Original Message-----
From: Gervase Markham [mailto:ge...@mozilla.org]
Sent: Thursday, August 25, 2016 12:31 AM
To: Jeremy Rowley <jeremy...@digicert.com>; mozilla-dev-s...@lists.mozilla.org
Cc: Richard Wang <ric...@wosign.com>
Subject: Re: Incidents involving the CA WoSign
On 24/08/16 17:12, Jeremy Rowley wrote:
> On incident 0, its unclear whether a cert was actually mis-issued.
> Although they used a higher level port, did the researcher
> successfully bypass WoSign's domain validation process? Is the only
> concern that WoSign permitted higher level ports?
The result of the incident was that a certificate was issued to someone who did not, in the normally understood sense of the word, have control of the domain in question. Mozilla feels that even without a specific injunction in the BRs, CAs should have known that ports > 1024 are not privileged and not done control checks using them.
The severity of the problem, of course, is a matter for discussion here.
R: We got report from Google, but don’t tell me if any mis-issued certificate happen, just tell us to forbidden this high level port.
> Although they used a higher level port, did the researcher
> successfully bypass WoSign's domain validation process? Is the only
> concern that WoSign permitted higher level ports?
The result of the incident was that a certificate was issued to someone who did not, in the normally understood sense of the word, have control of the domain in question. Mozilla feels that even without a specific injunction in the BRs, CAs should have known that ports > 1024 are not privileged and not done control checks using them.
The severity of the problem, of course, is a matter for discussion here.
We are searching our database to try to find if any mis-issued cert is issued. And will update to this mail list.
> On incident 2, it sounds like they are both using the same
> auto-generation script.
It seems like a bit more than that, doesn't it? Let's presume that WoSign did not ship a copy of their intermediate cert's private key to StartCom. Therefore, this cert must have been issued on the back end by some sort of WoSign system. So either WoSign's back-end issuing service has some form of authentication and the StartCom system had those credentials (why?), or the WoSign system does not have any form of authentication (concerning).
> Giving WoSign the benefit of the doubt, it sounds like maybe it was a
> bug in their software that permitted SHA1 certs not an intentional
> back-dating issue. Is there any clarity around how this worked?
Perhaps WoSign would like to provide some :-)
Gerv
Richard Wang
Aug 24, 2016, 11:42:44 PM8/24/16
to Peter Bowen, Gervase Markham, mozilla-dev-s...@lists.mozilla.org, Jeremy Rowley
Yes, correct.
Due to root inclusion problem, WoSign root is cross signed by StartCom since 2011. And we shared some facility with StartCom like CRL and OCSP distribution etc. But not this case, as I declared in the previous email, this is a API parameter option that can post data to any server located in any place.
Due to root inclusion problem, WoSign root is cross signed by StartCom since 2011. And we shared some facility with StartCom like CRL and OCSP distribution etc. But not this case, as I declared in the previous email, this is a API parameter option that can post data to any server located in any place.
Best Regards,
Richard
-----Original Message-----
Richard
-----Original Message-----
From: Peter Bowen [mailto:pzb...@gmail.com]
Sent: Thursday, August 25, 2016 12:45 AM
To: Gervase Markham <ge...@mozilla.org>
Sent: Thursday, August 25, 2016 12:45 AM
To: Gervase Markham <ge...@mozilla.org>
Cc: Jeremy Rowley <jeremy...@digicert.com>; mozilla-dev-s...@lists.mozilla.org; Richard Wang <ric...@wosign.com>
Subject: Re: Incidents involving the CA WoSign
Subject: Re: Incidents involving the CA WoSign
On Wed, Aug 24, 2016 at 9:30 AM, Gervase Markham <ge...@mozilla.org> wrote:
> On 24/08/16 17:12, Jeremy Rowley wrote:
>> On incident 2, it sounds like they are both using the same
>> auto-generation script.
>
> It seems like a bit more than that, doesn't it? Let's presume that
> WoSign did not ship a copy of their intermediate cert's private key to
> StartCom. Therefore, this cert must have been issued on the back end
> by some sort of WoSign system. So either WoSign's back-end issuing
> service has some form of authentication and the StartCom system had
> those credentials (why?), or the WoSign system does not have any form
> of authentication (concerning).
> On 24/08/16 17:12, Jeremy Rowley wrote:
>> On incident 2, it sounds like they are both using the same
>> auto-generation script.
>
> It seems like a bit more than that, doesn't it? Let's presume that
> WoSign did not ship a copy of their intermediate cert's private key to
> StartCom. Therefore, this cert must have been issued on the back end
> by some sort of WoSign system. So either WoSign's back-end issuing
> service has some form of authentication and the StartCom system had
> those credentials (why?), or the WoSign system does not have any form
> of authentication (concerning).
Richard Wang
Aug 25, 2016, 12:05:32 AM8/25/16
to s...@gmx.ch, dev-secur...@lists.mozilla.org
We revoked this certificate, and we know this certificate is for test only.
For transparency, WoSign announced full transparency for all SSL certificate from July 5th that post all issued SSL certificate to Google log server, browsers can distrust WoSign issued SSL certificate after that day if no SCT embedded data in the certificate.
And WoSign even plan to post the code signing certificate and client certificate to log server for full transparency for all certificates.
See this news if you missed: https://www.wosign.com/english/News/2016_wosign_CT.htm.
And we plan to setup an free alert service for worldwide users that if any SSL certificate for domain you care is issued from any CA, then you can get the alert email, this will benfit the ecosystem.
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of s...@gmx.ch
Sent: Thursday, August 25, 2016 8:18 AM
To: dev-secur...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
For transparency, WoSign announced full transparency for all SSL certificate from July 5th that post all issued SSL certificate to Google log server, browsers can distrust WoSign issued SSL certificate after that day if no SCT embedded data in the certificate.
And WoSign even plan to post the code signing certificate and client certificate to log server for full transparency for all certificates.
See this news if you missed: https://www.wosign.com/english/News/2016_wosign_CT.htm.
And we plan to setup an free alert service for worldwide users that if any SSL certificate for domain you care is issued from any CA, then you can get the alert email, this will benfit the ecosystem.
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of s...@gmx.ch
Sent: Thursday, August 25, 2016 8:18 AM
To: dev-secur...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
Of course, adding the affected certs to OneCRL should be done immediately.
WoSign also has to be transparent about all (mis) issued certs in the past and have to provide this info in the future.
If they can't, I think we may consider if the current certs that are valid for 3 years should be restricted to a shorter period.
Regards,
Jonas
WoSign also has to be transparent about all (mis) issued certs in the past and have to provide this info in the future.
If they can't, I think we may consider if the current certs that are valid for 3 years should be restricted to a shorter period.
Regards,
Jonas
> For the thread's reference, here's the crt.sh link for the misissued
> GitHub
> certificate:
>
> https://crt.sh/?id=29647048
>
> Valid for 3 years, for github.com. It's not in OneCRL, CRLset, or
> GitHub
> certificate:
>
> https://crt.sh/?id=29647048
>
Matt Palmer
Aug 25, 2016, 2:48:30 AM8/25/16
to dev-secur...@lists.mozilla.org
On Thu, Aug 25, 2016 at 04:03:04AM +0000, Richard Wang wrote:
> For transparency, WoSign announced full transparency for all SSL
> certificate from July 5th that post all issued SSL certificate to Google
> log server, browsers can distrust WoSign issued SSL certificate after that
> day if no SCT embedded data in the certificate.
That would be slightly more reassuring if there wasn't a history of certs
> For transparency, WoSign announced full transparency for all SSL
> certificate from July 5th that post all issued SSL certificate to Google
> log server, browsers can distrust WoSign issued SSL certificate after that
> day if no SCT embedded data in the certificate.
being issued with seemingly misleading notBefore values...
Separately, do you have any thoughts on the reports that WoSign's BR auditor
did not note any of the misissuances? Also, what changes, exactly, has
WoSign implemented to its policies and procedures to ensure that all trust
programs in which WoSign is a participant are notified of future incidents,
in line with each program's requirements?
- Matt
--
"The user-friendly computer is a red herring. The user-friendliness of a
book just makes it easier to turn pages. There's nothing user-friendly about
learning to read."
-- Alan Kay
Richard Wang
Aug 25, 2016, 3:14:10 AM8/25/16
to Matt Palmer, dev-secur...@lists.mozilla.org
Thanks for your friendly reminder.
We can post all 2015 issued SSL certificate to CT log server if necessary.
For BR auditor, I think this issue is too technical that fewer auditor can find out this problem.
We will add the quality control system to PKI system before issuing the certificate, and will check the crt.sh or use the CABF lint and X590 Lint to check the certificate before and after the certificate is issued to prevent such case, if such case happen, we will notify all browsers instantly.
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Matt Palmer
Sent: Thursday, August 25, 2016 2:48 PM
To: dev-secur...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
We can post all 2015 issued SSL certificate to CT log server if necessary.
For BR auditor, I think this issue is too technical that fewer auditor can find out this problem.
We will add the quality control system to PKI system before issuing the certificate, and will check the crt.sh or use the CABF lint and X590 Lint to check the certificate before and after the certificate is issued to prevent such case, if such case happen, we will notify all browsers instantly.
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Matt Palmer
Sent: Thursday, August 25, 2016 2:48 PM
To: dev-secur...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy
dev-security-policy mailing list
dev-secur...@lists.mozilla.org
Ryan Sleevi
Aug 25, 2016, 3:09:59 PM8/25/16
to mozilla-dev-s...@lists.mozilla.org
On Thursday, August 25, 2016 at 12:14:10 AM UTC-7, Richard Wang wrote:
> We can post all 2015 issued SSL certificate to CT log server if necessary.
Is there any reason not to do that proactively?
> We can post all 2015 issued SSL certificate to CT log server if necessary.
> For BR auditor, I think this issue is too technical that fewer auditor can find out this problem.
Management was aware of non-compliance, by virtue of revocation and system and procedural changes to align with compliance.
Thus, do you believe it was faithful and accurate for Management to warrant that the CA was operated in compliance with the BRs, given that Management was aware of incidents of non-compliance?
Matt Palmer
Aug 25, 2016, 7:35:39 PM8/25/16
to dev-secur...@lists.mozilla.org
On Thu, Aug 25, 2016 at 07:11:18AM +0000, Richard Wang wrote:
> We can post all 2015 issued SSL certificate to CT log server if necessary.
That doesn't provide any assurance, in the face of misleading notBefore
> We can post all 2015 issued SSL certificate to CT log server if necessary.
values in certificates. Without strong assurances that whatever failure of
systems or processes allowed misleading notBefore values to end up in
certificates has been corrected, the only way to be sure that there are no
shenanigans going on is for *all* certificates issued by WoSign,
*regardless* of their purported issuance date, to carry SCTs from a
qualified log (or have them presented at TLS establishment), and have the
"effective date" for the purposes of compliance with relevant standards and
guidelines to be the date of the earliest SCT presented. This is because
the notBefore value in the certificate, being produced by a flawed process
which allows misleading values to be used, is useless for the purposes of
determining when the certificate was *actually* issued. Any TLS connection
using a WoSign-issued certificate which does not present SCTs would have to
be considered completely untrustworthy.
> For BR auditor, I think this issue is too technical that fewer auditor can
> find out this problem.
statement.
> We will add the quality control system to PKI system before issuing the
> certificate, and will check the crt.sh or use the CABF lint and X590 Lint
> to check the certificate before and after the certificate is issued to
> prevent such case, if such case happen, we will notify all browsers
> instantly.
specificity to be reassuring. I asked:
> what changes, exactly, has WoSign implemented to its policies and
> procedures to ensure that all trust programs in which WoSign is a
> participant are notified of future incidents, in line with each program's
> requirements?
regarding *notification*, not quality control. What were WoSign's previous
policies and procedures regarding notification (obviously there was
something in place, since Google was notified), and what changes have been
made to improve those policies to ensure that all root programs are notified
in line with each program's requirements in the future?
- Matt
--
I told [my daughter] that if I see her digging a hole that she might not be
able to crawl out of, my job isn't to stand back and say "That's a *real*
nice hole you're digging there".
-- Paul Tomblin, ASR
Ryan Sleevi
Aug 25, 2016, 8:16:07 PM8/25/16
to mozilla-dev-s...@lists.mozilla.org
On Thursday, August 25, 2016 at 4:35:39 PM UTC-7, Matt Palmer wrote:
> I'm after the specifics of the changes to WoSign's policies and procedures
> regarding *notification*, not quality control. What were WoSign's previous
> policies and procedures regarding notification (obviously there was
> something in place, since Google was notified), and what changes have been
> made to improve those policies to ensure that all root programs are notified
> in line with each program's requirements in the future?
Clarification: In none of these incidents was Google notified proactively by WoSign. Instead, Google received communication from internal or external researchers regarding these issues, either prior to resolution or much later after the fact, and subsequently contacted WoSign regarding them.
> I'm after the specifics of the changes to WoSign's policies and procedures
> regarding *notification*, not quality control. What were WoSign's previous
> policies and procedures regarding notification (obviously there was
> something in place, since Google was notified), and what changes have been
> made to improve those policies to ensure that all root programs are notified
> in line with each program's requirements in the future?
It was only when Google found out recently that other programs were NOT notified, proactively, as had been expected, that Google shared the details it was aware of regarding various CA incidents, including those of WoSign, mentioned in this thread.
Matt Palmer
Aug 25, 2016, 10:03:50 PM8/25/16
to dev-secur...@lists.mozilla.org
On Thu, Aug 25, 2016 at 05:15:58PM -0700, Ryan Sleevi wrote:
> On Thursday, August 25, 2016 at 4:35:39 PM UTC-7, Matt Palmer wrote:
> On Thursday, August 25, 2016 at 4:35:39 PM UTC-7, Matt Palmer wrote:
> > I'm after the specifics of the changes to WoSign's policies and procedures
> > regarding *notification*, not quality control. What were WoSign's previous
> > policies and procedures regarding notification (obviously there was
> > something in place, since Google was notified), and what changes have been
> > made to improve those policies to ensure that all root programs are notified
> > in line with each program's requirements in the future?
>
> > regarding *notification*, not quality control. What were WoSign's previous
> > policies and procedures regarding notification (obviously there was
> > something in place, since Google was notified), and what changes have been
> > made to improve those policies to ensure that all root programs are notified
> > in line with each program's requirements in the future?
>
> Clarification: In none of these incidents was Google notified proactively
> by WoSign. Instead, Google received communication from internal or
> external researchers regarding these issues, either prior to resolution or
> much later after the fact, and subsequently contacted WoSign regarding
> them.
Oh, wow. I totally misread the initial report. That's almost certainly
> by WoSign. Instead, Google received communication from internal or
> external researchers regarding these issues, either prior to resolution or
> much later after the fact, and subsequently contacted WoSign regarding
> them.
much *worse*, then, because it's not simply a matter of some adjustments to
an existing process to improve it, but likely the development and deployment
of an entirely new process.
- Matt
--
The main advantages of Haynes and Chilton manuals are that they cost $15,
where the factory manuals cost $100 and up, and that they will tell you how
to use two hammers, a block of wood, and a meerkat to replace "special tool
no. 2-112-A" -- Matt Roberds in asr.
Richard Wang
Aug 25, 2016, 11:05:46 PM8/25/16
to Ryan Sleevi, mozilla-dev-s...@lists.mozilla.org
See below inline, thanks.
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Ryan Sleevi
Sent: Friday, August 26, 2016 3:10 AM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
Now we are posting the (1) using higher level port validated orders related to incident 0, total 72 certificates. To be clear, those certificates are validated by website control validation method that using other port except 80 and 443; (2) Mis-issued certificate with un-validated subdomain related to incident 1, total 33 certificates. I will list all crt.sh URL to this mail thread.
Some certificates are revoked after getting report from subscriber, but some still valid, if any subscriber think it must be revoked and replaced new one, please contact us in the system, thanks.
> For BR auditor, I think this issue is too technical that fewer auditor can find out this problem.
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Ryan Sleevi
Sent: Friday, August 26, 2016 3:10 AM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
On Thursday, August 25, 2016 at 12:14:10 AM UTC-7, Richard Wang wrote:
> We can post all 2015 issued SSL certificate to CT log server if necessary.
Is there any reason not to do that proactively?
R: OK, we will post all 2015 issued SSL certificates to CT log server, but this take time since we issued 115K SSL certificate in 2015.
Now we are posting the (1) using higher level port validated orders related to incident 0, total 72 certificates. To be clear, those certificates are validated by website control validation method that using other port except 80 and 443; (2) Mis-issued certificate with un-validated subdomain related to incident 1, total 33 certificates. I will list all crt.sh URL to this mail thread.
Some certificates are revoked after getting report from subscriber, but some still valid, if any subscriber think it must be revoked and replaced new one, please contact us in the system, thanks.
> For BR auditor, I think this issue is too technical that fewer auditor can find out this problem.
The audit letter included an attestation from Management that, during the time of the audit, management believed that the CA complied with the Baseline Requirements.
Management was aware of non-compliance, by virtue of revocation and system and procedural changes to align with compliance.
Thus, do you believe it was faithful and accurate for Management to warrant that the CA was operated in compliance with the BRs, given that Management was aware of incidents of non-compliance?
R: This is my fault that I think it is not serious enough to state in the assertion letter, now I know and every related employee know how to do this in the future.
Management was aware of non-compliance, by virtue of revocation and system and procedural changes to align with compliance.
Thus, do you believe it was faithful and accurate for Management to warrant that the CA was operated in compliance with the BRs, given that Management was aware of incidents of non-compliance?
Richard Wang
Aug 25, 2016, 11:32:50 PM8/25/16
to Matt Palmer, dev-secur...@lists.mozilla.org
See below inline, thanks
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Matt Palmer
Sent: Friday, August 26, 2016 7:35 AM
To: dev-secur...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
As we stated in WoSign website, browsers can distrust WoSign issued certificate after July 5th that no SCT embedded data, this is for full transparency.
> For BR auditor, I think this issue is too technical that fewer auditor
> can find out this problem.
> program's requirements?
I'm after the specifics of the changes to WoSign's policies and procedures regarding *notification*, not quality control. What were WoSign's previous policies and procedures regarding notification (obviously there was something in place, since Google was notified), and what changes have been made to improve those policies to ensure that all root programs are notified in line with each program's requirements in the future?
R: Sorry, I don't say it clear, please forgive my bad English since my native language is Chinese.
As I said this is my fault that we don't understand the Mozilla policy clearly that we don't think we need to report. But now we are clear that all mis-issued certificate case and any reported bug related system change also need to report. I and every related employee all clear now, then we can guarantee we will do it well in the future.
Why we log all SSL certificate from July 5th is for full transparency to let all related parties can report to us in the first time after the certificate is issued.
- Matt
--
I told [my daughter] that if I see her digging a hole that she might not be able to crawl out of, my job isn't to stand back and say "That's a *real* nice hole you're digging there".
-- Paul Tomblin, ASR
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Matt Palmer
Sent: Friday, August 26, 2016 7:35 AM
To: dev-secur...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
On Thu, Aug 25, 2016 at 07:11:18AM +0000, Richard Wang wrote:
> We can post all 2015 issued SSL certificate to CT log server if necessary.
> We can post all 2015 issued SSL certificate to CT log server if necessary.
That doesn't provide any assurance, in the face of misleading notBefore values in certificates. Without strong assurances that whatever failure of systems or processes allowed misleading notBefore values to end up in certificates has been corrected, the only way to be sure that there are no shenanigans going on is for *all* certificates issued by WoSign,
*regardless* of their purported issuance date, to carry SCTs from a qualified log (or have them presented at TLS establishment), and have the "effective date" for the purposes of compliance with relevant standards and guidelines to be the date of the earliest SCT presented. This is because the notBefore value in the certificate, being produced by a flawed process which allows misleading values to be used, is useless for the purposes of determining when the certificate was *actually* issued. Any TLS connection using a WoSign-issued certificate which does not present SCTs would have to be considered completely untrustworthy.
R: As I declared in the Bugzilla, only two certificate is the backdated certificate that not issued by normal system, it is a API bug that found and used by Computest to get the two backdates certificate, not WoSign normally issued this two cert. And this two test certificate are revoked after we got report, we stopped this API and delete the bug code.
*regardless* of their purported issuance date, to carry SCTs from a qualified log (or have them presented at TLS establishment), and have the "effective date" for the purposes of compliance with relevant standards and guidelines to be the date of the earliest SCT presented. This is because the notBefore value in the certificate, being produced by a flawed process which allows misleading values to be used, is useless for the purposes of determining when the certificate was *actually* issued. Any TLS connection using a WoSign-issued certificate which does not present SCTs would have to be considered completely untrustworthy.
As we stated in WoSign website, browsers can distrust WoSign issued certificate after July 5th that no SCT embedded data, this is for full transparency.
> For BR auditor, I think this issue is too technical that fewer auditor
> can find out this problem.
I believe Ryan has done an excellent job of outlining the concerns with this statement.
> We will add the quality control system to PKI system before issuing
> the certificate, and will check the crt.sh or use the CABF lint and
> X590 Lint to check the certificate before and after the certificate is
> issued to prevent such case, if such case happen, we will notify all
> browsers instantly.
This neither addresses the question I posed, nor does it contain sufficient specificity to be reassuring. I asked:
> what changes, exactly, has WoSign implemented to its policies and
> procedures to ensure that all trust programs in which WoSign is a
> participant are notified of future incidents, in line with each
> We will add the quality control system to PKI system before issuing
> the certificate, and will check the crt.sh or use the CABF lint and
> X590 Lint to check the certificate before and after the certificate is
> issued to prevent such case, if such case happen, we will notify all
> browsers instantly.
This neither addresses the question I posed, nor does it contain sufficient specificity to be reassuring. I asked:
> what changes, exactly, has WoSign implemented to its policies and
> procedures to ensure that all trust programs in which WoSign is a
> program's requirements?
I'm after the specifics of the changes to WoSign's policies and procedures regarding *notification*, not quality control. What were WoSign's previous policies and procedures regarding notification (obviously there was something in place, since Google was notified), and what changes have been made to improve those policies to ensure that all root programs are notified in line with each program's requirements in the future?
As I said this is my fault that we don't understand the Mozilla policy clearly that we don't think we need to report. But now we are clear that all mis-issued certificate case and any reported bug related system change also need to report. I and every related employee all clear now, then we can guarantee we will do it well in the future.
Why we log all SSL certificate from July 5th is for full transparency to let all related parties can report to us in the first time after the certificate is issued.
- Matt
--
I told [my daughter] that if I see her digging a hole that she might not be able to crawl out of, my job isn't to stand back and say "That's a *real* nice hole you're digging there".
-- Paul Tomblin, ASR
Richard Wang
Aug 25, 2016, 11:35:51 PM8/25/16
to Ryan Sleevi, mozilla-dev-s...@lists.mozilla.org
Yes, sorry for this.
As I admitted that this discussion gives us a big lesson that we know when we need to report incident to all browsers. We guarantee we will do it better.
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Ryan Sleevi
Sent: Friday, August 26, 2016 8:16 AM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
As I admitted that this discussion gives us a big lesson that we know when we need to report incident to all browsers. We guarantee we will do it better.
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Ryan Sleevi
Sent: Friday, August 26, 2016 8:16 AM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
On Thursday, August 25, 2016 at 4:35:39 PM UTC-7, Matt Palmer wrote:
> I'm after the specifics of the changes to WoSign's policies and
> procedures regarding *notification*, not quality control. What were
> WoSign's previous policies and procedures regarding notification
> (obviously there was something in place, since Google was notified),
> and what changes have been made to improve those policies to ensure
> that all root programs are notified in line with each program's requirements in the future?
> procedures regarding *notification*, not quality control. What were
> WoSign's previous policies and procedures regarding notification
> (obviously there was something in place, since Google was notified),
> and what changes have been made to improve those policies to ensure
> that all root programs are notified in line with each program's requirements in the future?
Clarification: In none of these incidents was Google notified proactively by WoSign. Instead, Google received communication from internal or external researchers regarding these issues, either prior to resolution or much later after the fact, and subsequently contacted WoSign regarding them.
It was only when Google found out recently that other programs were NOT notified, proactively, as had been expected, that Google shared the details it was aware of regarding various CA incidents, including those of WoSign, mentioned in this thread.
Richard Wang
Aug 25, 2016, 11:40:23 PM8/25/16
to Matt Palmer, dev-secur...@lists.mozilla.org
We know how to do in the future, and believe me we will do this better.
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Matt Palmer
Sent: Friday, August 26, 2016 10:03 AM
To: dev-secur...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Matt Palmer
Sent: Friday, August 26, 2016 10:03 AM
To: dev-secur...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
On Thu, Aug 25, 2016 at 05:15:58PM -0700, Ryan Sleevi wrote:
> On Thursday, August 25, 2016 at 4:35:39 PM UTC-7, Matt Palmer wrote:
> > I'm after the specifics of the changes to WoSign's policies and
> > procedures regarding *notification*, not quality control. What were
> > WoSign's previous policies and procedures regarding notification
> > (obviously there was something in place, since Google was notified),
> > and what changes have been made to improve those policies to ensure
> > that all root programs are notified in line with each program's requirements in the future?
>
> > procedures regarding *notification*, not quality control. What were
> > WoSign's previous policies and procedures regarding notification
> > (obviously there was something in place, since Google was notified),
> > and what changes have been made to improve those policies to ensure
> > that all root programs are notified in line with each program's requirements in the future?
>
> Clarification: In none of these incidents was Google notified
> proactively by WoSign. Instead, Google received communication from
> internal or external researchers regarding these issues, either prior
> to resolution or much later after the fact, and subsequently contacted
> WoSign regarding them.
> proactively by WoSign. Instead, Google received communication from
> internal or external researchers regarding these issues, either prior
> to resolution or much later after the fact, and subsequently contacted
> WoSign regarding them.
Oh, wow. I totally misread the initial report. That's almost certainly much *worse*, then, because it's not simply a matter of some adjustments to an existing process to improve it, but likely the development and deployment of an entirely new process.
- Matt
--
The main advantages of Haynes and Chilton manuals are that they cost $15, where the factory manuals cost $100 and up, and that they will tell you how to use two hammers, a block of wood, and a meerkat to replace "special tool
no. 2-112-A" -- Matt Roberds in asr.
- Matt
--
The main advantages of Haynes and Chilton manuals are that they cost $15, where the factory manuals cost $100 and up, and that they will tell you how to use two hammers, a block of wood, and a meerkat to replace "special tool
no. 2-112-A" -- Matt Roberds in asr.
percy...@gmail.com
Aug 26, 2016, 4:05:14 AM8/26/16
to mozilla-dev-s...@lists.mozilla.org
The news about possible sanction against WoSign was reported by Solidot http://www.solidot.org/story?sid=49448
(the Chinese version of Slashdot). Out of 12 comments in total (at the time of writing), 8 of them call for revocation of WoSign, the rest talks about the general bad security practices in China.
A quick intro of myself.
I'm Percy Alpha and I broke the news on GFW's MITM attack on iCloud, Outlook and Yahoo in 2014 and was later the victim of Great Cannon attack in 2015.
(the Chinese version of Slashdot). Out of 12 comments in total (at the time of writing), 8 of them call for revocation of WoSign, the rest talks about the general bad security practices in China.
A quick intro of myself.
I'm Percy Alpha and I broke the news on GFW's MITM attack on iCloud, Outlook and Yahoo in 2014 and was later the victim of Great Cannon attack in 2015.
percy...@gmail.com
Aug 26, 2016, 4:15:57 AM8/26/16
to mozilla-dev-s...@lists.mozilla.org
In most Chinese institutions, most checks and verifications are just formality. Contracting to the case of CNNIC CA, I'm not advocating for an outright removal of WoSign (even though I revoked the CA personally). But the incorrect notBefore date suggests that a mandatory inclusion of CT of all certs ever issued is needed. Of course, WoSign needs to address other issues raised by Matt and Ryan in addition to the CT requirement.
Richard Wang
Aug 26, 2016, 4:26:26 AM8/26/16
to percy...@gmail.com, mozilla-dev-s...@lists.mozilla.org
This is the standard way in China Internet, if a west company say something to China company, all will support the west company.
PLEASE don’t move this technical problem to political issue, thanks.
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of percy...@gmail.com
Sent: Friday, August 26, 2016 4:05 PM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
PLEASE don’t move this technical problem to political issue, thanks.
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of percy...@gmail.com
Sent: Friday, August 26, 2016 4:05 PM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
233sec Team
Aug 26, 2016, 3:57:56 PM8/26/16
to mozilla-dev-s...@lists.mozilla.org
Wosign's Issue mechanism is high risking for large enterprise.
This is one prove:
https://gist.github.com/xiaohuilam/8589f2dfaac435bae4bf8dfe0984f69e
Alicdn.com is the cdn asset domain name of Taobao/tmall who belong to alibaba, which are Chinese biggest online shopping websites.
With the fake cert's middle man attack, password stealing, information leaking...
This is one prove:
https://gist.github.com/xiaohuilam/8589f2dfaac435bae4bf8dfe0984f69e
Alicdn.com is the cdn asset domain name of Taobao/tmall who belong to alibaba, which are Chinese biggest online shopping websites.
With the fake cert's middle man attack, password stealing, information leaking...
Jonathan Rudenberg
Aug 26, 2016, 4:24:44 PM8/26/16
to 233sec Team, mozilla-dev-s...@lists.mozilla.org
Here’s the crt.sh link for this certificate: https://crt.sh/?id=29884704
Can you provide more details about where this certificate came from? Did you issue it using one of the vulnerabilities discussed in this thread?
Can you provide more details about where this certificate came from? Did you issue it using one of the vulnerabilities discussed in this thread?
Richard Wang
Aug 29, 2016, 12:49:33 AM8/29/16
to Ryan Sleevi, mozilla-dev-s...@lists.mozilla.org
On Thursday, August 25, 2016 at 12:14:10 AM UTC-7, Richard Wang wrote:
> We can post all 2015 issued SSL certificate to CT log server if necessary.
Is there any reason not to do that proactively?
R: OK, we will post all 2015 issued SSL certificates to CT log server, but this take time since we issued 115K SSL certificate in 2015.
For incident 1 - mis-issued certificate with un-validated subdomain, total 33 certificates. We have posted to CT log server and listed in crt.sh, here is the URL. Some certificates are revoked after getting report from subscriber, but some still valid, if any subscriber think it must be revoked and replaced new one, please contact us in the system, thanks.
R: OK, we will post all 2015 issued SSL certificates to CT log server, but this take time since we issued 115K SSL certificate in 2015.
https://crt.sh/?id=7036355
https://crt.sh/?id=29805552
https://crt.sh/?id=7678955
https://crt.sh/?id=29805553
https://crt.sh/?id=29805554
https://crt.sh/?id=29805555
https://crt.sh/?id=29805556
https://crt.sh/?id=6798197
https://crt.sh/?id=29805558
https://crt.sh/?id=6798107
https://crt.sh/?id=29805559
https://crt.sh/?id=7728281
https://crt.sh/?id=29805560
https://crt.sh/?id=6639307
https://crt.sh/?id=29805561
https://crt.sh/?id=29805562
https://crt.sh/?id=6805650
https://crt.sh/?id=6739981
https://crt.sh/?id=7966229
https://crt.sh/?id=7094833
https://crt.sh/?id=29805563
https://crt.sh/?id=29805564
https://crt.sh/?id=29805565
https://crt.sh/?id=29805566
https://crt.sh/?id=29805567
https://crt.sh/?id=6843440
https://crt.sh/?id=29805568
https://crt.sh/?id=6999366
https://crt.sh/?id=29805569
https://crt.sh/?id=9534934
https://crt.sh/?id=29806448
https://crt.sh/?id=29813139
https://crt.sh/?id=29647048
For incident 0, the certificate issued related using higher level port validated, total 72 certificates. To be clear, those certificates are validated by website control validation method that using other port except 80 and 443. So we think those certificate no need to be revoked. The crt.sh link just for your reference.
https://crt.sh/?id=29805572
https://crt.sh/?id=7022909
https://crt.sh/?id=7564839
https://crt.sh/?id=29805573
https://crt.sh/?id=29805574
https://crt.sh/?id=29805575
https://crt.sh/?id=29805576
https://crt.sh/?id=29805577
https://crt.sh/?id=6969460
https://crt.sh/?id=29805578
https://crt.sh/?id=29805579
https://crt.sh/?id=29805580
https://crt.sh/?id=29805581
https://crt.sh/?id=29805582
https://crt.sh/?id=29805584
https://crt.sh/?id=29805585
https://crt.sh/?id=29805586
https://crt.sh/?id=9911443
https://crt.sh/?id=29805587
https://crt.sh/?id=7122803
https://crt.sh/?id=29805588
https://crt.sh/?id=29805589
https://crt.sh/?id=9985267
https://crt.sh/?id=29805590
https://crt.sh/?id=29805591
https://crt.sh/?id=29805592
https://crt.sh/?id=29805593
https://crt.sh/?id=29805594
https://crt.sh/?id=7224978
https://crt.sh/?id=10917791
https://crt.sh/?id=29805595
https://crt.sh/?id=29805596
https://crt.sh/?id=29805597
https://crt.sh/?id=6788465
https://crt.sh/?id=7224923
https://crt.sh/?id=9169568
https://crt.sh/?id=6836953
https://crt.sh/?id=29805598
https://crt.sh/?id=8172756
https://crt.sh/?id=29805599
https://crt.sh/?id=29805600
https://crt.sh/?id=7021184
https://crt.sh/?id=29805601
https://crt.sh/?id=29805602
https://crt.sh/?id=29805603
https://crt.sh/?id=29805604
https://crt.sh/?id=6927114
https://crt.sh/?id=6777468
https://crt.sh/?id=9793847
https://crt.sh/?id=29805605
https://crt.sh/?id=29805606
https://crt.sh/?id=29805607
https://crt.sh/?id=29805608
https://crt.sh/?id=9932344
https://crt.sh/?id=29805609
https://crt.sh/?id=29805610
https://crt.sh/?id=6880740
https://crt.sh/?id=29805611
https://crt.sh/?id=29805612
https://crt.sh/?id=7015627
https://crt.sh/?id=10008992
https://crt.sh/?id=29805613
https://crt.sh/?id=29805614
https://crt.sh/?id=29805615
https://crt.sh/?id=29805616
https://crt.sh/?id=7046181
https://crt.sh/?id=29805617
https://crt.sh/?id=29805618
https://crt.sh/?id=29805619
https://crt.sh/?id=7121749
https://crt.sh/?id=29805620
https://crt.sh/?id=6999366
Best Regards,
Richard
Gervase Markham
Aug 29, 2016, 4:27:59 AM8/29/16
to Peter Bowen, Richard Wang, Jeremy Rowley
On 24/08/16 17:44, Peter Bowen wrote:
> I think you are missing the most likely option: CA hosting. My
> understanding is that it is not uncommon that one CA operator
> contracts with another CA operator to run a CA on behalf of the first
> operator. I don't think it has been clear what disclosure of this
> practice is required. Given that I believe this is widespread, I
> assumed that all of the issuing CAs in this case were operated by the
> same entity.
If StartCom are hosting WoSign's infra (seems less likely), then it's
> I think you are missing the most likely option: CA hosting. My
> understanding is that it is not uncommon that one CA operator
> contracts with another CA operator to run a CA on behalf of the first
> operator. I don't think it has been clear what disclosure of this
> practice is required. Given that I believe this is widespread, I
> assumed that all of the issuing CAs in this case were operated by the
> same entity.
still a pretty severe mistake to accidentally issue a certificate from
one of your customer's roots rather than your own, although one might
say the mistake in this case would be StartCom's.
If WoSign are hosting StartCom's infra, it still leaves open the
question of why StartCom are deploying code that WoSign are no longer
using, and haven't for six months, and why WoSign permitted the StartCom
UI to issue WoSign certificates at all.
Gerv
Gervase Markham
Aug 29, 2016, 4:33:25 AM8/29/16
to Richard Wang, Jeremy Rowley
On 25/08/16 04:38, Richard Wang wrote:
> R: NOT this case you think. Due to root inclusion problem, WoSign
> root is cross signed by StartCom since 2011. And we shared some
> facility with StartCom like CRL and OCSP distribution etc. But not
> this case, as I declared in the previous email, this is a API
> parameter option that can post data to any server located in any
> place.
At what point in this process is the domain control validation done?
> R: NOT this case you think. Due to root inclusion problem, WoSign
> root is cross signed by StartCom since 2011. And we shared some
> facility with StartCom like CRL and OCSP distribution etc. But not
> this case, as I declared in the previous email, this is a API
> parameter option that can post data to any server located in any
> place.
Gerv
Gervase Markham
Aug 29, 2016, 4:38:01 AM8/29/16
to Richard Wang, Ryan Sleevi
On 26/08/16 04:33, Richard Wang wrote:
> As I admitted that this discussion gives us a big lesson that we know
> when we need to report incident to all browsers. We guarantee we will
> do it better.
Richard,
> As I admitted that this discussion gives us a big lesson that we know
> when we need to report incident to all browsers. We guarantee we will
> do it better.
You have been involved in this (Mozilla) discussion group and in the CAB
Forum for several years. In that time, you will have seen multiple CAs
disclose instances where certificates were misissued, and you will have
seen root programs take such disclosures seriously and consider them
important. Did it not occur to you that the same standard of disclosure
that everyone else was using should also apply to WoSign?
Gerv
Gervase Markham
Aug 29, 2016, 4:41:06 AM8/29/16
to Richard Wang, Ryan Sleevi
On 29/08/16 05:46, Richard Wang wrote:
> For incident 1 - mis-issued certificate with un-validated subdomain,
> total 33 certificates. We have posted to CT log server and listed in
> crt.sh, here is the URL. Some certificates are revoked after getting
> report from subscriber, but some still valid, if any subscriber think
> it must be revoked and replaced new one, please contact us in the
> system, thanks.
Er, no. If these certificates were issued with unvalidated parent
> For incident 1 - mis-issued certificate with un-validated subdomain,
> total 33 certificates. We have posted to CT log server and listed in
> crt.sh, here is the URL. Some certificates are revoked after getting
> report from subscriber, but some still valid, if any subscriber think
> it must be revoked and replaced new one, please contact us in the
> system, thanks.
domains (e.g. with github.com when the person validation foo.github.com)
then they need to all be revoked. You should actively contact your
customers and issue them new certificates containing only validated
information, and then revoke these ones.
Gerv
Gervase Markham
Aug 29, 2016, 4:43:33 AM8/29/16
to 233sec Team
On 26/08/16 06:12, 233sec Team wrote:
> https://gist.github.com/xiaohuilam/8589f2dfaac435bae4bf8dfe0984f69e
>
> Alicdn.com is the cdn asset domain name of Taobao/tmall who belong to alibaba, which are Chinese biggest online shopping websites.
> With the fake cert's middle man attack, password stealing, information leaking...
Can you prove you have control of the private key associated with this
> https://gist.github.com/xiaohuilam/8589f2dfaac435bae4bf8dfe0984f69e
>
> Alicdn.com is the cdn asset domain name of Taobao/tmall who belong to alibaba, which are Chinese biggest online shopping websites.
> With the fake cert's middle man attack, password stealing, information leaking...
certificate?
Gerv
Patrick Figel
Aug 29, 2016, 9:55:50 AM8/29/16
to ric...@wosign.com, dev-secur...@lists.mozilla.org
Richard,
the problem with this approach is that the *subscriber* might not be
authorized to make this decision for the parent domain. To go back to
the GitHub case, the "owner" of a github.io subdomain telling you that
they are authorized to own a certificate that covers github.io is
irrelevant, as they have never demonstrated ownership of that domain.
The right approach would be to revoke the affected certificates
immediately and inform your subscribers that they will need to re-issue
their certificates (while also verifying ownership of the root domain).
Here's another similar case - cloudapp.net, which belongs to Microsoft
Azure. I'm fairly certain this certificate was not authorized by Microsoft:
https://crt.sh/?id=29805555
Thanks,
Patrick
On 29/08/16 11:30, Richard Wang wrote:
> Yes, we plan to revoke all after getting confirmation from
> subscriber. We are doing this.
>
> Regards,
>
> Richard
>
>> On 29 Aug 2016, at 16:38, Gervase Markham <ge...@mozilla.org>
>> wrote:
>>
>>> On 29/08/16 05:46, Richard Wang wrote: For incident 1 -
the problem with this approach is that the *subscriber* might not be
authorized to make this decision for the parent domain. To go back to
the GitHub case, the "owner" of a github.io subdomain telling you that
they are authorized to own a certificate that covers github.io is
irrelevant, as they have never demonstrated ownership of that domain.
The right approach would be to revoke the affected certificates
immediately and inform your subscribers that they will need to re-issue
their certificates (while also verifying ownership of the root domain).
Here's another similar case - cloudapp.net, which belongs to Microsoft
Azure. I'm fairly certain this certificate was not authorized by Microsoft:
https://crt.sh/?id=29805555
Thanks,
Patrick
On 29/08/16 11:30, Richard Wang wrote:
> Yes, we plan to revoke all after getting confirmation from
> subscriber. We are doing this.
>
> Regards,
>
> Richard
>
>> On 29 Aug 2016, at 16:38, Gervase Markham <ge...@mozilla.org>
>> wrote:
>>
>>> On 29/08/16 05:46, Richard Wang wrote: For incident 1 -
>>> mis-issued certificate with un-validated subdomain, total 33
>>> certificates. We have posted to CT log server and listed in
>>> crt.sh, here is the URL. Some certificates are revoked after
>>> getting report from subscriber, but some still valid, if any
>>> subscriber think it must be revoked and replaced new one, please
>>> contact us in the system, thanks.
>>
>>> certificates. We have posted to CT log server and listed in
>>> crt.sh, here is the URL. Some certificates are revoked after
>>> getting report from subscriber, but some still valid, if any
>>> subscriber think it must be revoked and replaced new one, please
>>> contact us in the system, thanks.
>>
>> Er, no. If these certificates were issued with unvalidated parent
>> domains (e.g. with github.com when the person validation
>> foo.github.com) then they need to all be revoked. You should
>> actively contact your customers and issue them new certificates
>> containing only validated information, and then revoke these ones.
>>
>> Gerv
>>
>>
>> domains (e.g. with github.com when the person validation
>> foo.github.com) then they need to all be revoked. You should
>> actively contact your customers and issue them new certificates
>> containing only validated information, and then revoke these ones.
>>
>> Gerv
>>
>>
233sec Team
Aug 29, 2016, 12:08:03 PM8/29/16
to mozilla-dev-s...@lists.mozilla.org
Not vulnerabilities mentioned in this thread, but a Human-Audit weak process.
Detail you can see the reply content i send to Mr.Wang
在 2016年8月27日星期六 UTC+8上午4:24:44，Jonathan Rudenberg：
Detail you can see the reply content i send to Mr.Wang
在 2016年8月27日星期六 UTC+8上午4:24:44，Jonathan Rudenberg：
蓝小灰
Aug 29, 2016, 12:08:03 PM8/29/16
to Gervase Markham, mozilla-dev-s...@lists.mozilla.org
Of course I have private key of this certificate
Gervase Markham <ge...@mozilla.org>于2016年8月29日 周一16:42写道：
> On 26/08/16 06:12, 233sec Team wrote:
Gervase Markham <ge...@mozilla.org>于2016年8月29日 周一16:42写道：
> On 26/08/16 06:12, 233sec Team wrote:
> > https://gist.github.com/xiaohuilam/8589f2dfaac435bae4bf8dfe0984f69e
> >
> > Alicdn.com is the cdn asset domain name of Taobao/tmall who belong to
> alibaba, which are Chinese biggest online shopping websites.
> > With the fake cert's middle man attack, password stealing, information
> leaking...
>
> >
> > Alicdn.com is the cdn asset domain name of Taobao/tmall who belong to
> alibaba, which are Chinese biggest online shopping websites.
> > With the fake cert's middle man attack, password stealing, information
> leaking...
>
Tobias Sachs
Aug 29, 2016, 12:08:17 PM8/29/16
to mozilla-dev-s...@lists.mozilla.org
Is there any plan to revoke the certificate in OneCRL soon? And how we could speed this up? @ Kathleen Wilson
mar...@marcan.st
Aug 29, 2016, 12:08:36 PM8/29/16
to mozilla-dev-s...@lists.mozilla.org
https://crt.sh/?id=29647048
https://crt.sh/?id=29805567
Why wasn't this additional cert identified and disclosed prior to the aforementioned list being posted? It seems a no-brainer to manually audit the list for obvious cases of mis-issuance (i.e. not only the domain was not validated, but the customer clearly has no ability to validate it if asked).
xcrai...@gmail.com
Aug 29, 2016, 12:09:05 PM8/29/16
to mozilla-dev-s...@lists.mozilla.org
On Friday, August 26, 2016 at 4:26:26 PM UTC+8, Richard Wang wrote:
> This is the standard way in China Internet, if a west company say something to China company, all will support the west company.
-- especially when local CAs are losing credibility to end-users. Microsoft Azure's Chinese website[1] has migrated from CNNIC CA, to WoSign, and recently to DigiCert. CNNIC itself, ironically, also moved to DigiCert.
> This is the standard way in China Internet, if a west company say something to China company, all will support the west company.
[1]: azure.cn
It is almost axiomatic that without a proper statement & fix (no more 233sec team exploits) made, WoSign will continue losing trust from end-users as well as webmasters.
> PLEASE don’t move this technical problem to political issue, thanks.
[2]: http://wayback.archive.org/web/20160828045112/https://www.wosign.com/about/WoSign_ForeignCA_compare.htm
* Security: handled by Chinese company itself, fully secure. (Foreign CA: System Security should not be a problem, but risks of random revokes and/or access failures exist.)
* Compliance with Chinese Law: Yes (Foreign CA: No, legal risks exist.)
Gervase Markham
Aug 29, 2016, 1:26:20 PM8/29/16
to mozilla-dev-s...@lists.mozilla.org
On 29/08/16 09:48, 蓝小灰 wrote:
> Of course I have private key of this certificate
I have asked 蓝小灰 for cryptographic proof of this.
> Of course I have private key of this certificate
Gerv
Percy
Aug 29, 2016, 5:53:25 PM8/29/16
to mozilla-dev-s...@lists.mozilla.org
It's unlikely that Alibaba will use a free cert from WoSign. As a commercial site, they usually use Verisign or globalSign
Percy
Aug 29, 2016, 7:03:57 PM8/29/16
to mozilla-dev-s...@lists.mozilla.org
"Some certificates are revoked after getting report from subscriber, but some still valid, if any subscriber think it must be revoked and replaced new one, please contact us in the system, thanks"
WoSign seems to lack the basic understanding of how a certificate is used in authentication, consequently unfit to be a CA, I call for revocation of WoSign from all root programs immediately. http://www.percya.com/2016/08/chinese-ca-wosign-faces-revocation.html
dion...@gmail.com
Aug 30, 2016, 11:18:56 AM8/30/16
to mozilla-dev-s...@lists.mozilla.org
If I understand correctly, these 105 certificates are all mis-issued using the incorrect policies of either (0) website control validation with higher port numbers, or (1) parent domain-name verification by demonstrating control of a subdomain.
It is unclear why, given the fact that incorrect validation was done for these certificates, they are not already revoked. I don't understand why you are expecting a report from your respective subscriber to do that, as they have not proven control of the domain names in any case.
These certificates must all be revoked immediately.
On Monday, August 29, 2016 at 7:49:33 AM UTC+3, Richard Wang wrote:
>
> For incident 1 - mis-issued certificate with un-validated subdomain, total 33 certificates. We have posted to CT log server and listed in crt.sh, here is the URL. Some certificates are revoked after getting report from subscriber, but some still valid, if any subscriber think it must be revoked and replaced new one, please contact us in the system, thanks.
>
It is unclear why, given the fact that incorrect validation was done for these certificates, they are not already revoked. I don't understand why you are expecting a report from your respective subscriber to do that, as they have not proven control of the domain names in any case.
These certificates must all be revoked immediately.
On Monday, August 29, 2016 at 7:49:33 AM UTC+3, Richard Wang wrote:
>
> For incident 1 - mis-issued certificate with un-validated subdomain, total 33 certificates. We have posted to CT log server and listed in crt.sh, here is the URL. Some certificates are revoked after getting report from subscriber, but some still valid, if any subscriber think it must be revoked and replaced new one, please contact us in the system, thanks.
>
dymu...@gmail.com
Aug 30, 2016, 11:19:18 AM8/30/16
to mozilla-dev-s...@lists.mozilla.org
It is interesting that WoSign followed the redirect. I suppose it is assumed that with a 301 permanent redirect that the new domain is controlled by the same person, but that seems a bit sketchy.
I had forgotten about the second github.io certificate, but looking back on my emails, it was also revoked the day after it was created.
Percy
Aug 30, 2016, 1:46:04 PM8/30/16
to mozilla-dev-s...@lists.mozilla.org
https://crt.sh is down. Maybe someone can check with comodo to see whether they got DDOSed?
Here are the Google CT for the possibly mis-issued certs mentioned in this thread. It would be a lot harder to take down the Google CT.
Possible fake cert for Github
https://www.google.com/transparencyreport/https/ct/#domain=github.io&incl_exp=false&incl_sub=false&issuer=lPrsb9Gbn4s%3D
Possible fake cert for Alibaba, the largest commercial site in China
https://www.google.com/transparencyreport/https/ct/#domain=alicdn.com&incl_exp=false&incl_sub=false&issuer=lPrsb9Gbn4s%3D
Possible fake cert for Microsoft
https://www.google.com/transparencyreport/https/ct/#domain=cloudapp.net&incl_exp=false&incl_sub=false&issuer=lPrsb9Gbn4s%3D
Here are the Google CT for the possibly mis-issued certs mentioned in this thread. It would be a lot harder to take down the Google CT.
Possible fake cert for Github
https://www.google.com/transparencyreport/https/ct/#domain=github.io&incl_exp=false&incl_sub=false&issuer=lPrsb9Gbn4s%3D
Possible fake cert for Alibaba, the largest commercial site in China
https://www.google.com/transparencyreport/https/ct/#domain=alicdn.com&incl_exp=false&incl_sub=false&issuer=lPrsb9Gbn4s%3D
Possible fake cert for Microsoft
https://www.google.com/transparencyreport/https/ct/#domain=cloudapp.net&incl_exp=false&incl_sub=false&issuer=lPrsb9Gbn4s%3D
Rob Stradling
Aug 30, 2016, 4:43:10 PM8/30/16
to Percy, mozilla-dev-s...@lists.mozilla.org
On 30/08/16 18:45, Percy wrote:
> https://crt.sh is down. Maybe someone can check with comodo to see whether they got DDOSed?
Sorry about that. crt.sh is back up now.
> https://crt.sh is down. Maybe someone can check with comodo to see whether they got DDOSed?
It wasn't a DDOS attack.
Every so often something goes awry with the database replication
(between crt.sh's master database and front-end slave databases), which
causes all of the front-end databases to crash. Somebody (usually me,
but I've been out for most of today) is normally around to restart the
crashed databases. I don't know why our NOC team didn't fix this
several hours ago, but I intend to find out. Perhaps there are some
improvements we need to make to our internal monitoring systems.
> Here are the Google CT for the possibly mis-issued certs mentioned in this thread. It would be a lot harder to take down the Google CT.
That said, I'll see what I can do to improve crt.sh's uptime. I already
have one offer of help...
https://twitter.com/FiloSottile/status/770642205304352768
> Possible fake cert for Github
> https://www.google.com/transparencyreport/https/ct/#domain=github.io&incl_exp=false&incl_sub=false&issuer=lPrsb9Gbn4s%3D
>
> Possible fake cert for Alibaba, the largest commercial site in China
> https://www.google.com/transparencyreport/https/ct/#domain=alicdn.com&incl_exp=false&incl_sub=false&issuer=lPrsb9Gbn4s%3D
>
> Possible fake cert for Microsoft
> https://www.google.com/transparencyreport/https/ct/#domain=cloudapp.net&incl_exp=false&incl_sub=false&issuer=lPrsb9Gbn4s%3D
Rob Stradling
Senior Research & Development Scientist
COMODO - Creating Trust Online
Nick Lamb
Aug 30, 2016, 9:19:46 PM8/30/16
to mozilla-dev-s...@lists.mozilla.org
On Tuesday, 30 August 2016 16:19:18 UTC+1, dymu...@gmail.com wrote:
> It is interesting that WoSign followed the redirect. I suppose it is assumed that with a 301 permanent redirect that the new domain is controlled by the same person, but that seems a bit sketchy.
Hmm. I think that if there's a 301 redirect in place for every URL then in practice the effect is that the target site does control the content of the origin site. Such circumstances already exist without redirects.
> It is interesting that WoSign followed the redirect. I suppose it is assumed that with a 301 permanent redirect that the new domain is controlled by the same person, but that seems a bit sketchy.
Suppose the site http://theclown.example.net/ functions as a thin proxy of another site, say http://famous-cloud-provider.example.com/ and simply copies the contents of equivalent resources at this other site, but replaces all instances of the word "cloud" in human-readable text with "clown". An amusing prank.
We can expect that the legitimate owner of famous-cloud-provider.example.com would be able to obtain a certificate for theclown.example.net by obeying either the typical ad hoc validation scheme of the sort which you experienced with WoSign, or a Ballot 169-compliant scheme like ACME. This is because it is very unlikely that the validation content would include the exact word "cloud" and so it would pass through the proxy unaltered and be readable by the CA's validation system when checking theclown.example.net
But this seems like no great loss, since in fact the contents of theclown.example.net are almost exactly controlled by famous-cloud-provider.example.com, the certificate's implications about who controls theclown.example.net are functionally true.
If the legitimate owner of theclown.example.net doesn't want this to be possible under prior ad hoc validation systems it's a bit tricky to prevent, but under Ballot 169 it's simply a matter of making /.well-known/ exempt from the proxy rule, which is anyway already a very good idea.
NB. If my musing inspires substantial further debate it should happen in another thread, not this one about WoSign.
Peter Bowen
Aug 30, 2016, 10:44:45 PM8/30/16
to Gervase Markham, Richard Wang, mozilla-dev-s...@lists.mozilla.org
On Wed, Aug 24, 2016 at 6:08 AM, Gervase Markham <ge...@mozilla.org> wrote:
> Dear m.d.s.policy,
>
> Several incidents have come to our attention involving the CA "WoSign".
> Mozilla is considering what action it should take in response to these
> incidents. This email sets out our understanding of the situation.
>
> Before we begin, we note that Section 1 of the Mozilla CA Certificate
> Enforcement Policy[0] says: "When a serious security concern is noticed,
> such as a major root compromise, it should be treated as a
> security-sensitive bug, and the Mozilla Policy for Handling Security
> Bugs should be followed." It is clear to us, and appears to be clear to
> other CAs based on their actions, that misissuances where domain control
> checks have failed fall into the category of "serious security concern".
I have run into another bug that appears to be fixed in WoSign's
infrastructure but is worth noting.
In April 2015, two different WoSign CAs issued multiple certificates
to distinct subjects using the same serial number. The CT logs have
picked up two instances of this occuring:
https://crt.sh/?serial=0D3BBDC3A0175E38F9D0070CD050986A shows eight
distinct certificates with the same serial number, all with notBefore
dates of 2015-04-14.
https://crt.sh/?serial=056D1570DA645BF6B44C0A7077CC6769 shows dozens
of distinct certificates with the same serial number, with notBefore
dates between 2015-04-10 and 2015-04-14.
I have not examined their management assertions to see if this was
documented and I do not know if this was reported to Mozilla at the
time. These certificates do not appear to meet RFC 5280's
requirements, which say:
"The serial number MUST be a positive integer assigned by the CA to
each certificate. It MUST be unique for each certificate issued by a
given CA (i.e., the issuer name and serial number identify a unique
certificate)"
(https://tools.ietf.org/html/rfc5280#section-4.1.2.2)
Was Mozilla advised of this issue?
Thanks,
Peter
> Dear m.d.s.policy,
>
> Several incidents have come to our attention involving the CA "WoSign".
> Mozilla is considering what action it should take in response to these
> incidents. This email sets out our understanding of the situation.
>
> Before we begin, we note that Section 1 of the Mozilla CA Certificate
> Enforcement Policy[0] says: "When a serious security concern is noticed,
> such as a major root compromise, it should be treated as a
> security-sensitive bug, and the Mozilla Policy for Handling Security
> Bugs should be followed." It is clear to us, and appears to be clear to
> other CAs based on their actions, that misissuances where domain control
> checks have failed fall into the category of "serious security concern".
I have run into another bug that appears to be fixed in WoSign's
infrastructure but is worth noting.
In April 2015, two different WoSign CAs issued multiple certificates
to distinct subjects using the same serial number. The CT logs have
picked up two instances of this occuring:
https://crt.sh/?serial=0D3BBDC3A0175E38F9D0070CD050986A shows eight
distinct certificates with the same serial number, all with notBefore
dates of 2015-04-14.
https://crt.sh/?serial=056D1570DA645BF6B44C0A7077CC6769 shows dozens
of distinct certificates with the same serial number, with notBefore
dates between 2015-04-10 and 2015-04-14.
I have not examined their management assertions to see if this was
documented and I do not know if this was reported to Mozilla at the
time. These certificates do not appear to meet RFC 5280's
requirements, which say:
"The serial number MUST be a positive integer assigned by the CA to
each certificate. It MUST be unique for each certificate issued by a
given CA (i.e., the issuer name and serial number identify a unique
certificate)"
(https://tools.ietf.org/html/rfc5280#section-4.1.2.2)
Was Mozilla advised of this issue?
Thanks,
Peter
Message has been deleted
Richard Wang
Aug 30, 2016, 11:07:49 PM8/30/16
to Peter Bowen, Gervase Markham, mozilla-dev-s...@lists.mozilla.org
This case is in the BR report: https://cert.webtrust.org/SealFile?seal=2019&file=pdf
Thanks.
Best Regards,
Richard
-----Original Message-----
From: Peter Bowen [mailto:pzb...@gmail.com]
Sent: Wednesday, August 31, 2016 10:45 AM
To: Gervase Markham <ge...@mozilla.org>
Cc: mozilla-dev-s...@lists.mozilla.org; Richard Wang <ric...@wosign.com>
Subject: Re: Incidents involving the CA WoSign
Thanks.
Best Regards,
Richard
-----Original Message-----
From: Peter Bowen [mailto:pzb...@gmail.com]
Sent: Wednesday, August 31, 2016 10:45 AM
To: Gervase Markham <ge...@mozilla.org>
Cc: mozilla-dev-s...@lists.mozilla.org; Richard Wang <ric...@wosign.com>
Subject: Re: Incidents involving the CA WoSign
Percy
Aug 31, 2016, 4:49:23 AM8/31/16
to mozilla-dev-s...@lists.mozilla.org
On Tuesday, August 30, 2016 at 7:47:43 PM UTC-7, itk9...@gmail.com wrote:
> Wosign indirectly bought StartSSL, https://www.letsphish.org
Ha! It makes so much sense now why StartEncrypt is such a catastrophe(https://www.google.com/search?q=StartEncrypt). I've revoked all StarCom certs in my OS. Thanks for the heads up!
> Wosign indirectly bought StartSSL, https://www.letsphish.org
Ha! It makes so much sense now why StartEncrypt is such a catastrophe(https://www.google.com/search?q=StartEncrypt). I've revoked all StarCom certs in my OS. Thanks for the heads up!
Peter Gutmann
Aug 31, 2016, 5:17:08 AM8/31/16
to itk9...@gmail.com, mozilla-dev-s...@lists.mozilla.org
itk9...@gmail.com <itk9...@gmail.com> writes:
>Wosign indirectly bought StartSSL, https://www.letsphish.org
Has there been any independent investigation into this? We know that CAs are
bought and sold like baseball trading cards, but it's usually done publicly
and freely acknowledged, whereas this one seems to have been done in a
somewhat underhanded manner...
Peter.
Gervase Markham
Aug 31, 2016, 6:16:29 AM8/31/16
to Percy
On 29/08/16 22:53, Percy wrote:
> Gerv, I've notified the security team in Alibaba about this possible fake cert and ask them to confirm that they have not applied a cert.
> It's unlikely that Alibaba will use a free cert from WoSign. As a commercial site, they usually use Verisign or globalSign
That might also help; thank you. Please ask them to contact me directly
> Gerv, I've notified the security team in Alibaba about this possible fake cert and ask them to confirm that they have not applied a cert.
> It's unlikely that Alibaba will use a free cert from WoSign. As a commercial site, they usually use Verisign or globalSign
to confirm this cert was not requested by them.
Gerv
s...@samspin.net
Aug 31, 2016, 8:42:45 AM8/31/16
to mozilla-dev-s...@lists.mozilla.org
To the policymakers at Mozilla, my name is Samuel Pinder.
I consider myself an computer network analyst and have a degree in Web Systems Development. I also host a small number of websites on a technical level. I have used Startcom's services for a number of years. I only recently came across WoSign when I discovered that they offer 3-year valid certificates for free for up to 5 hostnames. That seemed quite an excessively long time in my opinion for a free certificate, almost too good to be true. As a customer of effectively both WoSign and Startcom, I have been having gradually growing concerns for some time about their recent practices and this recent news has not helped. I am currently looking into using other CA's for my needs but will watch what happens next closely before making a decision. I firmly believe WoSign have almost 'purchased' Startcom as I noticed that the nameserver domain for startssl.com (namely startcomca.net) would bring up WoSign's website for a time when accessed via a web browser as www.startcomca.net. I personally emailed Eddy Nigg (Startcom founder) about this, and this was eventually changed to redirect to startssl.com. So it is obvious that WoSign is hosting/sharing Startcom's infrastucture on it's own CA hardware/software, which Eddie confirmed in his email reply. The SSL certificate at the time for www.startcom.org had expired for over a week and still does have many broken links. Using the 'Live Chat' on www.startssl.com had a very confused employee tell me that startcom.org is no longer the website for Startcom, but startssl.com is. I stressed that this was confusing to people and that they should redirect the latter if that's the case. However... startcom.org remains using Israeli-based nameservers. Just how closely related Startcom is to WoSign is unknown, but it does mean that almost anything that renders WoSign vulnerable, also applies to Startcom. I would hope that Startcom re-evaluate their relationship with WoSign following this mess at the very least.
Regarding WoSign in particular, the SHA-1 miss-issued certificates does look like a genuine mistake, being issued off of an SHA-2 intermediate makes their existence rather pointless in the case of backward compatibility, as old operating systems need a whole SHA-1 chain to validate properly. So there is no obvious motive for this API functionality to be left there on purpose, unless.... a rogue employee/subcontractor left it in place for when a SHA-1 hash collision becomes possible? What is likely here is that SHA-1 used to be the default signature choice, and was indeed legacy functionality left behind. It is very concerning however that despite WoSign having an external audit (which all CA's are meant to have) it appears nobody has independently performed a code audit of their API backend to prevent this being possible in a production environment. With regards to the mis-issued certificates applying to whole base domains validated only sub-domain, that prospect is horrifying. With regards to WoSign's API following 301 redirects from one domain to another though, I have noticed that Let's Encrypt's API (another CA at letsencrypt.org) also has this behaviour. This may be standard behaviour of the ACME protocol. Perhaps it would be prudent to decide in overall policy whether 301 redirects of verification URLs are acceptable. There are legitimate reasons why this would happen, for example if a customer owned www.example1.com and www.example2.net, but wanted the latter to always redirect to the other (because for example of a name change, but had many hardcoded https:// links), they would otherwise have to temporarily disable the redirect to get an SSL certificate covering both domains when validating with this method. With regards to ports other than 80 and 443 and 8080 being used for validation, I think the motive of this is because China has a mandatory media licencing requirement (ICP licence) for all servers that operate on either ports. It may not be practical for an individual in China to get one of these licences just to be able to gain an SSL certificate with this validation method. Of course allowing higher numbered ports does not give much excuse. I tested WoSign's website today however and can confirm that they have removed URL-based file verification, and now only allow DNS or email (from WHOIS) validation. All that's well and good, but the fact remains that there could be many other mis-issued certificates already in existence and the scope remains unknown at this point just how many. We only have WoSign's word that all of the mis-issued certificates are known and/or revoked. If it cannot be certain that WoSign have revoked all incorrectly-validated certificates, then the obvious choice is to add the "WoSign CA Free SSL Certificate G2" intermediate to OneCRL, and possibly their DV intermediate too, as these intermediates are known to have mis-issued certificates whilst their other intermediates have always had tighter validation requirements for OV and EV validation. That approach would certainly be painful for existing users and would require WoSign to immediately contact current and past customers to have new certificates issued from a new intermediate, once it is certain that all the bugs in the validation procedures have been resolved. Would it be appropriate to give customers time before such action is taken, given that it would break many websites, or does the risk of data leaks from unknown mis-issued certificates outweigh the need to give customers time? The other option of simply preventing any newly-issued certificates from being accepted in browers does not seem a very good idea given that WoSign have been known to backdate certificates. In fact, I would even suggest that WoSign cease issuing free certificates (and most possibly paid DV certificates) and in the meantime have an independent audit of their backend to ensure all vulnerablities are closed, with the results made public. Only then should
they recommence issuance- at least on an automated scale. GlobalSign did something similar to this when there was speculation that the hacker responsible for compromising Diginotar had also breached them, at great risk to their business but ultimately gained respect in the process. Given the severity of these incidents and the fact they went unreported though I would not be surprised if WoSign's whole roots are blacklisted eventually if WoSign do not get their act together. The promise to completely commit to Certificate Transparency is quite a step, but it does not change what has already happened. WoSign's recent announcement gives me the impression they expect others to manually scour the CT logs once they upload their 2015 certificate data for mis-issuances that need revoking rather than do this themselves. Whilst it may *technically* be 'transparent' with CT, they don't seem in the spirit of things being completely transparent with their practices- in particular the relationship between Startcom and WoSign. At least one of these incidents actually occurred during research into Startcom, and it turned out the API was being shared by both CA's. Given the circumstances, I think it is appropriate to ask just what the relationship is between them. For me, it appears they are now one and the same, while the original 'Startcom' infratructure I once trusted is gathering dust and has been abandoned. I think I'll be taking my needs to another CA very soon if things don't look up!
Samuel Pinder
On Wednesday, 24 August 2016 14:09:02 UTC+1, Gervase Markham wrote:
> Dear m.d.s.policy,
>
> Several incidents have come to our attention involving the CA "WoSign".
> Mozilla is considering what action it should take in response to these
> incidents. This email sets out our understanding of the situation.
>
> Before we begin, we note that Section 1 of the Mozilla CA Certificate
> Enforcement Policy[0] says: "When a serious security concern is noticed,
> such as a major root compromise, it should be treated as a
> security-sensitive bug, and the Mozilla Policy for Handling Security
> Bugs should be followed." It is clear to us, and appears to be clear to
> other CAs based on their actions, that misissuances where domain control
> checks have failed fall into the category of "serious security concern".
>
> Incident 0
> ----------
>
> On or around April 23rd, 2015, WoSign's certificate issuance system for
> their free certificates allowed the applicant to choose any port for
> validation. Once validation had been completed, WoSign would
> issue certificates for that domain. A researcher was able to obtain a
> certificate for a university by opening a high-numbered port (>50,000)
> and getting WoSign to use that port for validation of control.
>
> This problem was reported to Google, and thence to WoSign and resolved.
> Mozilla only became aware of it recently.
>
> * Before the recent passage of Ballot 169 in the CAB Forum, which limits
> the ports and paths which can be used, the Baseline Requirements said
> that one acceptable method of domain validation was "Having the
> Applicant demonstrate practical control over the FQDN by making an
> agreed‐upon change to information found on an online Web page identified
> by a uniform resource identifier containing the FQDN". This method
> therefore did not violate the letter of the BRs. However, Mozilla
> considers the basic security knowledge that ports over 1024 are
> unprivileged should have led all CAs not to accept validations of domain
> control on such ports, even when not documented in the BRs.
>
> * The misissuance incident was not reported to Mozilla by WoSign as it
> should have been (see above).
>
> * This misissuance incident did not turn up on WoSign's subsequent BR
> audit[1].
>
> Incident 1
> ----------
>
> In June 2015, an applicant found a problem with WoSign's free
> certificate service, which allowed them to get a certificate for the
> base domain if they were able to prove control of a subdomain.
>
> The reporter proved the problem in two ways. They accidentally
> discovered it when trying to get a certificate for med.ucf.edu and
> mistakenly also applied for www.ucf.edu, which was approved. They then
> confirmed the problem by using their control of
> theiraccount.github.com/theiraccount.github.io to get a cert for
> github.com, github.io, and www.github.io.
>
> They reported this to WoSign, giving only the Github certificate as an
> example. That cert was revoked and the vulnerability was fixed. However
> recently, they got in touch with Google to note that the ucf.edu cert
> still had not been revoked almost a year later.
>
> * The lack of revocation of the ucf.edu certificate (still unrevoked at
> time of writing, although it may have been by time of posting) strongly
> suggests that WoSign either did not or could not search their issuance
> databases for other occurrences of the same problem. Mozilla considers
> such a search a basic part of the response to disclosure of a
> vulnerability which causes misissuance, and expects CAs to keep records
> detailed enough to make it possible.
>
> * This misissuance incident was not reported to Mozilla by WoSign as it
> should have been (see above).
>
> * This misissuance incident did not turn up on WoSign's subsequent BR
> audit[1].
>
> Incident 2
> ----------
>
> In July 2016, it became clear that there was some problems with the
> StartEncrypt automatic issuance service recently deployed by the CA
> StartCom. As well as other problems it had, which are outside the scope
> of this discussion, changing a simple API parameter in the POST request
> on the submission page changed the root certificate to which the
> resulting certificate chained up. The value "2" made a certificate
> signed by "StartCom Class 1 DV Server CA", "1" selected "WoSign CA Free
> SSL Certificate G2" and "0" selected "CA 沃通根证书", another root
> certificate owned by WoSign and trusted by Firefox.
>
> Using the value "1" led to a certificate which had a notBefore date
> (usage start date) of 20th December 2015, and which was signed using the
> SHA-1 checksum algorithm.
>
> * The issuance of certificates using SHA-1 has been banned by the
> Baseline Requirements since January 1st, 2016. Browsers, including
> Firefox, planned to enforce this[2] by not trusting certs with a
> notBefore date after that date, but in the case of Firefox the fix had
> to be backed out due to web compatibility issues. However, we are
> considering how/when to reintroduce it, and CAs presumably know this.
>
> * The issuance of backdated certificates is not forbidden, but is listed
> in Mozilla's list of Problematic Practices[3]. It says "Minor tweaking
> for technical compatibility reasons is accepted, but backdating
> certificates in order to avoid some deadline or code-enforced
> restriction is not."
>
> * WoSign deny that their code backdated the certificates in order to
> avoid browser-based restrictions - they say "this date is the day we
> stop to use this code"[4]. If that is true, it is not clear to us how
> StartCom came to deploy WoSign code that WoSign itself had abandoned.
>
> * It seems clear from publicly available information that StartCom's
> issuance systems are linked to WoSign's issuance systems in some way.
> Nevertheless, it should not have been possible for an application for a
> cert from StartCom to produce a cert signed by WoSign.
>
> * This misissuance incident was not reported to Mozilla by WoSign as it
> should have been.
>
>
> Taking into account all these incidents and the actions of this CA,
> Mozilla is considering what action to take. Your input is welcomed.
>
> Gerv, Kathleen and Richard
>
>
> [0]
> https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/enforcement/
> [1] https://cert.webtrust.org/SealFile?seal=2019&file=pdf
> [2] https://bugzilla.mozilla.org/show_bug.cgi?id=942515
> [3]
> https://wiki.mozilla.org/CA:Problematic_Practices#Backdating_the_notBefore_date
> [4] https://bugzilla.mozilla.org/show_bug.cgi?id=1293366
I consider myself an computer network analyst and have a degree in Web Systems Development. I also host a small number of websites on a technical level. I have used Startcom's services for a number of years. I only recently came across WoSign when I discovered that they offer 3-year valid certificates for free for up to 5 hostnames. That seemed quite an excessively long time in my opinion for a free certificate, almost too good to be true. As a customer of effectively both WoSign and Startcom, I have been having gradually growing concerns for some time about their recent practices and this recent news has not helped. I am currently looking into using other CA's for my needs but will watch what happens next closely before making a decision. I firmly believe WoSign have almost 'purchased' Startcom as I noticed that the nameserver domain for startssl.com (namely startcomca.net) would bring up WoSign's website for a time when accessed via a web browser as www.startcomca.net. I personally emailed Eddy Nigg (Startcom founder) about this, and this was eventually changed to redirect to startssl.com. So it is obvious that WoSign is hosting/sharing Startcom's infrastucture on it's own CA hardware/software, which Eddie confirmed in his email reply. The SSL certificate at the time for www.startcom.org had expired for over a week and still does have many broken links. Using the 'Live Chat' on www.startssl.com had a very confused employee tell me that startcom.org is no longer the website for Startcom, but startssl.com is. I stressed that this was confusing to people and that they should redirect the latter if that's the case. However... startcom.org remains using Israeli-based nameservers. Just how closely related Startcom is to WoSign is unknown, but it does mean that almost anything that renders WoSign vulnerable, also applies to Startcom. I would hope that Startcom re-evaluate their relationship with WoSign following this mess at the very least.
Regarding WoSign in particular, the SHA-1 miss-issued certificates does look like a genuine mistake, being issued off of an SHA-2 intermediate makes their existence rather pointless in the case of backward compatibility, as old operating systems need a whole SHA-1 chain to validate properly. So there is no obvious motive for this API functionality to be left there on purpose, unless.... a rogue employee/subcontractor left it in place for when a SHA-1 hash collision becomes possible? What is likely here is that SHA-1 used to be the default signature choice, and was indeed legacy functionality left behind. It is very concerning however that despite WoSign having an external audit (which all CA's are meant to have) it appears nobody has independently performed a code audit of their API backend to prevent this being possible in a production environment. With regards to the mis-issued certificates applying to whole base domains validated only sub-domain, that prospect is horrifying. With regards to WoSign's API following 301 redirects from one domain to another though, I have noticed that Let's Encrypt's API (another CA at letsencrypt.org) also has this behaviour. This may be standard behaviour of the ACME protocol. Perhaps it would be prudent to decide in overall policy whether 301 redirects of verification URLs are acceptable. There are legitimate reasons why this would happen, for example if a customer owned www.example1.com and www.example2.net, but wanted the latter to always redirect to the other (because for example of a name change, but had many hardcoded https:// links), they would otherwise have to temporarily disable the redirect to get an SSL certificate covering both domains when validating with this method. With regards to ports other than 80 and 443 and 8080 being used for validation, I think the motive of this is because China has a mandatory media licencing requirement (ICP licence) for all servers that operate on either ports. It may not be practical for an individual in China to get one of these licences just to be able to gain an SSL certificate with this validation method. Of course allowing higher numbered ports does not give much excuse. I tested WoSign's website today however and can confirm that they have removed URL-based file verification, and now only allow DNS or email (from WHOIS) validation. All that's well and good, but the fact remains that there could be many other mis-issued certificates already in existence and the scope remains unknown at this point just how many. We only have WoSign's word that all of the mis-issued certificates are known and/or revoked. If it cannot be certain that WoSign have revoked all incorrectly-validated certificates, then the obvious choice is to add the "WoSign CA Free SSL Certificate G2" intermediate to OneCRL, and possibly their DV intermediate too, as these intermediates are known to have mis-issued certificates whilst their other intermediates have always had tighter validation requirements for OV and EV validation. That approach would certainly be painful for existing users and would require WoSign to immediately contact current and past customers to have new certificates issued from a new intermediate, once it is certain that all the bugs in the validation procedures have been resolved. Would it be appropriate to give customers time before such action is taken, given that it would break many websites, or does the risk of data leaks from unknown mis-issued certificates outweigh the need to give customers time? The other option of simply preventing any newly-issued certificates from being accepted in browers does not seem a very good idea given that WoSign have been known to backdate certificates. In fact, I would even suggest that WoSign cease issuing free certificates (and most possibly paid DV certificates) and in the meantime have an independent audit of their backend to ensure all vulnerablities are closed, with the results made public. Only then should
they recommence issuance- at least on an automated scale. GlobalSign did something similar to this when there was speculation that the hacker responsible for compromising Diginotar had also breached them, at great risk to their business but ultimately gained respect in the process. Given the severity of these incidents and the fact they went unreported though I would not be surprised if WoSign's whole roots are blacklisted eventually if WoSign do not get their act together. The promise to completely commit to Certificate Transparency is quite a step, but it does not change what has already happened. WoSign's recent announcement gives me the impression they expect others to manually scour the CT logs once they upload their 2015 certificate data for mis-issuances that need revoking rather than do this themselves. Whilst it may *technically* be 'transparent' with CT, they don't seem in the spirit of things being completely transparent with their practices- in particular the relationship between Startcom and WoSign. At least one of these incidents actually occurred during research into Startcom, and it turned out the API was being shared by both CA's. Given the circumstances, I think it is appropriate to ask just what the relationship is between them. For me, it appears they are now one and the same, while the original 'Startcom' infratructure I once trusted is gathering dust and has been abandoned. I think I'll be taking my needs to another CA very soon if things don't look up!
Samuel Pinder
On Wednesday, 24 August 2016 14:09:02 UTC+1, Gervase Markham wrote:
> Dear m.d.s.policy,
>
> Several incidents have come to our attention involving the CA "WoSign".
> Mozilla is considering what action it should take in response to these
> incidents. This email sets out our understanding of the situation.
>
> Before we begin, we note that Section 1 of the Mozilla CA Certificate
> Enforcement Policy[0] says: "When a serious security concern is noticed,
> such as a major root compromise, it should be treated as a
> security-sensitive bug, and the Mozilla Policy for Handling Security
> Bugs should be followed." It is clear to us, and appears to be clear to
> other CAs based on their actions, that misissuances where domain control
> checks have failed fall into the category of "serious security concern".
>
> ----------
>
> On or around April 23rd, 2015, WoSign's certificate issuance system for
> their free certificates allowed the applicant to choose any port for
> validation. Once validation had been completed, WoSign would
> issue certificates for that domain. A researcher was able to obtain a
> certificate for a university by opening a high-numbered port (>50,000)
> and getting WoSign to use that port for validation of control.
>
> This problem was reported to Google, and thence to WoSign and resolved.
> Mozilla only became aware of it recently.
>
> * Before the recent passage of Ballot 169 in the CAB Forum, which limits
> the ports and paths which can be used, the Baseline Requirements said
> that one acceptable method of domain validation was "Having the
> Applicant demonstrate practical control over the FQDN by making an
> agreed‐upon change to information found on an online Web page identified
> by a uniform resource identifier containing the FQDN". This method
> therefore did not violate the letter of the BRs. However, Mozilla
> considers the basic security knowledge that ports over 1024 are
> unprivileged should have led all CAs not to accept validations of domain
> control on such ports, even when not documented in the BRs.
>
> * The misissuance incident was not reported to Mozilla by WoSign as it
> should have been (see above).
>
> * This misissuance incident did not turn up on WoSign's subsequent BR
> audit[1].
>
> Incident 1
> ----------
>
> In June 2015, an applicant found a problem with WoSign's free
> certificate service, which allowed them to get a certificate for the
> base domain if they were able to prove control of a subdomain.
>
> The reporter proved the problem in two ways. They accidentally
> discovered it when trying to get a certificate for med.ucf.edu and
> mistakenly also applied for www.ucf.edu, which was approved. They then
> confirmed the problem by using their control of
> theiraccount.github.com/theiraccount.github.io to get a cert for
> github.com, github.io, and www.github.io.
>
> They reported this to WoSign, giving only the Github certificate as an
> example. That cert was revoked and the vulnerability was fixed. However
> recently, they got in touch with Google to note that the ucf.edu cert
> still had not been revoked almost a year later.
>
> * The lack of revocation of the ucf.edu certificate (still unrevoked at
> time of writing, although it may have been by time of posting) strongly
> suggests that WoSign either did not or could not search their issuance
> databases for other occurrences of the same problem. Mozilla considers
> such a search a basic part of the response to disclosure of a
> vulnerability which causes misissuance, and expects CAs to keep records
> detailed enough to make it possible.
>
> * This misissuance incident was not reported to Mozilla by WoSign as it
> should have been (see above).
>
> * This misissuance incident did not turn up on WoSign's subsequent BR
> audit[1].
>
> Incident 2
> ----------
>
> In July 2016, it became clear that there was some problems with the
> StartEncrypt automatic issuance service recently deployed by the CA
> StartCom. As well as other problems it had, which are outside the scope
> of this discussion, changing a simple API parameter in the POST request
> on the submission page changed the root certificate to which the
> resulting certificate chained up. The value "2" made a certificate
> signed by "StartCom Class 1 DV Server CA", "1" selected "WoSign CA Free
> SSL Certificate G2" and "0" selected "CA 沃通根证书", another root
> certificate owned by WoSign and trusted by Firefox.
>
> Using the value "1" led to a certificate which had a notBefore date
> (usage start date) of 20th December 2015, and which was signed using the
> SHA-1 checksum algorithm.
>
> * The issuance of certificates using SHA-1 has been banned by the
> Baseline Requirements since January 1st, 2016. Browsers, including
> Firefox, planned to enforce this[2] by not trusting certs with a
> notBefore date after that date, but in the case of Firefox the fix had
> to be backed out due to web compatibility issues. However, we are
> considering how/when to reintroduce it, and CAs presumably know this.
>
> * The issuance of backdated certificates is not forbidden, but is listed
> in Mozilla's list of Problematic Practices[3]. It says "Minor tweaking
> for technical compatibility reasons is accepted, but backdating
> certificates in order to avoid some deadline or code-enforced
> restriction is not."
>
> * WoSign deny that their code backdated the certificates in order to
> avoid browser-based restrictions - they say "this date is the day we
> stop to use this code"[4]. If that is true, it is not clear to us how
> StartCom came to deploy WoSign code that WoSign itself had abandoned.
>
> * It seems clear from publicly available information that StartCom's
> issuance systems are linked to WoSign's issuance systems in some way.
> Nevertheless, it should not have been possible for an application for a
> cert from StartCom to produce a cert signed by WoSign.
>
> * This misissuance incident was not reported to Mozilla by WoSign as it
> should have been.
>
>
> Taking into account all these incidents and the actions of this CA,
> Mozilla is considering what action to take. Your input is welcomed.
>
> Gerv, Kathleen and Richard
>
>
> [0]
> https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/enforcement/
> [1] https://cert.webtrust.org/SealFile?seal=2019&file=pdf
> [2] https://bugzilla.mozilla.org/show_bug.cgi?id=942515
> [3]
> https://wiki.mozilla.org/CA:Problematic_Practices#Backdating_the_notBefore_date
> [4] https://bugzilla.mozilla.org/show_bug.cgi?id=1293366
Gervase Markham
Aug 31, 2016, 9:45:09 AM8/31/16
to mozilla-dev-s...@lists.mozilla.org
On 24/08/16 14:08, Gervase Markham wrote:
> * The issuance of certificates using SHA-1 has been banned by the
> Baseline Requirements since January 1st, 2016. Browsers, including
> Firefox, planned to enforce this[2] by not trusting certs with a
> notBefore date after that date, but in the case of Firefox the fix had
> to be backed out due to web compatibility issues.
Just as a note, this information is incomplete - the enforcement
> * The issuance of certificates using SHA-1 has been banned by the
> Baseline Requirements since January 1st, 2016. Browsers, including
> Firefox, planned to enforce this[2] by not trusting certs with a
> notBefore date after that date, but in the case of Firefox the fix had
> to be backed out due to web compatibility issues.
returned for publicly-trusted CAs in bug
https://bugzilla.mozilla.org/show_bug.cgi?id=1254667 , since Firefox 48.
Gerv
ian....@gmail.com
Aug 31, 2016, 1:06:45 PM8/31/16
to mozilla-dev-s...@lists.mozilla.org
On Tuesday, August 30, 2016 at 1:03:57 AM UTC+2, Percy wrote:
> "Some certificates are revoked after getting report from subscriber, but some still valid, if any subscriber think it must be revoked and replaced new one, please contact us in the system, thanks"
>
> WoSign seems to lack the basic understanding of how a certificate is used in authentication, consequently unfit to be a CA, I call for revocation of WoSign from all root programs immediately. http://www.percya.com/2016/08/chinese-ca-wosign-faces-revocation.html
I concur absolutely and as such, urge all vendors to revoke WoSign as a trusted CA.
> "Some certificates are revoked after getting report from subscriber, but some still valid, if any subscriber think it must be revoked and replaced new one, please contact us in the system, thanks"
>
> WoSign seems to lack the basic understanding of how a certificate is used in authentication, consequently unfit to be a CA, I call for revocation of WoSign from all root programs immediately. http://www.percya.com/2016/08/chinese-ca-wosign-faces-revocation.html
jozef...@gmail.com
Aug 31, 2016, 1:07:03 PM8/31/16
to mozilla-dev-s...@lists.mozilla.org
As an admin I want to check the WoSign Issuer Policy provided by their "WoSign CA Free SSL Certificate G2" certificate.
Issuer Policy is linked to http://www.wosign.com/policy/
This page shows the source code instead of actual policy.
<% Dim strAcceptLanguage strAcceptLanguage=Request.ServerVariables("HTTP_ACCEPT_LANGUAGE") 'response.write strAcceptLanguage if instr(strAcceptLanguage,"zh")>0 then Response.Redirect "cps.htm" else Response.Redirect "cps_e.htm" end if %>
WoSign does not look like trust worthy CA. Unfortunately their certificates are trusted because the StartCom CA is trusted by OS.
Issuer Policy is linked to http://www.wosign.com/policy/
This page shows the source code instead of actual policy.
<% Dim strAcceptLanguage strAcceptLanguage=Request.ServerVariables("HTTP_ACCEPT_LANGUAGE") 'response.write strAcceptLanguage if instr(strAcceptLanguage,"zh")>0 then Response.Redirect "cps.htm" else Response.Redirect "cps_e.htm" end if %>
WoSign does not look like trust worthy CA. Unfortunately their certificates are trusted because the StartCom CA is trusted by OS.
watso...@gmail.com
Aug 31, 2016, 1:07:19 PM8/31/16
to mozilla-dev-s...@lists.mozilla.org
On Tuesday, August 30, 2016 at 8:07:49 PM UTC-7, Richard Wang wrote:
> This case is in the BR report: https://cert.webtrust.org/SealFile?seal=2019&file=pdf
>
> Thanks.
>
> Best Regards,
>
> Richard
>
Dear Richard,
> This case is in the BR report: https://cert.webtrust.org/SealFile?seal=2019&file=pdf
>
> Thanks.
>
> Best Regards,
>
> Richard
>
It's clear WoSign has continuing compliance issues with CA/Browser forum rules, and has repeatedly failed to correct them. Furthermore there has been lots of questions about what it would take to improve CA practices given the degree of incompetence some have practiced, and it's clear penalties would go a long way.
Is there a reason we shouldn't permanently ban WoSign, and all CAs controlled by people involved with it, from the root program? It seems clear that WoSign has been misleading its auditors repeatedly and has insufficient tracking of which certificates are issued, has been aware of these problems for years, and has not been forthright in addressing them. Making an example would do a lot to encourage better CA behavior in general.
Sincerely,
Watson Ladd
Ryan Sleevi
Aug 31, 2016, 2:13:49 PM8/31/16
to mozilla-dev-s...@lists.mozilla.org
On Wednesday, August 31, 2016 at 10:07:19 AM UTC-7, watso...@gmail.com wrote:
> Dear Richard,
>
> It's clear WoSign has continuing compliance issues with CA/Browser forum rules, and has repeatedly failed to correct them. Furthermore there has been lots of questions about what it would take to improve CA practices given the degree of incompetence some have practiced, and it's clear penalties would go a long way.
>
> Is there a reason we shouldn't permanently ban WoSign, and all CAs controlled by people involved with it, from the root program?
For how long?
> Dear Richard,
>
> It's clear WoSign has continuing compliance issues with CA/Browser forum rules, and has repeatedly failed to correct them. Furthermore there has been lots of questions about what it would take to improve CA practices given the degree of incompetence some have practiced, and it's clear penalties would go a long way.
>
> Is there a reason we shouldn't permanently ban WoSign, and all CAs controlled by people involved with it, from the root program?
> It seems clear that WoSign has been misleading its auditors repeatedly and has insufficient tracking of which certificates are issued, has been aware of these problems for years, and has not been forthright in addressing them.
> Making an example would do a lot to encourage better CA behavior in general.
At present, it looks like there's around 60K (active, valid) WoSign certs presently logged in CT logs, either through WoSign's logging or through secondary sources (such as Google's crawling). Richard has indicated that WoSign issued around 115K certificates in 2015 alone, but it's unclear how much overlap that would have; if we assume the worst case, and that they're all unique certs not previously before seen/logged (for example, perhaps due to the Great Firewall causing difficulty for Google's crawler), and if we generously extrapolate the same amounts for 2014, and take a quarter of that for 2013 certs that could still be valid within the 39 month window, we end up with an "Extreme Worst Case" of 319K certs. While I suspect the number if likely much lower, especially when considering when WoSign switched to SHA-2 (as the SHA-1 certs will be invalid by January 1, 2017), let's operate on two assumptions:
1) "Best Case" of say, ~200K valid certs
2) "Worst Case" of, say, ~319K valid certs
If browser vendors/root stores move to distrust WoSign, all of these certs would be invalidated. We know that a number of sites within the Alexa Top 1M are (intentionally) using WoSign, so we can expect a large number of users, across browser vendors, are accessing these sites, and would thus be seeing a significant amount of SSL/TLS error pages if the CA was distrusted. We know that major platform providers (such as Microsoft Azure) have partnered with WoSign as well, and thus further suggest a larger than desired user impact.
Setting aside for a second whether or not distrusting is the right action, let's think about what possible responses.
A) Remove the CA. Users may manually trust it if they re-add it, but it will not be trusted by default.
B) Actively distrust the CA. Even if manually added (by user or enterprise policy), it will not be trusted.
C) Remove the CA. Develop a whitelist of pre-existing certificates to be trusted.
- What form should this whitelist take?
* Shipping it in the binary is unacceptably large.
* Downloading it in full on demand is unacceptably large/unreliable.
* Checking with a central server for serial number can lead to misleading results (WoSign has shown they issue duplicate serials, and nothing would prevent them from doing so in the future)
* Checking with a central server for certificate hash may have privacy considerations.
* Conclusion: Something SafeBrowsing-like would have to be developed ( https://developers.google.com/safe-browsing/v4/ ), which could be months away.
D) Distrust any certificate without appropriate CT information. Whitelist certs before 2016.
- See whitelist problems above
E) Distrust certs without appropriate CT information, wholesale.
- Note: It appears that WoSign is or has had similar issues to Symantec, failing to log to a diverse-enough set of logs to ensure a robust CT implementation. A quick and random sampling shows, for example, that precertificates are only being logged to Google logs (such as for 8-30-16). Thus, unless an implementation is willing to fully trust Google CT logs alone - something Google themselves are unwilling to do - then this suggests that this may be the same as wholesale distrusting.
In effect, a number of these options boil to a set of concerns:
- Distrusting can be significantly disruptive to end-users, potentially habituating them to SSL warnings or errors
- Distrusting potentially could interfere with those who may still want to trust WoSign manually, themselves
- Distrusting in a way that minimizes disruption has concerns for privacy, stability, or timeliness.
I'm not trying to suggest that distrusting is right or wrong, but I am curious for those who would advocate distrusting how browser vendors, such as Google and Mozilla, for example, might appropriately balance the concerns of the broader community, while also minimizing any damage, particularly to their users, and avoiding any reactionary responses.
It would seem like a form of whitelist (whether to continue trusting WoSign with CT enablement, or distrusting but grandfathering in disclosed certs, ala CNNIC) would require active development and assistance from the broader security and privacy community on how best to balance and scale such concerns, and regardless, would take time to implement, test, and deploy, but would be useful and possibly scalable for other future CA incidents. However, are there alternatives or concerns that I omitted from the above list?
In either event, hopefully this thought experiment brings into light the set of concerns that vendors, such as Mozilla and Google, may have to consider, and may help find an appropriate response that reflects the gravity of these incidents, and the handling of them, and may spark the community for ideas and solutions that can help balance those needs.
Richard Wang
Aug 31, 2016, 11:05:57 PM8/31/16
to Ryan Sleevi, mozilla-dev-s...@lists.mozilla.org
Fair enough, thank you, Ryan.
This is my last formal statement for this issue that I am tired of this argument, I need to go to hospital now :-).
First, please treat WoSign as a global trusted CA, DON'T stamp as China CA. We need a fair treatment as other worldwide CAs that I am sure WoSign is not the first CA that have incident and not the serious one;
Second, I supplement some data for your reference, please consider those subscribers benefit, especially from many underdeveloped countries that can't afford the too expansive SSL certificate.
(1) WoSign totally issued 100K SSL certificates in 2015 that we are posting to CT log server (not 115K, Sorry, I used the wrong search condition);
(2) WoSign totally issued 230K SSL certificates till now for worldwide websites about 208 countries and regions. 55% from China, 45% is out of China that 14% from Russia, 4.2% from Germany, 3.6% from Ukraine, 2.1% from USA. Those worldwide subscribers mostly are using WoSign free SSL certificate that shared the benefit of China economy grow.
Third, I believe no one dare to say his system no bug, WoSign admitted we have system bug that issued the wrong certificate and fixed. This is why WoSign is the first CA in the world for volunteering to "Require CT", we like to use CT mechanism to find out the bug quickly and reduce the lost to minimum, we logged all issued certificate to CT log server and embedded SCT data to certificate since July 5th. Thanks for Google invent so good transparency system.
Finally, I am very sorry to all browsers that we don't execute the incident report policy properly, WoSign get a deep lesson for how to deal with this kind of incident now, I wish everyone give us a chance to be a good boy, at least one-time chance. Thanks a million.
Best Regards,
Richard Wang
CEO
WoSign CA Limited
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Ryan Sleevi
Sent: Thursday, September 1, 2016 2:14 AM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
> It seems clear that WoSign has been misleading its auditors repeatedly and has insufficient tracking of which certificates are issued, has been aware of these problems for years, and has not been forthright in addressing them.
This is my last formal statement for this issue that I am tired of this argument, I need to go to hospital now :-).
First, please treat WoSign as a global trusted CA, DON'T stamp as China CA. We need a fair treatment as other worldwide CAs that I am sure WoSign is not the first CA that have incident and not the serious one;
Second, I supplement some data for your reference, please consider those subscribers benefit, especially from many underdeveloped countries that can't afford the too expansive SSL certificate.
(1) WoSign totally issued 100K SSL certificates in 2015 that we are posting to CT log server (not 115K, Sorry, I used the wrong search condition);
(2) WoSign totally issued 230K SSL certificates till now for worldwide websites about 208 countries and regions. 55% from China, 45% is out of China that 14% from Russia, 4.2% from Germany, 3.6% from Ukraine, 2.1% from USA. Those worldwide subscribers mostly are using WoSign free SSL certificate that shared the benefit of China economy grow.
Third, I believe no one dare to say his system no bug, WoSign admitted we have system bug that issued the wrong certificate and fixed. This is why WoSign is the first CA in the world for volunteering to "Require CT", we like to use CT mechanism to find out the bug quickly and reduce the lost to minimum, we logged all issued certificate to CT log server and embedded SCT data to certificate since July 5th. Thanks for Google invent so good transparency system.
Finally, I am very sorry to all browsers that we don't execute the incident report policy properly, WoSign get a deep lesson for how to deal with this kind of incident now, I wish everyone give us a chance to be a good boy, at least one-time chance. Thanks a million.
Best Regards,
Richard Wang
CEO
WoSign CA Limited
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Ryan Sleevi
Sent: Thursday, September 1, 2016 2:14 AM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
On Wednesday, August 31, 2016 at 10:07:19 AM UTC-7, watso...@gmail.com wrote:
> Dear Richard,
>
> It's clear WoSign has continuing compliance issues with CA/Browser forum rules, and has repeatedly failed to correct them. Furthermore there has been lots of questions about what it would take to improve CA practices given the degree of incompetence some have practiced, and it's clear penalties would go a long way.
>
> Is there a reason we shouldn't permanently ban WoSign, and all CAs controlled by people involved with it, from the root program?
For how long?
>
> It's clear WoSign has continuing compliance issues with CA/Browser forum rules, and has repeatedly failed to correct them. Furthermore there has been lots of questions about what it would take to improve CA practices given the degree of incompetence some have practiced, and it's clear penalties would go a long way.
>
> Is there a reason we shouldn't permanently ban WoSign, and all CAs controlled by people involved with it, from the root program?
> It seems clear that WoSign has been misleading its auditors repeatedly and has insufficient tracking of which certificates are issued, has been aware of these problems for years, and has not been forthright in addressing them.
These are all very compelling arguments.
> Making an example would do a lot to encourage better CA behavior in general.
Ryan Sleevi
Sep 1, 2016, 12:45:39 AM9/1/16
to mozilla-dev-s...@lists.mozilla.org
On Wednesday, August 31, 2016 at 8:05:57 PM UTC-7, Richard Wang wrote:
> First, please treat WoSign as a global trusted CA, DON'T stamp as China CA. We need a fair treatment as other worldwide CAs that I am sure WoSign is not the first CA that have incident and not the serious one;
I would have hoped, through all of this, that you would have come to realize the seriousness and gravity of the multiple problems that WoSign has had over the past 18 months, rather than continue to dismiss it.
> First, please treat WoSign as a global trusted CA, DON'T stamp as China CA. We need a fair treatment as other worldwide CAs that I am sure WoSign is not the first CA that have incident and not the serious one;
You misissued certificates to people who were not authorized. Repeatedly. In multiple separate instances.
You have had multiple issues with adhering to the Baseline Requirements, and those have not been disclosed, to your auditors or browsers. Consider Incident -1, which was not listed here in this thread yet, on April 4, 2015, which caused you to update your CPS ( http://www.wosign.com/policy/wosign-policy-1-2-10.pdf ) to correct several issues that Google reported to you regarding non-compliance to your stated policies and to the BRs.
Ultimately, CAs are based on trust:
- Trust that they're performing the necessary validation and vetting procedures
- Trust that they are securing their internal systems from both internal and external threats
- Trust that they understand the seriousness of their role as a provider of global certificates, any of which might be used to compromise or attack users, and take adequate steps to ensure the correctness of those operations.
Your responses, unfortunately, have done little to instill or affirm that the trust is well placed. Even more so, you're a CA authorized to issue the most rigorous certificates available (Extended Validation), and so the bar is set even higher for your operational controls and processes!
- You failed to understand the gravity of the misissuance, and thus failed to revoke these certificates. I would argue this constitutes a further BR violation itself, per Section 4.9.1.1 of the BRs ( https://cabforum.org/wp-content/uploads/CA-Browser-Forum-BR-1.3.8.pdf ), in particular, Items 4 and Items 9 in that list, but would be curious if you disagree.
- You have repeatedly suggested this was an honest mistake, suggesting it was a singular issue, when the reality is that it is a pattern of mistakes that have spanned over a year and have led to multiple misissued certificates.
- These incidents - including the issue with the CPS page - suggest a software development methodology that fails to adequately ensure the robustness of the systems, and fails to understand the security risks that must be mitigated in such systems.
When there are concerns that undermine trust, it's important to engage to restore trust, and so I truly appreciate your involvement in these discussions, and your efforts to restore that trust. And while it's understandable that, in any security incident, there will be disagreements regarding severity or impact, it's vitally important for an entity whose product depends on trust takes steps to understand the issues, understand the perspectives, and takes a careful look to understand why so many people are disagreeing with the risk assessment or mitigation strategy.
> Third, I believe no one dare to say his system no bug, WoSign admitted we have system bug that issued the wrong certificate and fixed.
This is a vast departure from, say, the thread last month - https://groups.google.com/d/msg/mozilla.dev.security.policy/9QKw1C3m4Lo/nkg1rcJyAgAJ - and shows how significant the gap in perception and user trust can be, based on how a CA handles an incident and the subsequent public discussion.
> This is why WoSign is the first CA in the world for volunteering to "Require CT", we like to use CT mechanism to find out the bug quickly and reduce the lost to minimum, we logged all issued certificate to CT log server and embedded SCT data to certificate since July 5th. Thanks for Google invent so good transparency system.
While it's laudable that you're logging your certificates at all, there's two important pieces being omitted here:
- By only logging to Google logs, you're not necessarily improving trust, because now the burden is to trust Google.
- This is an issue I personally informed you about and discussed at length with you on May 25, 2016.
For example, as highlighted in the message you're replying to, if Chrome were to do what you requested - and require CT for WoSign certificates - few to none of the WoSign certificates issued since would be trusted.
Consider this Precertificate - https://crt.sh/?id=30466652 - and its associated certificate, https://crt.sh/?id=30468277 - which you only logged to Google's Pilot and Aviator logs. That means that in order to trust WoSign certificates, you'd have to trust Google as a single point of trust - which, as I highlighted earlier, is something even Google isn't willing to ask of the public.
> Finally, I am very sorry to all browsers that we don't execute the incident report policy properly, WoSign get a deep lesson for how to deal with this kind of incident now, I wish everyone give us a chance to be a good boy, at least one-time chance. Thanks a million.
I realize this may seem like a debate between there "three incidents" (although there have been at least two more BR violating incidents, as highlighted in this message) and "four incidents" (giving you one more chance), but a key goal of this thread was to question whether or not it was believed that WoSign would improve.
Percy
Sep 1, 2016, 2:03:11 AM9/1/16
to mozilla-dev-s...@lists.mozilla.org
WoSign totally issued 230K SSL certificates till now for worldwide websites about 208 countries and regions.
>
>
> If browser vendors/root stores move to distrust WoSign, all of these certs would be invalidated. We know that a number of sites within the Alexa Top 1M are (intentionally) using WoSign, so we can expect a large number of users, across browser vendors, are accessing these sites, and would thus be seeing a significant amount of SSL/TLS error pages if the CA was distrusted. We know that major platform providers (such as Microsoft Azure) have partnered with WoSign as well, and thus further suggest a larger than desired user impact.
// Microsoft Azure China has moved from CNNIC to WoSign and ultimately to DigiCert and I assume many large companies will follow suit. If they have not, they will certainly do once they're aware WoSign is pending removal (if removal is decided)
>
>
> Setting aside for a second whether or not distrusting is the right action, let's think about what possible responses.
>
> A) Remove the CA. Users may manually trust it if they re-add it, but it will not be trusted by default.
> B) Actively distrust the CA. Even if manually added (by user or enterprise policy), it will not be trusted.
> C) Remove the CA. Develop a whitelist of pre-existing certificates to be trusted.
> - What form should this whitelist take?
> * Shipping it in the binary is unacceptably large.
> * Downloading it in full on demand is unacceptably large/unreliable.
> * Checking with a central server for serial number can lead to misleading results (WoSign has shown they issue duplicate serials, and nothing would prevent them from doing so in the future)
> * Checking with a central server for certificate hash may have privacy considerations.
> * Conclusion: Something SafeBrowsing-like would have to be developed ( https://developers.google.com/safe-browsing/v4/ ), which could be months away.
> D) Distrust any certificate without appropriate CT information. Whitelist certs before 2016.
> - See whitelist problems above
> E) Distrust certs without appropriate CT information, wholesale.
> - Note: It appears that WoSign is or has had similar issues to Symantec, failing to log to a diverse-enough set of logs to ensure a robust CT implementation. A quick and random sampling shows, for example, that precertificates are only being logged to Google logs (such as for 8-30-16). Thus, unless an implementation is willing to fully trust Google CT logs alone - something Google themselves are unwilling to do - then this suggests that this may be the same as wholesale distrusting.
>
> In effect, a number of these options boil to a set of concerns:
> - Distrusting can be significantly disruptive to end-users, potentially habituating them to SSL warnings or errors
> - Distrusting potentially could interfere with those who may still want to trust WoSign manually, themselves
> - Distrusting in a way that minimizes disruption has concerns for privacy, stability, or timeliness.
>
> I'm not trying to suggest that distrusting is right or wrong, but I am curious for those who would advocate distrusting how browser vendors, such as Google and Mozilla, for example, might appropriately balance the concerns of the broader community, while also minimizing any damage, particularly to their users, and avoiding any reactionary responses.
On the practical implementation, I suggest the following.
All existing certs issued by WoSign need to be logged by enough CT logs by say Dec 31, 2016.
After Jan 1, 2017, the browser will only trust a cert from WoSign if
1) CT is present
2) The cert is submitted to CT logs before Dec 31, 2016.
Or we can use an offline whitelist. How about include SHA-2 of existing WoSign certificates in the binary? So the browser would first check whether it's signed by WoSign, if so, compare the hash of the cert with the offline list. 224 bit hash * 230K certificate = 6.5 MB. Considering the Chrome installer file is almost 70MB, this might be acceptable.
Ryan Sleevi
Sep 1, 2016, 3:27:11 AM9/1/16
to mozilla-dev-s...@lists.mozilla.org
On Wednesday, August 31, 2016 at 11:03:11 PM UTC-7, Percy wrote:
> Indeed, WoSign has become too big to fail. I would suggest that the decision whether to remove WoSign should be independent of whether it's practical to implement such removal. Otherwise, larger CA basically gained "natural protection" from mere usage numbers over smaller CA in terms of enforcement actions.
Note: I intentionally did not phrase it in terms of practicality of removal, but practicality of options. Whatever decision is made, it must be implementable, otherwise, it's theatre. So you have to consider what is possible when considering what is appropriate. If we allow ourselves to consider the impossible as options, then it might as well say that any CA that screws up needs to travel back in time and unissue the certificates, since surely that too would solve the problem.
> Indeed, WoSign has become too big to fail. I would suggest that the decision whether to remove WoSign should be independent of whether it's practical to implement such removal. Otherwise, larger CA basically gained "natural protection" from mere usage numbers over smaller CA in terms of enforcement actions.
> On the practical implementation, I suggest the following.
> All existing certs issued by WoSign need to be logged by enough CT logs by say Dec 31, 2016.
> After Jan 1, 2017, the browser will only trust a cert from WoSign if
> 1) CT is present
> 2) The cert is submitted to CT logs before Dec 31, 2016.
> Or we can use an offline whitelist. How about include SHA-2 of existing WoSign certificates in the binary? So the browser would first check whether it's signed by WoSign, if so, compare the hash of the cert with the offline list. 224 bit hash * 230K certificate = 6.5 MB. Considering the Chrome installer file is almost 70MB, this might be acceptable.
2) A 100KB increase is unacceptable, especially for mobile users.
Kurt Roeckx
Sep 1, 2016, 5:37:04 AM9/1/16
to mozilla-dev-s...@lists.mozilla.org
On 2016-08-31 20:13, Ryan Sleevi wrote:
> Setting aside for a second whether or not distrusting is the right action, let's think about what possible responses.
>
> A) Remove the CA. Users may manually trust it if they re-add it, but it will not be trusted by default.
> B) Actively distrust the CA. Even if manually added (by user or enterprise policy), it will not be trusted.
> C) Remove the CA. Develop a whitelist of pre-existing certificates to be trusted.
> - What form should this whitelist take?
> * Shipping it in the binary is unacceptably large.
> * Downloading it in full on demand is unacceptably large/unreliable.
> * Checking with a central server for serial number can lead to misleading results (WoSign has shown they issue duplicate serials, and nothing would prevent them from doing so in the future)
> * Checking with a central server for certificate hash may have privacy considerations.
> * Conclusion: Something SafeBrowsing-like would have to be developed ( https://developers.google.com/safe-browsing/v4/ ), which could be months away.
> D) Distrust any certificate without appropriate CT information. Whitelist certs before 2016.
> - See whitelist problems above
> E) Distrust certs without appropriate CT information, wholesale.
> - Note: It appears that WoSign is or has had similar issues to Symantec, failing to log to a diverse-enough set of logs to ensure a robust CT implementation. A quick and random sampling shows, for example, that precertificates are only being logged to Google logs (such as for 8-30-16). Thus, unless an implementation is willing to fully trust Google CT logs alone - something Google themselves are unwilling to do - then this suggests that this may be the same as wholesale distrusting.
An other option is to only trust certificates issued before a certain date.
> Setting aside for a second whether or not distrusting is the right action, let's think about what possible responses.
>
> A) Remove the CA. Users may manually trust it if they re-add it, but it will not be trusted by default.
> B) Actively distrust the CA. Even if manually added (by user or enterprise policy), it will not be trusted.
> C) Remove the CA. Develop a whitelist of pre-existing certificates to be trusted.
> - What form should this whitelist take?
> * Shipping it in the binary is unacceptably large.
> * Downloading it in full on demand is unacceptably large/unreliable.
> * Checking with a central server for serial number can lead to misleading results (WoSign has shown they issue duplicate serials, and nothing would prevent them from doing so in the future)
> * Checking with a central server for certificate hash may have privacy considerations.
> * Conclusion: Something SafeBrowsing-like would have to be developed ( https://developers.google.com/safe-browsing/v4/ ), which could be months away.
> D) Distrust any certificate without appropriate CT information. Whitelist certs before 2016.
> - See whitelist problems above
> E) Distrust certs without appropriate CT information, wholesale.
> - Note: It appears that WoSign is or has had similar issues to Symantec, failing to log to a diverse-enough set of logs to ensure a robust CT implementation. A quick and random sampling shows, for example, that precertificates are only being logged to Google logs (such as for 8-30-16). Thus, unless an implementation is willing to fully trust Google CT logs alone - something Google themselves are unwilling to do - then this suggests that this may be the same as wholesale distrusting.
We seem to have a problem trusting the date in the certificate, so this
might need to be in combination with an SCT from before that date. I
think the easiest way to do this is have the SCT in the OCSP response,
but it would require the server to do OCSP stapling. It would then be
up to the CA to make sure they are submitted to enough logs, that the
OCSP server returns them, and that they inform their clients to make
sure OCSP stapling is turned on.
Kurt
Erwann Abalea
Sep 1, 2016, 8:30:28 AM9/1/16
to mozilla-dev-s...@lists.mozilla.org
Bonjour,
Le jeudi 1 septembre 2016 09:27:11 UTC+2, Ryan Sleevi a écrit :
> On Wednesday, August 31, 2016 at 11:03:11 PM UTC-7, Percy wrote:
[...]
Today, this list contains around 110k certificates, and it's less than 680KB.
Le jeudi 1 septembre 2016 09:27:11 UTC+2, Ryan Sleevi a écrit :
> On Wednesday, August 31, 2016 at 11:03:11 PM UTC-7, Percy wrote:
> > Or we can use an offline whitelist. How about include SHA-2 of existing WoSign certificates in the binary? So the browser would first check whether it's signed by WoSign, if so, compare the hash of the cert with the offline list. 224 bit hash * 230K certificate = 6.5 MB. Considering the Chrome installer file is almost 70MB, this might be acceptable.
>
> 1) SHA-2 is 256-bit, not 224-bit
> 2) A 100KB increase is unacceptable, especially for mobile users.
The whitelist for EV logged before 01/01/15 contained around 180k certificates, each one identified by a 64bits digest, the list was compressed in order to gain 25%, the result was an object slightly larger than 1MB.
>
> 1) SHA-2 is 256-bit, not 224-bit
> 2) A 100KB increase is unacceptable, especially for mobile users.
Today, this list contains around 110k certificates, and it's less than 680KB.
Peter Bowen
Sep 1, 2016, 11:10:45 AM9/1/16
to Richard Wang, Ryan Sleevi, mozilla-dev-s...@lists.mozilla.org
On Wed, Aug 31, 2016 at 8:04 PM, Richard Wang <ric...@wosign.com> wrote:
> (1) WoSign totally issued 100K SSL certificates in 2015 that we are posting to CT log server (not 115K, Sorry, I used the wrong search condition);
> (2) WoSign totally issued 230K SSL certificates till now for worldwide websites about 208 countries and regions.
> (1) WoSign totally issued 100K SSL certificates in 2015 that we are posting to CT log server (not 115K, Sorry, I used the wrong search condition);
As of yesterday, I found 131924 unique certificates (either in final
or precertificate form) in CT logs where the issuer contained the
string "wosign". Of these, 62412 have notBefore with the year 2015.
Of the total set, 20723 have already expired (15.7%). Of the 2015
certs, 9380 are already expired (15.0%).
Based on the 230K total number, it seems save to assume about 196K
certs are probably unexpired at this point. Does that seem accurate?
Ryan Sleevi
Sep 1, 2016, 11:50:16 AM9/1/16
to mozilla-dev-s...@lists.mozilla.org
On Thursday, September 1, 2016 at 5:30:28 AM UTC-7, Erwann Abalea wrote:
> The whitelist for EV logged before 01/01/15 contained around 180k certificates, each one identified by a 64bits digest, the list was compressed in order to gain 25%, the result was an object slightly larger than 1MB.
> Today, this list contains around 110k certificates, and it's less than 680KB.
Note: The EV whitelist uses a probablistic structure that Google was comfortable with when determining EV, since it would also have needed to have the EV policy OIDs, and thus unlikely to be "gamed," but when considering trust, is unacceptably high in the false-positive case when considering a CA that may not be trustworthy.
> The whitelist for EV logged before 01/01/15 contained around 180k certificates, each one identified by a 64bits digest, the list was compressed in order to gain 25%, the result was an object slightly larger than 1MB.
> Today, this list contains around 110k certificates, and it's less than 680KB.
For example, the CNNIC whitelist, as implemented by both Chrome and Firefox, uses the full 32-byte hashes.
If we accept 110K (for your number), that's 3.5 megs (uncompressed). If we accept Peter's estimate, that's 6.3 megs (uncompressed). If we accept the full 230K Richard posted, that's 6.4 megs (uncompressed).
We should probably fork off to the other thread if we're going to consider to discuss whitelist technical solutions, if there is community interest, and we can discuss the various concerns and tradeoffs between probablistic data structures (such as Golomb coded sets like the EV whitelist uses, Bloom filters as was proposed elsewhere in the past, and other forms of compression), as well as run concrete analysis of various implementations.
As of yet, there doesn't seem to be a good inline solution that adequately reflects the facts surrounding it, if a whitelist is determined to be suitable.
- Date based retroactive trust is impaired by known backdating
- Full whitelists are insufficiently large
- Probabablistic whitelists can be gamed with false positives (And I'm actively factoring in the CPU cost involved with signing as part of this threat model, assuming both "worst case" and "reasonable case")
- Online lookups have privacy flaws (same as OCSP)
I say "SafeBrowsing-like", but for sake of the PKIX familiar, it might otherwise be rephrased as "shared certificate *Trust* lists with issuerDistributionPoints", which is effectively the same thing :)
keyc...@gmail.com
Sep 1, 2016, 11:50:32 AM9/1/16
to mozilla-dev-s...@lists.mozilla.org
> It is clear to us, and appears to be clear to
> other CAs based on their actions, that misissuances where domain control
> checks have failed fall into the category of "serious security concern".
>
...
> other CAs based on their actions, that misissuances where domain control
> checks have failed fall into the category of "serious security concern".
>
> * It seems clear from publicly available information that StartCom's
> issuance systems are linked to WoSign's issuance systems in some way.
> Nevertheless, it should not have been possible for an application for a
> cert from StartCom to produce a cert signed by WoSign.
>
> * This misissuance incident was not reported to Mozilla by WoSign as it
> should have been.
>
>
> Taking into account all these incidents and the actions of this CA,
> Mozilla is considering what action to take. Your input is welcomed.
>
I have read the details of this incident made public to date, and in my view, this comes down to two fairly simple questions: Did WoSign betray the public trust, and when mistakes were made were they proactive and transparent in resolving them?
> issuance systems are linked to WoSign's issuance systems in some way.
> Nevertheless, it should not have been possible for an application for a
> cert from StartCom to produce a cert signed by WoSign.
>
> * This misissuance incident was not reported to Mozilla by WoSign as it
> should have been.
>
>
> Taking into account all these incidents and the actions of this CA,
> Mozilla is considering what action to take. Your input is welcomed.
>
The *only* currency for a public CA is trust, and WoSign broke that trust by attesting to control of (critical) TLDs when in fact, such control did not exist. There is no difference between issuing fraudulent certificates to a security researcher as there is to any bad actor. Worse, after being made aware of the issue (and related vulnerabilities in their system), WoSign acted in bad faith by failing to proactively revoke all fraudulent certificates, notify their auditor, or inform Google and Mozilla. The behavior with StartCom only punctuates that continued bad faith and unwillingness to conform with the most basic obligations of a CA.
I probably don't need to remind most of you here how already tenuous is the confidence in the existing public key system generally, and TLS security specifically.
In light of their continued actions, I strongly urge the community to remove WoSign from the respective root trust stores.
Kenn White
https://opencryptoaudit.org/people
Vincent Lynch
Sep 1, 2016, 11:50:51 AM9/1/16
to mozilla-dev-s...@lists.mozilla.org
This may be getting a bit ahead of the discussion, but...
The exact relationship between WoSign and StartCom seems relevant to how these violations should be handled.
Whether browsers decide to distrust WoSign, require CTs for all/future certs, take some other "probationary" decision, or do nothing at all, the relationship between these two CAs needs to be fully understood to properly execute that decision.
If WoSign's violations are a result of bad policies/systems, and they own StartCom, should both CAs not face the same oversight/punitive action? If WoSign certs are to be logged in CT, do StartCom certs also need to be logged? If tomorrow, StartCom was to violate the BRs, is that viewed as a separate incident? Or grouped in with the other violations WoSign has had?
The question of who owns/operates StartCom has been something the CA/Browser community has wondered about for the last few months.
Last night, https://www.letsphish.org was shared to this thread. The contents of that site are currently unavailable for stated legal reasons, but the site can still be accessed through Google's Cache: http://webcache.googleusercontent.com/search?q=cache:https://www.letsphish.org/?part=1
This site made the following claim (and provided supporting documentation):
"Reviewing StartCom registry in the Israeli company directory reveal that on November 1st, 2015 all the shares of the private held company were transfered to a UK based company named "StartCom CA Limited". This company, "StartCom CA" is owned by Gaohua Wang, who is of Chinese nationality."
The site further claims that Gaohua Wang and Richard Wang are the same person.
Previously in this thread, Richard wrote:
"[WoSign] shared some facility with StartCom like CRL and OCSP distribution etc."
However, the claims raised by LetsPhish.org, the connections between StartCom's StartEncrypt system and WoSign's issuance systems, and other assertions (https://pierrekim.github.io/blog/2016-02-16-why-i-stopped-using-startssl-because-of-qihoo-360.html) have made it obvious that we do not *know* very much.
I think Eddy Nigg (founder of StartCom) and/or Richard Wang (of WoSign) should make a statement about this.
The exact relationship between WoSign and StartCom seems relevant to how these violations should be handled.
Whether browsers decide to distrust WoSign, require CTs for all/future certs, take some other "probationary" decision, or do nothing at all, the relationship between these two CAs needs to be fully understood to properly execute that decision.
If WoSign's violations are a result of bad policies/systems, and they own StartCom, should both CAs not face the same oversight/punitive action? If WoSign certs are to be logged in CT, do StartCom certs also need to be logged? If tomorrow, StartCom was to violate the BRs, is that viewed as a separate incident? Or grouped in with the other violations WoSign has had?
The question of who owns/operates StartCom has been something the CA/Browser community has wondered about for the last few months.
Last night, https://www.letsphish.org was shared to this thread. The contents of that site are currently unavailable for stated legal reasons, but the site can still be accessed through Google's Cache: http://webcache.googleusercontent.com/search?q=cache:https://www.letsphish.org/?part=1
This site made the following claim (and provided supporting documentation):
"Reviewing StartCom registry in the Israeli company directory reveal that on November 1st, 2015 all the shares of the private held company were transfered to a UK based company named "StartCom CA Limited". This company, "StartCom CA" is owned by Gaohua Wang, who is of Chinese nationality."
The site further claims that Gaohua Wang and Richard Wang are the same person.
Previously in this thread, Richard wrote:
"[WoSign] shared some facility with StartCom like CRL and OCSP distribution etc."
However, the claims raised by LetsPhish.org, the connections between StartCom's StartEncrypt system and WoSign's issuance systems, and other assertions (https://pierrekim.github.io/blog/2016-02-16-why-i-stopped-using-startssl-because-of-qihoo-360.html) have made it obvious that we do not *know* very much.
I think Eddy Nigg (founder of StartCom) and/or Richard Wang (of WoSign) should make a statement about this.
Ryan Sleevi
Sep 1, 2016, 12:01:15 PM9/1/16
to Richard Wang, mozilla-dev-s...@lists.mozilla.org
On Wed, August 31, 2016 10:09 pm, Richard Wang wrote:
> Thanks for your so detail instruction.
> Yes, we are improved. The two case is happened in 2015 and the mis-issued
> certificate period is only 5 months that we fixed 3 big bugs during the 5
> months.
> For CT, we will improve the posting system.
I had a little trouble parsing this, but let's make sure we're on the same
page. I've continued Gerv's original numbering:
Incident -2: 16 January 2015 - 5 March 2015 - 1,132 BR-violating SHA-1
certificates ( https://cert.webtrust.org/SealFile?seal=2019&file=pdf )
Incident -1: April 4, 2015 - WoSign is informed it's routinely violating
its CPS for issued certificates (
https://www.wosign.com/policy/wosign-policy-1-2-10.pdf )
Incident X: April 9 - April 14, 2015 - 392 duplicate serial numbers
Incident 0: April 23, 2015 - 72 potentially dangerous port-validated
certificates
Incident 1: June, 2015 - 33 unvalidated base-domain from sub-domain
certificates
Incident 2: July, 2016 - At least 1 backdated SHA-1 certificate (was this
the only one? I wasn't clear from
https://groups.google.com/d/msg/mozilla.dev.security.policy/k9PBmyLCi8I/gksYkOTLCwAJ
)
Just making sure we're in agreement about the facts and timelines
surrounding these, so that it's easier than debating 2 or 3 or 5 or more.
> Thanks for your so detail instruction.
> Yes, we are improved. The two case is happened in 2015 and the mis-issued
> certificate period is only 5 months that we fixed 3 big bugs during the 5
> months.
> For CT, we will improve the posting system.
I had a little trouble parsing this, but let's make sure we're on the same
page. I've continued Gerv's original numbering:
Incident -2: 16 January 2015 - 5 March 2015 - 1,132 BR-violating SHA-1
certificates ( https://cert.webtrust.org/SealFile?seal=2019&file=pdf )
Incident -1: April 4, 2015 - WoSign is informed it's routinely violating
its CPS for issued certificates (
https://www.wosign.com/policy/wosign-policy-1-2-10.pdf )
Incident X: April 9 - April 14, 2015 - 392 duplicate serial numbers
Incident 0: April 23, 2015 - 72 potentially dangerous port-validated
certificates
Incident 1: June, 2015 - 33 unvalidated base-domain from sub-domain
certificates
Incident 2: July, 2016 - At least 1 backdated SHA-1 certificate (was this
the only one? I wasn't clear from
https://groups.google.com/d/msg/mozilla.dev.security.policy/k9PBmyLCi8I/gksYkOTLCwAJ
)
Just making sure we're in agreement about the facts and timelines
surrounding these, so that it's easier than debating 2 or 3 or 5 or more.
Andrew Ayer
Sep 1, 2016, 1:41:25 PM9/1/16
to mozilla-dev-s...@lists.mozilla.org
On Thu, 1 Sep 2016 09:00:38 -0700
"Ryan Sleevi" <ry...@sleevi.com> wrote:
> Incident -2: 16 January 2015 - 5 March 2015 - 1,132 BR-violating SHA-1
> certificates ( https://cert.webtrust.org/SealFile?seal=2019&file=pdf )
This was a violation of a "SHOULD NOT" (not a "MUST NOT") issue SHA-1
"Ryan Sleevi" <ry...@sleevi.com> wrote:
> Incident -2: 16 January 2015 - 5 March 2015 - 1,132 BR-violating SHA-1
> certificates ( https://cert.webtrust.org/SealFile?seal=2019&file=pdf )
certificates that expire after 2016. Since issuing SHA-1 certificates
was not forbidden in 2015 and the notAfter date is immaterial to the
risk of SHA-1 collisions[1], it would be unfair and counterproductive
to hold this against WoSign.
Regards,
Andrew
[1] In fact, stockpiling long-lived SHA-1 certs in 2015 would have been
vastly better for the ecosystem than using "legacy" roots or
requesting an exception in 2016.
Percy
Sep 1, 2016, 5:50:31 PM9/1/16
to Gervase Markham, mozilla-dev-s...@lists.mozilla.org
They have confirmed that it's a fake cert. Alibaba knew this prior to my
contact and said they already contacted WoSign.
Percy Alpha(PGP
<https://pgp.mit.edu/pks/lookup?op=vindex&search=0xF30D100F7FE124AE>)
contact and said they already contacted WoSign.
Percy Alpha(PGP
<https://pgp.mit.edu/pks/lookup?op=vindex&search=0xF30D100F7FE124AE>)
Richard Wang
Sep 1, 2016, 8:52:29 PM9/1/16
to Peter Bowen, Ryan Sleevi, mozilla-dev-s...@lists.mozilla.org
The posting to log server still not finished.
Best Regards,
Richard
-----Original Message-----
From: Peter Bowen [mailto:pzb...@gmail.com]
Sent: Thursday, September 1, 2016 11:11 PM
To: Richard Wang <ric...@wosign.com>
Cc: Ryan Sleevi <ry...@sleevi.com>; mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
Best Regards,
Richard
-----Original Message-----
From: Peter Bowen [mailto:pzb...@gmail.com]
Sent: Thursday, September 1, 2016 11:11 PM
To: Richard Wang <ric...@wosign.com>
Cc: Ryan Sleevi <ry...@sleevi.com>; mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
Peter Gutmann
Sep 2, 2016, 12:00:04 AM9/2/16
to Vincent Lynch, mozilla-dev-s...@lists.mozilla.org
Vincent Lynch <vtl...@gmail.com> writes:
>I think Eddy Nigg (founder of StartCom) and/or Richard Wang (of WoSign)
>should make a statement about this.
+1. I'd already asked for something like this earlier and got silence as a
response, which isn't inspiring confidence.
Peter.
Percy
Sep 2, 2016, 1:53:58 AM9/2/16
to mozilla-dev-s...@lists.mozilla.org
The former employee of Starcom and author of https://www.letsphish.org/ took the content down, presumably facing legal pressure from StarCom or Wosign. Please see the full site mirror here https://archive.is/8bSp6
Richard Wang
Sep 2, 2016, 2:01:08 AM9/2/16
to mozilla-dev-s...@lists.mozilla.org
OK I try to say some that I wish I don't violate my company confidential policy.
1. Eddy told me that this guy is the former employee of StartCom, he violates the signed NDA that he must shutdown the site within the limit time. Every re-distribution the wrong information will heavy his penalty (including site cache or mirror site). I am sure every company don't like its former employee to expose company's confidential information.
2. WoSign invested in 5 companies worldwide including in North America, Europe and Asia (China), but my company is a private company that no any liability to expose everything that we don't like to expose. And Mozilla also don't have the policy that every CA must expose its shareholder and director.
3. Please don't bind WoSign incident problem with StartCom, it is two independent company that one registered in China and one located in Israel. StartCom and WoSign have maintained a business relationship for many years since 2011 when WoSign startup CA business. And WoSign root is cross signed by StartCom root due to the problem that root inclusion took long time.
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Peter Gutmann
Sent: Friday, September 2, 2016 11:59 AM
To: Vincent Lynch <vtl...@gmail.com>; mozilla-dev-s...@lists.mozilla.org
Subject: RE: Incidents involving the CA WoSign
Vincent Lynch <vtl...@gmail.com> writes:
>I think Eddy Nigg (founder of StartCom) and/or Richard Wang (of WoSign)
>should make a statement about this.
1. Eddy told me that this guy is the former employee of StartCom, he violates the signed NDA that he must shutdown the site within the limit time. Every re-distribution the wrong information will heavy his penalty (including site cache or mirror site). I am sure every company don't like its former employee to expose company's confidential information.
2. WoSign invested in 5 companies worldwide including in North America, Europe and Asia (China), but my company is a private company that no any liability to expose everything that we don't like to expose. And Mozilla also don't have the policy that every CA must expose its shareholder and director.
3. Please don't bind WoSign incident problem with StartCom, it is two independent company that one registered in China and one located in Israel. StartCom and WoSign have maintained a business relationship for many years since 2011 when WoSign startup CA business. And WoSign root is cross signed by StartCom root due to the problem that root inclusion took long time.
Best Regards,
Richard
-----Original Message-----
Sent: Friday, September 2, 2016 11:59 AM
To: Vincent Lynch <vtl...@gmail.com>; mozilla-dev-s...@lists.mozilla.org
Subject: RE: Incidents involving the CA WoSign
Vincent Lynch <vtl...@gmail.com> writes:
>I think Eddy Nigg (founder of StartCom) and/or Richard Wang (of WoSign)
>should make a statement about this.
+1. I'd already asked for something like this earlier and got silence
+as a
response, which isn't inspiring confidence.
Peter.
Peter.
Percy
Sep 2, 2016, 2:23:27 AM9/2/16
to mozilla-dev-s...@lists.mozilla.org
On Thursday, September 1, 2016 at 11:01:08 PM UTC-7, Richard Wang wrote:
> OK I try to say some that I wish I don't violate my company confidential policy.
>
> 1. Eddy told me that this guy is the former employee of StartCom, he violates the signed NDA that he must shutdown the site within the limit time. Every re-distribution the wrong information will heavy his penalty (including site cache or mirror site). I am sure every company don't like its former employee to expose company's confidential information.
>
In case WoSign tries to send take down request to Achieve.is, I mirrored the content on pastebin too http://pastebin.com/hiKxmGMH Good luck taking that down.
> 2. WoSign invested in 5 companies worldwide including in North America, Europe and Asia (China), but my company is a private company that no any liability to expose everything that we don't like to expose. And Mozilla also don't have the policy that every CA must expose its shareholder and director.
>
> 3. Please don't bind WoSign incident problem with StartCom, it is two independent company that one registered in China and one located in Israel. StartCom and WoSign have maintained a business relationship for many years since 2011 when WoSign startup CA business. And WoSign root is cross signed by StartCom root due to the problem that root inclusion took long time.
>
Richard Wang
Sep 2, 2016, 2:36:13 AM9/2/16
to Percy, mozilla-dev-s...@lists.mozilla.org
Please remember this sentence:
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Percy
Sent: Friday, September 2, 2016 2:23 PM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
On Thursday, September 1, 2016 at 11:01:08 PM UTC-7, Richard Wang wrote:
> OK I try to say some that I wish I don't violate my company confidential policy.
>
> 1. Eddy told me that this guy is the former employee of StartCom, he violates the signed NDA that he must shutdown the site within the limit time. Every re-distribution the wrong information will heavy his penalty (including site cache or mirror site). I am sure every company don't like its former employee to expose company's confidential information.
>
Every re-distribution the wrong information will heavy his penalty (including site cache or mirror site).
You are harming him!
Best Regards,
Richard
-----Original Message-----
Sent: Friday, September 2, 2016 2:23 PM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
On Thursday, September 1, 2016 at 11:01:08 PM UTC-7, Richard Wang wrote:
> OK I try to say some that I wish I don't violate my company confidential policy.
>
> 1. Eddy told me that this guy is the former employee of StartCom, he violates the signed NDA that he must shutdown the site within the limit time. Every re-distribution the wrong information will heavy his penalty (including site cache or mirror site). I am sure every company don't like its former employee to expose company's confidential information.
>
NDA only applies for information that's privileged. The content here https://archive.is/8bSp6 can be obtained all from public sources, hence exempted from NDA.
In case WoSign tries to send take down request to Achieve.is, I mirrored the content on pastebin too http://pastebin.com/hiKxmGMH Good luck taking that down.
In case WoSign tries to send take down request to Achieve.is, I mirrored the content on pastebin too http://pastebin.com/hiKxmGMH Good luck taking that down.
> 2. WoSign invested in 5 companies worldwide including in North America, Europe and Asia (China), but my company is a private company that no any liability to expose everything that we don't like to expose. And Mozilla also don't have the policy that every CA must expose its shareholder and director.
>
>
Sure, your company is a private company. But the public doesn't have an obligation to trust you either.
> 3. Please don't bind WoSign incident problem with StartCom, it is two independent company that one registered in China and one located in Israel. StartCom and WoSign have maintained a business relationship for many years since 2011 when WoSign startup CA business. And WoSign root is cross signed by StartCom root due to the problem that root inclusion took long time.
>
>
Two independent companies that share the same infrastructure, director and user trust according to https://archive.is/8bSp6 , doesn't look very independent to me.
>
>
Matt Palmer
Sep 2, 2016, 2:49:05 AM9/2/16
to dev-secur...@lists.mozilla.org
On Fri, Sep 02, 2016 at 05:59:19AM +0000, Richard Wang wrote:
> 1. Eddy told me that this guy is the former employee of StartCom, he
> violates the signed NDA that he must shutdown the site within the limit
> time. Every re-distribution the wrong information will heavy his penalty
> (including site cache or mirror site). I am sure every company don't like
> its former employee to expose company's confidential information.
I don't see anything particularly confidential, and waving around legal
> 1. Eddy told me that this guy is the former employee of StartCom, he
> violates the signed NDA that he must shutdown the site within the limit
> time. Every re-distribution the wrong information will heavy his penalty
> (including site cache or mirror site). I am sure every company don't like
> its former employee to expose company's confidential information.
threats really does seem like there's something to hide. Why not address
the concerns raised by that site, rather than shutting it down, if the
accusations are entirely baseless?
> 2. WoSign invested in 5 companies worldwide including in North America,
> Europe and Asia (China), but my company is a private company that no any
> liability to expose everything that we don't like to expose. And Mozilla
> also don't have the policy that every CA must expose its shareholder and
> director.
of fairly egregious breaches of the public trust, either.
> 3. Please don't bind WoSign incident problem with StartCom, it is two
> independent company that one registered in China and one located in
> Israel.
- Matt
Percy
Sep 2, 2016, 2:55:06 AM9/2/16
to mozilla-dev-s...@lists.mozilla.org
On Thursday, September 1, 2016 at 11:36:13 PM UTC-7, Richard Wang wrote:
> Please remember this sentence:
> Every re-distribution the wrong information will heavy his penalty (including site cache or mirror site).
>
> You are harming him!
You stated that he was a former employee of StartCom in 2015. After he left the company, what he learnt from public sources in 2016 is not bound by NDA. I do not appreciate you holding him hostage to suppress public and crucial information on understanding the trust of CA. Since WoSign is trying so hard to suppress such critical information, it's especially important for us to understand the consequences of such info. The entire article is reproduced below.
> Please remember this sentence:
> Every re-distribution the wrong information will heavy his penalty (including site cache or mirror site).
>
> You are harming him!
--------
Being a Certificate Authority is all about trust.
Start Commercial LTD "is" an Israeli Certificate Authority, Their certificates are trusted by billion of devices (computers, mobile phones, routers, etc) and they claim to be "the 6th biggest CA in the world". StartCom launched it's activities as we know it today around 2006 with the brandname StartSSL.
Their site didn't had much UI changes during those years. Until 2016...
February 16th, 2016, Pierre Kim in his security blog wrote about why he stopped using StartSSL. The article was about how some of StartSSL's infrastructure is hosted in China/by Chinese companies. But he showed only small part of the whole picture, not going into who owns StartCom and the brandname StartSSL.
Reviewing StartCom registry in the Israeli company directory reveal that on November 1st, 2015 all the shares of the private held company were transfered to a UK based company named "StartCom CA Limited". This company, "StartCom CA" is owned by Gaohua Wang, who is of Chinese nationality.
But no news about it. 2016 is a major year for StartCom, new UI, new tools and new features, and yet, no news regarding the new ownership. The only news related to the matter was a minor post about expending their activities in China.
In the previous part we saw that the ownership of the company has switched, from Israeli hands to Chinese hands (via a UK based company to operate as a front organization). Pierre Kim in his blog post showed that some of StartSSL infrastructure is hosted in China/by Chinese companies. In this part I will present that currently (June 2016) StartSSL is operating from China (their employees are located in China).
During the first half year of 2016 I've contacted StartSSL several times. The first time was when I notified them about their SPF TXT records being incorrect [1], the reply was originated from 113.104.213.84 (China Telecom, CHINANET Guangdong province network) with the "Content-Language" equals to "zh-cn" and the localtime of the email was UTC+0800. The email is signed with "certm...@startssl.com" private key.
The second time I've contacted StartSSL was in regard their OCSP replies for expired certificates [2], again the reply was originated in China 183.37.124.147 (China Telecom, CHINANET Guangdong province network) with China's localtime (UTC+0800).
The third and last time I've contacted StartSSL was regarding their expired certificates on some of their hosts [3], this time the reply seem to be generated via some kind of a ticket system, but still from China. The ticket system itself (MX server at least) seem to be in China, 124.251.21.41 (21ViaNet(China),Inc), and the person who replied to my email was also from China, 14.153.60.139 (China Telecom, CHINANET Guangdong province network) with "Accept-Language:" set to "zh-cn".
And what about StartSSL automated emails, old ones (during January) seem to originate in China, they came from 106.39.1.130 (China Telecom, CHINANET-BJ) [4]. But later ones, come from 104.192.108.9-10 (China Telecom (Americas) Corrporation (CTUC)) [5]. According the the whois, this is a Chinese company with an IP infrastructure in the US, but the localtime is still set to China's localtime.
In part 1 I showed that all shares from Start Commercial LTD (company based in Israel) were transferred to a front organization in the UK, named "StartCom CA Limited", which their sole director is Gaohua Wang. In part 2 I showed that StartSSL is actually operating from China (last verified, June 2016). In this part I will disclose who actually owns StartCom and more specifically the "StartSSL" brandname.
The key figure is Gaohua Wang (aka Richard Wang). It may not be so easy to connect him to the company in matter (searching for "Gaohua Wang certificate authority" will do the trick), but Gaohua Wang is also a director of another CA company based in China, named WoSign [1].
StartCom doesn't share this information with their customers, past, present and probably near future. I even tried to ask them directly via their Live Chat, but they haven't given me a straight answer ("not really", "close relationship" and "share infrastructure") [2] [3]. It seem StartCom is trying really hard not to disclose that StartCom was sold indirectly to a Chinese company.
Lets break down the answers to the question "Did WoSign bought you?"
"Not really" - WoSign didn't bought StartCom directly, Gaohua Wang (which also owns WoSign) used a front organization in the UK to buy StartCom.
"Close Relationship" - StartCom in the past cross-signed some of WoSign's intermediate CA, you may consider it as "close relations".
"Share Infrastructure" - This will explain Pierre Kim's post, but it doesn't explain why StartCom will require that, most StartSSL's customers are in Europe and in the US, not in China nor Asia [4].
But there are holes in the story. Why the operations (mail replies, core service like 'auth.startssl.com') is in China? When trying to dial the Israeli number (+972.8.634.4170) I got an unplugged number tone [5], is the office in Israel is unavailable? But some of StartCom infrastructure is still hosted in Israel.
I will conclude with that, the same person (Gaohua Wang) owns WoSign and StartCom. I will leave connecting the dots for you...
Richard Wang
Sep 2, 2016, 2:55:21 AM9/2/16
to Matt Palmer, dev-secur...@lists.mozilla.org
I think we are out of topic.
Please send me offlist email if you like.
Best Regards,
Richard
-----Original Message-----
Please send me offlist email if you like.
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Matt Palmer
Sent: Friday, September 2, 2016 2:49 PM
To: dev-secur...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
Sent: Friday, September 2, 2016 2:49 PM
To: dev-secur...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
Richard Wang
Sep 2, 2016, 3:39:45 AM9/2/16
to Peter Bowen, Ryan Sleevi, mozilla-dev-s...@lists.mozilla.org
We finished the CT posting, all 2015 issued SSL certificate is posted to WoSign CT log server: https://ctlog.wosign.com, total 101,410 certificates.
Best Regards,
Richard
Best Regards,
Richard
Sent: Thursday, September 1, 2016 11:11 PM
To: Richard Wang <ric...@wosign.com>
Cc: Ryan Sleevi <ry...@sleevi.com>; mozilla-dev-s...@lists.mozilla.org
To: Richard Wang <ric...@wosign.com>
Cc: Ryan Sleevi <ry...@sleevi.com>; mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
Percy
Sep 2, 2016, 3:49:46 AM9/2/16
to mozilla-dev-s...@lists.mozilla.org
Just wrote a blog post on this. WoSign's secret purchase of StartCom; WoSign threatened legal actions over the disclosure http://www.percya.com/2016/09/wosigns-secret-purchase-of-startcom.html
Kurt Roeckx
Sep 2, 2016, 3:59:52 AM9/2/16
to mozilla-dev-s...@lists.mozilla.org
Kurt
Matt Palmer
Sep 2, 2016, 4:51:02 AM9/2/16
to dev-secur...@lists.mozilla.org
On Fri, Sep 02, 2016 at 06:53:23AM +0000, Richard Wang wrote:
> I think we are out of topic.
On the contrary, the trustworthiness of CAs is *entirely* on topic.
> I think we are out of topic.
- Matt
Richard Wang
Sep 2, 2016, 5:03:44 AM9/2/16
to dev-secur...@lists.mozilla.org
You mean if a Chinese, a Chinese company own a USA CA, then the USA CA become un-trustworthiness?
I still think this topic is out of THE Topic - Incident.
Best Regards,
Richard
-----Original Message-----
I still think this topic is out of THE Topic - Incident.
Best Regards,
Richard
-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Matt Palmer
Sent: Friday, September 2, 2016 4:51 PM
To: dev-secur...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
To: dev-secur...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
Gervase Markham
Sep 2, 2016, 6:07:46 AM9/2/16
to Richard Wang
Hi Richard,
On 01/09/16 04:04, Richard Wang wrote:
> First, please treat WoSign as a global trusted CA, DON'T stamp as
> China CA. We need a fair treatment as other worldwide CAs that I am
> sure WoSign is not the first CA that have incident and not the
> serious one;
We are keen to treat WoSign as a global CA. It's certainly true that we
would be having this discussion about any other global CA which had had
such a list of incidents. However, it seems that you are advancing
arguments - such as "we are Chinese; we can't be expected to fully
understand standards written in English" - which ask for special
consideration as a Chinese CA rather than a global CA. And, as others
have pointed out in this thread, WoSign is very happy to be seen as a
China CA for marketing purposes inside China.
> Second, I supplement some data for your reference, please consider
> those subscribers benefit, especially from many underdeveloped
> countries that can't afford the too expansive SSL certificate.
WoSign is not the only company to offer free SSL certificates. But also,
this seems like arguing "we're too big for you to take action against us".
> Third, I believe no one dare to say his system no bug, WoSign
> admitted we have system bug that issued the wrong certificate and
> fixed. This is why WoSign is the first CA in the world for
> volunteering to "Require CT", we like to use CT mechanism to find out
> the bug quickly and reduce the lost to minimum,
That seems to me to be outsourcing your quality control to a set of
third parties. I would say that any CA should have independent systems
which check every certificate issued, before it is sent to the customer,
for a long list of possible faults, and hold the certificate for manual
review if any of those faults are found. That's what I'd do if I were
running a CA, anyway. Saying "it's all in CT, so we can find problems
after issuance" does not, to my mind, take misissuance appropriately
seriously.
> Thanks a million.
(Just as a note, I would advise against using this English phrase, as it
has acquired a sarcastic overtone in normal usage.)
Gerv
On 01/09/16 04:04, Richard Wang wrote:
> First, please treat WoSign as a global trusted CA, DON'T stamp as
> China CA. We need a fair treatment as other worldwide CAs that I am
> sure WoSign is not the first CA that have incident and not the
> serious one;
We are keen to treat WoSign as a global CA. It's certainly true that we
would be having this discussion about any other global CA which had had
such a list of incidents. However, it seems that you are advancing
arguments - such as "we are Chinese; we can't be expected to fully
understand standards written in English" - which ask for special
consideration as a Chinese CA rather than a global CA. And, as others
have pointed out in this thread, WoSign is very happy to be seen as a
China CA for marketing purposes inside China.
> Second, I supplement some data for your reference, please consider
> those subscribers benefit, especially from many underdeveloped
> countries that can't afford the too expansive SSL certificate.
WoSign is not the only company to offer free SSL certificates. But also,
this seems like arguing "we're too big for you to take action against us".
> Third, I believe no one dare to say his system no bug, WoSign
> admitted we have system bug that issued the wrong certificate and
> fixed. This is why WoSign is the first CA in the world for
> volunteering to "Require CT", we like to use CT mechanism to find out
> the bug quickly and reduce the lost to minimum,
That seems to me to be outsourcing your quality control to a set of
third parties. I would say that any CA should have independent systems
which check every certificate issued, before it is sent to the customer,
for a long list of possible faults, and hold the certificate for manual
review if any of those faults are found. That's what I'd do if I were
running a CA, anyway. Saying "it's all in CT, so we can find problems
after issuance" does not, to my mind, take misissuance appropriately
seriously.
> Thanks a million.
(Just as a note, I would advise against using this English phrase, as it
has acquired a sarcastic overtone in normal usage.)
Gerv
Gervase Markham
Sep 2, 2016, 6:10:28 AM9/2/16
to Richard Wang
Hi Richard,
On 02/09/16 06:59, Richard Wang wrote:
> 1. Eddy told me that this guy is the former employee of StartCom, he
> violates the signed NDA that he must shutdown the site within the
> limit time. Every re-distribution the wrong information will heavy
> his penalty (including site cache or mirror site). I am sure every
> company don't like its former employee to expose company's
> confidential information.
What information on that site is company-confidential? It all seems to
point to public sources.
Also, it would help if you pointed out what information is incorrect, so
we can make sure we don't accidentally accept any information which is
incorrect.
Gerv
On 02/09/16 06:59, Richard Wang wrote:
> 1. Eddy told me that this guy is the former employee of StartCom, he
> violates the signed NDA that he must shutdown the site within the
> limit time. Every re-distribution the wrong information will heavy
> his penalty (including site cache or mirror site). I am sure every
> company don't like its former employee to expose company's
> confidential information.
point to public sources.
Also, it would help if you pointed out what information is incorrect, so
we can make sure we don't accidentally accept any information which is
incorrect.
Gerv
Gervase Markham
Sep 2, 2016, 6:20:02 AM9/2/16
to Ryan Sleevi
On 31/08/16 19:13, Ryan Sleevi wrote:
> A) Remove the CA. Users may manually trust it if they re-add it, but it will not be trusted by default.
....
> A) Remove the CA. Users may manually trust it if they re-add it, but it will not be trusted by default.
F) Distrust all certs with a notBefore date after date X, and require
the CA to apply for re-inclusion to get the distrust lifted. (I.e. what
happened to CNNIC.) It's theoretically possible for a CA to backdate
notBefore, but if they are logging everything to CT, that will be
noticable. And if they didn't log to CT, they would be breaking their
promise to log everything to CT, which would be evidence of
untrustworthiness.
Gerv
Richard Wang
Sep 2, 2016, 6:28:53 AM9/2/16
to Gervase Markham, mozilla-dev-s...@lists.mozilla.org
Hi Gerv,
Forgive me my bad English, you know my English level. :-)
It seems my bad English mislead you to wrong place, so I try to correct:
(1) Don't care about the marketing word, it is an advertisement;
(2) What I mean is please think about the current users if any action; 10% from government website, 6 customers is the top 10 eCommerce website in China;
(3) We have quality control; I will send the blocking system screenshot to you since this mail list can't send. But we think CT is a good solution for mis-issued problem.
Best Regards,
Richard
-----Original Message-----
From: Gervase Markham [mailto:ge...@mozilla.org]
Sent: Friday, September 2, 2016 6:07 PM
To: Richard Wang <ric...@wosign.com>; mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
Forgive me my bad English, you know my English level. :-)
It seems my bad English mislead you to wrong place, so I try to correct:
(1) Don't care about the marketing word, it is an advertisement;
(2) What I mean is please think about the current users if any action; 10% from government website, 6 customers is the top 10 eCommerce website in China;
(3) We have quality control; I will send the blocking system screenshot to you since this mail list can't send. But we think CT is a good solution for mis-issued problem.
Best Regards,
Richard
-----Original Message-----
From: Gervase Markham [mailto:ge...@mozilla.org]
Sent: Friday, September 2, 2016 6:07 PM
To: Richard Wang <ric...@wosign.com>; mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
Richard Wang
Sep 2, 2016, 9:30:33 AM9/2/16
to Gervase Markham, mozilla-dev-s...@lists.mozilla.org
-----Original Message-----
From: Gervase Markham [mailto:ge...@mozilla.org]
Sent: Friday, September 2, 2016 6:07 PM
To: Richard Wang <ric...@wosign.com>; mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
> And, as others have pointed out in this thread, WoSign is very happy to be seen as a China CA for marketing purposes inside China.
We deleted the related two pages.
From: Gervase Markham [mailto:ge...@mozilla.org]
Sent: Friday, September 2, 2016 6:07 PM
To: Richard Wang <ric...@wosign.com>; mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incidents involving the CA WoSign
> And, as others have pointed out in this thread, WoSign is very happy to be seen as a China CA for marketing purposes inside China.
Best Regards,
Richard
Peter Bowen
Sep 2, 2016, 10:55:45 AM9/2/16
to Richard Wang, Ryan Sleevi, mozilla-dev-s...@lists.mozilla.org
On Fri, Sep 2, 2016 at 12:37 AM, Richard Wang <ric...@wosign.com> wrote:
> We finished the CT posting, all 2015 issued SSL certificate is posted to WoSign CT log server: https://ctlog.wosign.com, total 101,410 certificates.
Richard,
> We finished the CT posting, all 2015 issued SSL certificate is posted to WoSign CT log server: https://ctlog.wosign.com, total 101,410 certificates.
Based on CT logs, I have seen certificates from the CAs below, all of
which have "WoSign" in the name. Have you logged all certificates
which are signed by these CAs and have a notBefore date of
20150101000000Z or later to the WoSign CT log?
Do you also plan to submit these to at least one Google-operated log?
Thanks,
Peter
Peter Bowen
Sep 2, 2016, 10:56:33 AM9/2/16
to Richard Wang, Ryan Sleevi, mozilla-dev-s...@lists.mozilla.org
(forgot the list)
CN=360 EV Server CA G2,O=WoSign CA Limited,C=CN
CN=360 OV Server CA G2,O=WoSign CA Limited,C=CN
CN=CA WoSign ECC Root,O=WoSign CA Limited,C=CN
CN=CA 沃通 Email 客户端证书 G2,O=WoSign CA Limited,C=CN
CN=CA 沃通 EV Pro 服务器证书 G2,O=WoSign CA Limited,C=CN
CN=CA 沃通 EV 服务器证书 G2,O=WoSign CA Limited,C=CN
CN=CA 沃通 IV 服务器证书 G2,O=WoSign CA Limited,C=CN
CN=CA 沃通 OV Pro 服务器证书 G2,O=WoSign CA Limited,C=CN
CN=CA 沃通 OV 服务器证书 G2,O=WoSign CA Limited,C=CN
CN=CA 沃通免费SSL证书 G2,O=WoSign CA Limited,C=CN
CN=CA 沃通免费SSL证书,O=WoSign CA Limited,C=CN
CN=CA 沃通根证书,O=WoSign CA Limited,C=CN
CN=Certification Authority of WoSign G2,O=WoSign CA Limited,C=CN
CN=Certification Authority of WoSign,O=WoSign CA Limited,C=CN
CN=Certification Authority of WoSign,O=WoSign eCommerce Services Limited,C=CN
CN=GZCA EV Server CA,OU=WoSign Trust Network,O=广州市电子签名中心,C=CN
CN=IMS CA,O=WoSign CA Limited,C=CN
CN=IMS Class 4 EV Pro Server CA,O=WoSign CA Limited,C=CN
CN=IMS Class 4 EV Server CA,O=WoSign CA Limited,C=CN
CN=IMS CT Monitor CA,O=WoSign CA Limited,C=CN
CN=IMS EV 服务器根证书,O=WoSign CA Limited,C=CN
CN=IMS 根证书,O=WoSign CA Limited,C=CN
CN=WoSign CA Free SSL Certificate G2,O=WoSign CA Limited,C=CN
CN=WoSign CA Free SSL Certificate,O=WoSign CA Limited,C=CN
CN=WoSign Class 1 Client CA G2,O=WoSign CA Limited,C=CN
CN=WoSign Class 1 Client CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 1 DV Server CA G2,O=WoSign CA Limited,C=CN
CN=WoSign Class 1 DV Server CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 1 DV Server CA,O=WoSign eCommerce Services Limited,C=CN
CN=WoSign Class 2 IV Server CA G2,O=WoSign CA Limited,C=CN
CN=WoSign Class 2 IV Server CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 3 Code Signing CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 3 OV Pro Server CA G2,O=WoSign CA Limited,C=CN
CN=WoSign Class 3 OV Pro Server CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 3 OV Server CA G2,O=WoSign CA Limited,C=CN
CN=WoSign Class 3 OV Server CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 3 OV Server CA,O=WoSign eCommerce Services Limited,C=CN
CN=WoSign Class 4 EV ECC Server CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 4 EV ECC SSL CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 4 EV Pro Server CA G2,O=WoSign CA Limited,C=CN
CN=WoSign Class 4 EV Pro Server CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 4 EV Server CA G2,O=WoSign CA Limited,C=CN
CN=WoSign Class 4 EV Server CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 4 EV SSL CA G2,O=WoSign CA Limited,C=CN
CN=WoSign Premium Server Authority,O=WoSign\, Inc.,C=US
CN=WoSign Server Authority,O=WoSign\, Inc.,C=US
CN=WoSign SGC Server Authority,O=WoSign\, Inc.,C=US
CN=中国湖南 EV 服务器证书,OU=WoSign Trust Network,O=东方新诚信数字认证中心,C=CN
CN=沃通 Email 客户端根证书,O=WoSign CA Limited,C=CN
CN=沃通 EV 服务器根证书,O=WoSign CA Limited,C=CN
L=ShenZhen,ST=GuangDong,O=WoSign CT Monitor Intermediate,C=CN
CN=360 OV Server CA G2,O=WoSign CA Limited,C=CN
CN=CA WoSign ECC Root,O=WoSign CA Limited,C=CN
CN=CA 沃通 Email 客户端证书 G2,O=WoSign CA Limited,C=CN
CN=CA 沃通 EV Pro 服务器证书 G2,O=WoSign CA Limited,C=CN
CN=CA 沃通 EV 服务器证书 G2,O=WoSign CA Limited,C=CN
CN=CA 沃通 IV 服务器证书 G2,O=WoSign CA Limited,C=CN
CN=CA 沃通 OV Pro 服务器证书 G2,O=WoSign CA Limited,C=CN
CN=CA 沃通 OV 服务器证书 G2,O=WoSign CA Limited,C=CN
CN=CA 沃通免费SSL证书 G2,O=WoSign CA Limited,C=CN
CN=CA 沃通免费SSL证书,O=WoSign CA Limited,C=CN
CN=CA 沃通根证书,O=WoSign CA Limited,C=CN
CN=Certification Authority of WoSign G2,O=WoSign CA Limited,C=CN
CN=Certification Authority of WoSign,O=WoSign CA Limited,C=CN
CN=Certification Authority of WoSign,O=WoSign eCommerce Services Limited,C=CN
CN=GZCA EV Server CA,OU=WoSign Trust Network,O=广州市电子签名中心,C=CN
CN=IMS CA,O=WoSign CA Limited,C=CN
CN=IMS Class 4 EV Pro Server CA,O=WoSign CA Limited,C=CN
CN=IMS Class 4 EV Server CA,O=WoSign CA Limited,C=CN
CN=IMS CT Monitor CA,O=WoSign CA Limited,C=CN
CN=IMS EV 服务器根证书,O=WoSign CA Limited,C=CN
CN=IMS 根证书,O=WoSign CA Limited,C=CN
CN=WoSign CA Free SSL Certificate G2,O=WoSign CA Limited,C=CN
CN=WoSign CA Free SSL Certificate,O=WoSign CA Limited,C=CN
CN=WoSign Class 1 Client CA G2,O=WoSign CA Limited,C=CN
CN=WoSign Class 1 Client CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 1 DV Server CA G2,O=WoSign CA Limited,C=CN
CN=WoSign Class 1 DV Server CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 1 DV Server CA,O=WoSign eCommerce Services Limited,C=CN
CN=WoSign Class 2 IV Server CA G2,O=WoSign CA Limited,C=CN
CN=WoSign Class 2 IV Server CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 3 Code Signing CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 3 OV Pro Server CA G2,O=WoSign CA Limited,C=CN
CN=WoSign Class 3 OV Pro Server CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 3 OV Server CA G2,O=WoSign CA Limited,C=CN
CN=WoSign Class 3 OV Server CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 3 OV Server CA,O=WoSign eCommerce Services Limited,C=CN
CN=WoSign Class 4 EV ECC Server CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 4 EV ECC SSL CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 4 EV Pro Server CA G2,O=WoSign CA Limited,C=CN
CN=WoSign Class 4 EV Pro Server CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 4 EV Server CA G2,O=WoSign CA Limited,C=CN
CN=WoSign Class 4 EV Server CA,O=WoSign CA Limited,C=CN
CN=WoSign Class 4 EV SSL CA G2,O=WoSign CA Limited,C=CN
CN=WoSign Premium Server Authority,O=WoSign\, Inc.,C=US
CN=WoSign Server Authority,O=WoSign\, Inc.,C=US
CN=WoSign SGC Server Authority,O=WoSign\, Inc.,C=US
CN=中国湖南 EV 服务器证书,OU=WoSign Trust Network,O=东方新诚信数字认证中心,C=CN
CN=沃通 Email 客户端根证书,O=WoSign CA Limited,C=CN
CN=沃通 EV 服务器根证书,O=WoSign CA Limited,C=CN
L=ShenZhen,ST=GuangDong,O=WoSign CT Monitor Intermediate,C=CN
Peter Bowen
Sep 2, 2016, 11:21:27 AM9/2/16
to Richard Wang, Ryan Sleevi, mozilla-dev-s...@lists.mozilla.org
On Fri, Sep 2, 2016 at 8:11 AM, Richard Wang <ric...@wosign.com> wrote:
> Yes, we posted all 2015 issued SSL from WoSign trusted root.
It seems then there is a newly exposed bug.
https://www.censys.io/certificates/e2665bb07940b5bee73145f47c99dcf5781edbe9d78f9cada8f1d702d5e340ad
shows a certificate issued by your CA that has a notBefore in March
2015. It does not appear in the CT log. However another certificate
with identical serial number and subject, but different Validity, does
appear in the log.
Are you aware of a bug where you were issuing certificates identical
except for validity period?
Thanks,
Peter
> Yes, we posted all 2015 issued SSL from WoSign trusted root.
>
> On 2 Sep 2016, at 22:55, Peter Bowen <pzb...@gmail.com> wrote:
>> Based on CT logs, I have seen certificates from the CAs below, all of
>> which have "WoSign" in the name. Have you logged all certificates
>> which are signed by these CAs and have a notBefore date of
>> 20150101000000Z or later to the WoSign CT log?
Richard,
> On 2 Sep 2016, at 22:55, Peter Bowen <pzb...@gmail.com> wrote:
>> Based on CT logs, I have seen certificates from the CAs below, all of
>> which have "WoSign" in the name. Have you logged all certificates
>> which are signed by these CAs and have a notBefore date of
>> 20150101000000Z or later to the WoSign CT log?
It seems then there is a newly exposed bug.
https://www.censys.io/certificates/e2665bb07940b5bee73145f47c99dcf5781edbe9d78f9cada8f1d702d5e340ad
shows a certificate issued by your CA that has a notBefore in March
2015. It does not appear in the CT log. However another certificate
with identical serial number and subject, but different Validity, does
appear in the log.
Are you aware of a bug where you were issuing certificates identical
except for validity period?
Thanks,
Peter
Andrew Ayer
Sep 2, 2016, 1:01:08 PM9/2/16
to mozilla-dev-s...@lists.mozilla.org
1. WoSign has already been caught backdating the notBefore date, and
2. A certificate has already been found which they didn't log to CT
despite their assertion that they had logged all certificates,
I don't think relying on the notBefore date is a viable option.
WoSign seems to have such a poor handle on their operations that I
think it would be inevitable that someone would find a certificate in
the wild with a notBefore date in the past that had not been
disclosed. What action would Mozilla take if that happened?
Regards,
Andrew
Kurt Roeckx
Sep 2, 2016, 1:27:52 PM9/2/16
to Andrew Ayer, mozilla-dev-s...@lists.mozilla.org
On Fri, Sep 02, 2016 at 10:00:28AM -0700, Andrew Ayer wrote:
> 2. A certificate has already been found which they didn't log to CT
> despite their assertion that they had logged all certificates,
Can you please point to those that weren't logged?
> 2. A certificate has already been found which they didn't log to CT
> despite their assertion that they had logged all certificates,
Kurt
Kurt Roeckx
Sep 2, 2016, 1:30:08 PM9/2/16
to Andrew Ayer, mozilla-dev-s...@lists.mozilla.org
https://www.censys.io/certificates/e2665bb07940b5bee73145f47c99dcf5781edbe9d78f9cada8f1d702d5e340ad
Kurt
Percy
Sep 2, 2016, 1:45:37 PM9/2/16
to mozilla-dev-s...@lists.mozilla.org
Some facts for Mozilla to consider. WoSign Root is never trusted by Apple https://support.apple.com/en-ca/HT205205 https://support.apple.com/en-ca/HT205204
However, all WoSign leaf certs are trusted on Apple devices because WoSign intermediate authority is signed by StartCom.
However, all WoSign leaf certs are trusted on Apple devices because WoSign intermediate authority is signed by StartCom.
Percy
Sep 2, 2016, 2:17:17 PM9/2/16
to mozilla-dev-s...@lists.mozilla.org
On Friday, September 2, 2016 at 3:07:46 AM UTC-7, Gervase Markham wrote:
> Hi Richard,
>
> On 01/09/16 04:04, Richard Wang wrote:
> > First, please treat WoSign as a global trusted CA, DON'T stamp as
> > China CA. We need a fair treatment as other worldwide CAs that I am
> > sure WoSign is not the first CA that have incident and not the
> > serious one;
>
> We are keen to treat WoSign as a global CA. It's certainly true that we
> would be having this discussion about any other global CA which had had
> such a list of incidents. However, it seems that you are advancing
> arguments - such as "we are Chinese; we can't be expected to fully
> understand standards written in English" - which ask for special
> consideration as a Chinese CA rather than a global CA. And, as others
> have pointed out in this thread, WoSign is very happy to be seen as a
> China CA for marketing purposes inside China.
WoSign in fact actively emphasis that it's a China CA and the global politics in marketing. WoSign claimed foreign CA might revoke certs to Chinese orgs due to politics and claimed that foreign CA will collect all users information. This is a typical marketing email they sent. https://pbs.twimg.com/media/CrXf7w3W8AA2zd7.jpg:large Translated below.
> Hi Richard,
>
> On 01/09/16 04:04, Richard Wang wrote:
> > First, please treat WoSign as a global trusted CA, DON'T stamp as
> > China CA. We need a fair treatment as other worldwide CAs that I am
> > sure WoSign is not the first CA that have incident and not the
> > serious one;
>
> We are keen to treat WoSign as a global CA. It's certainly true that we
> would be having this discussion about any other global CA which had had
> such a list of incidents. However, it seems that you are advancing
> arguments - such as "we are Chinese; we can't be expected to fully
> understand standards written in English" - which ask for special
> consideration as a Chinese CA rather than a global CA. And, as others
> have pointed out in this thread, WoSign is very happy to be seen as a
> China CA for marketing purposes inside China.
-------
Dear friend:
I'm *** from WoSign CA. WoSign is the first SSL cert company in China. Your website *****'s SSL cert is from Let's Encrypt, expiring at Oct, 2016. If you switch to WoSign before the expiration you can enjoy buy one year get one year free.
The risks associated with foreign CA:
1. Cert revocation
If foreign CA is influenced by politics and revoke certs for important Chinese organizations, the entire system will be paralyzed.
2. Information security risks
If the website uses foreign certs, users need to send information to foreign servers in every visit. Time of the visit, the location of the visit, IP addresses, and the browser, frequency of the visits are all collected by foreign CA. This will leak commercial secrets and sensitive data, and is a very risky!
3. Server latency
Foreign CA cannot provide 24*7 local support. Servers are overseas and affected by submarine cables, latency is 10X. If something happens to submarine cables, and cert revocation list is not accessible, important systems with foreign certs will be paralyzed. In 2012, there is a incident that submarine cables was broken.
.... (contact info stuff)
Best regards and thanks,
WoSign CA Limited.
Erwann Abalea
Sep 2, 2016, 3:15:13 PM9/2/16
to mozilla-dev-s...@lists.mozilla.org
Le vendredi 2 septembre 2016 19:45:37 UTC+2, Percy a écrit :
> Some facts for Mozilla to consider. WoSign Root is never trusted by Apple https://support.apple.com/en-ca/HT205205 https://support.apple.com/en-ca/HT205204
>
> However, all WoSign leaf certs are trusted on Apple devices because WoSign intermediate authority is signed by StartCom.
That's not WoSign's fault. It's really hard to communicate with Apple and their Root Program, it's a concern for other CAs as well.
> Some facts for Mozilla to consider. WoSign Root is never trusted by Apple https://support.apple.com/en-ca/HT205205 https://support.apple.com/en-ca/HT205204
>
> However, all WoSign leaf certs are trusted on Apple devices because WoSign intermediate authority is signed by StartCom.
Matt Palmer
Sep 2, 2016, 6:53:34 PM9/2/16
to dev-secur...@lists.mozilla.org
On Fri, Sep 02, 2016 at 09:01:47AM +0000, Richard Wang wrote:
> You mean if a Chinese, a Chinese company own a USA CA, then the USA CA become un-trustworthiness?
If the Chinese company or US CA are making legal threats to try and suppress
> You mean if a Chinese, a Chinese company own a USA CA, then the USA CA become un-trustworthiness?
disclosure of the ownership, and the Chinese company is running another CA
with some pretty serious concerns over its trustworthiness, then yes, I'd be
concerned over the trustworthiness of the US CA.
> I still think this topic is out of THE Topic - Incident.
privately", indicating that you want to avoid public discussion of these
issues of trustworthiness and fidelity.
- Matt
Matt Palmer
Sep 2, 2016, 7:25:12 PM9/2/16
to dev-secur...@lists.mozilla.org
On Fri, Sep 02, 2016 at 07:55:36AM -0700, Peter Bowen wrote:
> Do you also plan to submit these to at least one Google-operated log?
Did you mean "non-Google-operated log"? I was under the impression that we
> Do you also plan to submit these to at least one Google-operated log?
didn't want everything being stuffed into just Google logs.
- Matt
--
I really didn't foresee the Internet. But then, neither did the computer
industry. Not that that tells us very much of course -- the computer
industry didn't even foresee that the century was going to end.
-- Douglas Adams
It is loading more messages.
0 new messages