StartCom cross-signs disclosed by Certinomis

Jonathan Rudenberg

unread,

Aug 2, 2017, 6:39:24 PM8/2/17

to mozilla-dev-s...@lists.mozilla.org

Kathleen Wilson

unread,

Aug 2, 2017, 8:07:03 PM8/2/17

to mozilla-dev-s...@lists.mozilla.org

Jonathan, Thank you for bringing this to our attention.

I have filed two bugs...

1) https://bugzilla.mozilla.org/show_bug.cgi?id=1386891
Certinomis: Cross-signing of StartCom intermediate certs, and delay in reporting it in CCADB

2) https://bugzilla.mozilla.org/show_bug.cgi?id=1386894
Add "StartCom BR SSL ICA” and "StartCom EV SSL ICA” intermediate certs to OneCRL

We will look into this further, and will appreciate any further relevant information.

I will also appreciate explanation from Certinomis and StartCom.

Thanks,
Kathleen

Matt Palmer

unread,

Aug 2, 2017, 8:12:18 PM8/2/17

to dev-secur...@lists.mozilla.org

On Wed, Aug 02, 2017 at 06:38:44PM -0400, Jonathan Rudenberg via dev-security-policy wrote:
> I think the correct response is to add both intermediates to OneCRL
> immediately, especially given the historic issues with StartCom.

+1. Also a strongly worded letter of "are you f%*king kidding me?!?" to
Certinomis. Everyone even ephemerally involved in the WebPKI should know by
now that StartCom/WoSign are viewed with deep suspicion, and blithely
signing an intermediate for them is not a smart move.

- Matt

okaphone.e...@gmail.com

unread,

Aug 3, 2017, 2:23:18 AM8/3/17

to mozilla-dev-s...@lists.mozilla.org

It just means Certinomis now has to answer for the misissuance, doesn't it? I don't see the problem. They can choose to risk their own webtrust if they want to. ;-)

CU Hans

Franck Leroy

unread,

Aug 3, 2017, 3:59:35 AM8/3/17

to mozilla-dev-s...@lists.mozilla.org

Hello,

the 2 CA certificates signed by Certinomis has been retained till a full successful webtrust audit.

On end of June the audit report form PwC was available but with still some minor issues. I asked StartCom to correct them.

On July 14th the audit report and the policy were updated and published on StartCom website.

The first of August I received the agreement from my management to send the CA certificates to StartCom. So I disclose them in the CCADB, with the corresponding policy documents and audit reports before sending them to Inigo.

So StartCom was not able to use the path our Root before yesterday.

If there are some previous issued TLS certificates that does not comply with BR, then theses TLS certificates has to be revoked.

Best regards
Franck Leroy

Inigo Barreira

unread,

Aug 3, 2017, 4:44:06 AM8/3/17

to Franck Leroy, mozilla-dev-s...@lists.mozilla.org

Hi, this is my reply in the bugzilla

Hi all,

what Fanck is saying is true and we haven´t started to issue any cert using
this new path.

Regarding the info that is in this bug I´m really shocked because the
majority of them are revoked and don´t understand why have been included
here.

For those which are not revoked are due to use different curves (P-384,
P-521) that have been discussed in the mozilla m.d.s.p as well as the CAB
Forum and there´s no conclusion yet, but in any case we´re not allowing to
use them anymore. There´re curves allowed in the BRs that Mozilla does not
include.

1. The un-revoked test certificates are those pre-sign ones with uncompleted
ctlog. So they are not completed certificates.

https://crt.sh/?opt=cablint&id=134843670
https://crt.sh/?opt=cablint&id=134843674
https://crt.sh/?opt=cablint&id=134843685

https://crt.sh/?opt=cablint&id=139640371

2. Other un-revoked certificates have the same error “ ERROR: Unallowed key
usage for EC public key (Key Encipherment) ”
https://crt.sh/?opt=cablint&id=153404034
https://crt.sh/?opt=cablint&id=160150786
https://crt.sh/?opt=cablint&id=149445010
https://crt.sh/?opt=cablint&id=150133570

And what I don´t understand are those comments of "very sloppy isuance
practices" , "many non-BR compliants", "specially given the historic issues
with StartCom" and consider them very unfair. These are subjective opinions
which are very dangerous and not fair.
This is a totally new system that is not related with "the historic issues"
at all, so whatever happened in the past is not related (and we could talk a
lot of why StartCom was distrusted in the past), only the name is the same.
Some of the issues are also related what has been discussing in the CABF
related to the Unicode and punnycode in domanins, etc. and even there´s no
conclussion as the ballot failed, we decided to revoke those to avoid issues
but you include them here as non BR compliants and very sloppy issuance
practices.

Finally I´d like to understand also why has been asked to create OneCRL
entries for these subCAs.

I may think this post and some other comments in the m.d.s.p are malicious
and don´t know why.

Regards

Best regards

Iñigo Barreira
CEO
StartCom CA Limited

_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Patrick Figel

unread,

Aug 3, 2017, 7:07:08 AM8/3/17

to Inigo Barreira, Franck Leroy, mozilla-dev-s...@lists.mozilla.org

On 03/08/2017 10:47, Inigo Barreira via dev-security-policy wrote> 1.

The un-revoked test certificates are those pre-sign ones with uncompleted
> ctlog. So they are not completed certificates.

> https://crt.sh/?opt=cablint&id=134843670
> https://crt.sh/?opt=cablint&id=134843674
> https://crt.sh/?opt=cablint&id=134843685

> https://crt.sh/?opt=cablint&id=139640371

My understanding of Mozilla's policy is that misissued precerts are
considered misissuance nonetheless[1].

[1]:
https://groups.google.com/d/msg/mozilla.dev.security.policy/6pBLHJBFNts/kM3kEJKMAgAJ

Nick Lamb

unread,

Aug 3, 2017, 7:11:56 AM8/3/17

to mozilla-dev-s...@lists.mozilla.org

1. It is well established that logging pre-certs constitutes "issuance" for purposes of policy compliance. If you wouldn't issue it, don't log it. Not difficult. And this isn't new.

2. When a new path comes into existence in the Web PKI you don't need to explicitly "use" it as a CA, the Relying Parties may rely on this path for a variety of reasons out of your control. If you don't want a particular path to be used don't create it.

(An example not related to the present circumstances: Let's Encrypt has a path from their Issuing CAs to the ISRG root. Today this root is not widely trusted and so they provide subscribers with an alternative intermediate CA cert leading to Identrust. But some clients, including Firefox, will use the ISRG path anyway because they trust it.)

3. The non-disclosure of the CA:TRUE certs is unambiguously a policy violation. Adding them to OneCRL immediately seems like exactly the right response.

Inigo Barreira

unread,

Aug 3, 2017, 7:17:06 AM8/3/17

to Patrick Figel, Franck Leroy, mozilla-dev-s...@lists.mozilla.org

We´re revoking all those unrevoked certs to avoid any more problems.

Regarding the pre-certs, yes, I was aware of the discussion. As Gerv says
there´s a binding statement of "intent" ... the problem with these is that
we generated the pre-certs and logged in the CT log, where crt.sh looks or
monitor, but those weren´t finally issued, so there are not such certs.
In any case, as said, we´re revoking all of those listed and will update the
bugzilla accordingly

Best regards

Iñigo Barreira
CEO
StartCom CA Limited

-----Original Message-----
From: Patrick Figel [mailto:pat...@figel.email]
Sent: jueves, 3 de agosto de 2017 13:07
To: Inigo Barreira <in...@startcomca.com>; Franck Leroy
<fr.l...@gmail.com>; mozilla-dev-s...@lists.mozilla.org
Subject: Re: StartCom cross-signs disclosed by Certinomis

On 03/08/2017 10:47, Inigo Barreira via dev-security-policy wrote> 1.
The un-revoked test certificates are those pre-sign ones with uncompleted
> ctlog. So they are not completed certificates.

> https://crt.sh/?opt=cablint&id=134843670
> https://crt.sh/?opt=cablint&id=134843674
> https://crt.sh/?opt=cablint&id=134843685

Alex Gaynor

unread,

Aug 3, 2017, 9:10:34 AM8/3/17

to Inigo Barreira, Patrick Figel, mozilla-dev-s...@lists.mozilla.org, Franck Leroy

>From RFC6962:

The signature on the TBSCertificate indicates the certificate
authority's intent to issue a certificate. This intent is considered
binding (i.e., misissuance of the Precertificate is considered equal
to misissuance of the final certificate).

I don't think this text could be any more clear, and I'm frankly
astounded that any CA would try to argue they shouldn't be held to
account for them.

If you wouldn't issue a cert, don't issue the pre-cert. It's really that simple.

Alex

On Thu, Aug 3, 2017 at 7:20 AM, Inigo Barreira via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> We´re revoking all those unrevoked certs to avoid any more problems.
>
> Regarding the pre-certs, yes, I was aware of the discussion. As Gerv says
> there´s a binding statement of "intent" ... the problem with these is that
> we generated the pre-certs and logged in the CT log, where crt.sh looks or
> monitor, but those weren´t finally issued, so there are not such certs.
> In any case, as said, we´re revoking all of those listed and will update
> the
> bugzilla accordingly
>
> Best regards
>
> Iñigo Barreira
> CEO
> StartCom CA Limited
>
> -----Original Message-----
> From: Patrick Figel [mailto:pat...@figel.email]
> Sent: jueves, 3 de agosto de 2017 13:07
> To: Inigo Barreira <in...@startcomca.com>; Franck Leroy
> <fr.l...@gmail.com>; mozilla-dev-s...@lists.mozilla.org
> Subject: Re: StartCom cross-signs disclosed by Certinomis
>
> On 03/08/2017 10:47, Inigo Barreira via dev-security-policy wrote> 1.
> The un-revoked test certificates are those pre-sign ones with uncompleted
> > ctlog. So they are not completed certificates.

> > https://crt.sh/?opt=cablint&id=134843670
> > https://crt.sh/?opt=cablint&id=134843674
> > https://crt.sh/?opt=cablint&id=134843685

> > https://crt.sh/?opt=cablint&id=139640371
>
> My understanding of Mozilla's policy is that misissued precerts are
> considered misissuance nonetheless[1].
>
> [1]:
> https://groups.google.com/d/msg/mozilla.dev.security.
> policy/6pBLHJBFNts/kM3k
> EJKMAgAJ
>

Jonathan Rudenberg

unread,

Aug 3, 2017, 10:52:37 AM8/3/17

to Inigo Barreira, mozilla-dev-s...@lists.mozilla.org

> On Aug 3, 2017, at 04:47, Inigo Barreira via dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
>
> For those which are not revoked are due to use different curves (P-384,
> P-521) that have been discussed in the mozilla m.d.s.p as well as the CAB
> Forum and there´s no conclusion yet, but in any case we´re not allowing to
> use them anymore. There´re curves allowed in the BRs that Mozilla does not
> include.
>

> 2. Other un-revoked certificates have the same error “ ERROR: Unallowed key
> usage for EC public key (Key Encipherment) ”
> https://crt.sh/?opt=cablint&id=153404034
> https://crt.sh/?opt=cablint&id=160150786
> https://crt.sh/?opt=cablint&id=149445010
> https://crt.sh/?opt=cablint&id=150133570

Let’s break this down, as you have confused a few issues with this subset of the misissued certificates. Two certificates were issued with P-521 ECDSA keys, which is not allowed by Mozilla policy (note that P-384 keys are allowed):

- https://crt.sh/?q=87304EBF0F9391B8FFF7C8ED8D567F0340BCBAA6741972C030364DE5618C6757
- https://crt.sh/?q=962C955ABC03FC00F514EA41B2838D85826CA7CAA419A85EC186F1646AD5C9B5

Thirteen certificates (including the two P-521 certificates) were issued with the keyEncipherment bit set in the keyUsage extension (this is the message you mentioned above) which is not allowed (RFC 3279 section 2.3.5, incorporated by reference by RFC 5280 section 4.2.1.3, incorporated by reference by Baseline Requirements section 7.1.2.4).

One certificate linked above was issued without the key parameters field set, which is not allowed (RFC 3279 section 2.3.1, incorporated by reference by RFC 5280 section 4.1.2.7, incorporated by reference by Baseline Requirements section 7.1.2.4):

- https://crt.sh/?opt=cablint&q=160150786

Hopefully this clarifies any misunderstandings around the problems with these specific certificates.

Jonathan

Inigo Barreira

unread,

Aug 3, 2017, 10:56:05 AM8/3/17

to Jonathan Rudenberg, mozilla-dev-s...@lists.mozilla.org

Thanks Jonathan

Yes, I answered after just looking quickly about the main issues not focusing on the different sizes, etc. As you can see in the post, we have revoked all of them.

Best regards

Iñigo Barreira
CEO
StartCom CA Limited

-----Original Message-----
From: Jonathan Rudenberg [mailto:jona...@titanous.com]
Sent: jueves, 3 de agosto de 2017 16:52
To: Inigo Barreira <in...@startcomca.com>
Cc: mozilla-dev-s...@lists.mozilla.org
Subject: Re: StartCom cross-signs disclosed by Certinomis

Kathleen Wilson

unread,

Aug 3, 2017, 12:27:03 PM8/3/17

to mozilla-dev-s...@lists.mozilla.org

All,

I have conflicting opinions about this situation:

On the one hand, I want to see better behavior, and am inclinded to add these two intermediate certs to OneCRL, and tell StartCom and Certinomis to start over and do things right.

On the other hand, I'm not convinced yet that the issued non-BR-compliant certs are any worse than we've seen from other CAs for whom we tell them to fix the problem but do not add their relevant intermediate certs to OneCRL.

Kathleen

Jonathan Rudenberg

unread,

Aug 3, 2017, 12:49:41 PM8/3/17

to Kathleen Wilson, mozilla-dev-s...@lists.mozilla.org

Even absent the BR-violating certificates and disclosure timeline, I believe this cross-sign is problematic because it appears to circumvent the prerequisites and process described in https://bugzilla.mozilla.org/show_bug.cgi?id=1311832 for StartCom’s application for re-inclusion into the Mozilla root store. It’s not clear to me what the point of those requirements is if they can be avoided by obtaining cross-signatures from other CAs that are currently trusted by Mozilla.

As far as the misissued certificates are concerned, here are a couple points to consider:

1) Many of the certificates are improperly validated “test” certificates, a practice that is extremely problematic and indicates a lack of or circumvention of technical controls.

2) StartCom's systems are apparently new, but they have failed to correctly implement simple aspects of the certificate profile such as keyUsage bits and character encodings. These issues are trivially detected by running tools like certlint and should have been caught well before the system made it into production.

Jonathan

Kathleen Wilson

unread,

Aug 3, 2017, 4:43:19 PM8/3/17

to mozilla-dev-s...@lists.mozilla.org

On Thursday, August 3, 2017 at 9:49:41 AM UTC-7, Jonathan Rudenberg wrote:
> Even absent the BR-violating certificates and disclosure timeline, I believe this cross-sign is problematic because it appears to circumvent the prerequisites and process described in https://bugzilla.mozilla.org/show_bug.cgi?id=1311832 for StartCom’s application for re-inclusion into the Mozilla root store. It’s not clear to me what the point of those requirements is if they can be avoided by obtaining cross-signatures from other CAs that are currently trusted by Mozilla.

It is common practice for a CA to get cross-signed by a currently-included CA, so their cert chain is trusted while they are going through Mozilla's long inclusion process. This is OK, as long as the currently-included CA ensures that the subCA follows Mozilla's Root Store Policy.
See section 5.3 of
https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/

I would have a problem if Certinomis cross-signed with the old StartCom roots for which we do not trust certs issued after October 2016. In my opinion, that would be grounds for removing Certinomis' roots.

However, I think it is fine for Certinomis to cross-sign with new StartCom subCA certs, as long as Certinomis ensures that Mozilla's Root Store Policy is being followed.

> As far as the misissued certificates are concerned, here are a couple points to consider:
>
> 1) Many of the certificates are improperly validated “test” certificates, a practice that is extremely problematic and indicates a lack of or circumvention of technical controls.

Yes, this is problematic. But other CAs have also had this problem, and for the other CAs we have worked with them to ensure the practice is stopped, tools/process put in place to prevent it in the future, problematic certs revoked, etc. But this type of mis-issuance has not yet been considered grounds for adding the relevant intermediate cert to OneCRL.

> 2) StartCom's systems are apparently new, but they have failed to correctly implement simple aspects of the certificate profile such as keyUsage bits and character encodings. These issues are trivially detected by running tools like certlint and should have been caught well before the system made it into production.

This is the part that particularly concerns me. I would expect StartCom to be on their best behavior and setting up an exemplary PKI. These types of mistakes should not be happening. This is the main reason I am considering adding the two intermediate certs to OneCRL and telling the CA to start over, and make sure they do it right this time.

Thanks,
Kathleen

Alex Gaynor

unread,

Aug 3, 2017, 4:46:26 PM8/3/17

to Kathleen Wilson, mozilla-dev-s...@lists.mozilla.org

I think it's reasonable to consider mistakes in StartCom's new PKI of this
nature to be a part of "continuing pattern of behavior" from their previous
PKI, and not something which should be considered in isolation. In that
context, I'm not sure comparisons with other CAs which were "first time
offenders", and which were given an opportunity to clean up, make sense.

Alex

On Thu, Aug 3, 2017 at 4:43 PM, Kathleen Wilson via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On Thursday, August 3, 2017 at 9:49:41 AM UTC-7, Jonathan Rudenberg wrote:

> > Even absent the BR-violating certificates and disclosure timeline, I
> believe this cross-sign is problematic because it appears to circumvent the
> prerequisites and process described in https://bugzilla.mozilla.org/
> show_bug.cgi?id=1311832 for StartCom’s application for re-inclusion into
> the Mozilla root store. It’s not clear to me what the point of those
> requirements is if they can be avoided by obtaining cross-signatures from
> other CAs that are currently trusted by Mozilla.
>

> It is common practice for a CA to get cross-signed by a currently-included
> CA, so their cert chain is trusted while they are going through Mozilla's
> long inclusion process. This is OK, as long as the currently-included CA
> ensures that the subCA follows Mozilla's Root Store Policy.
> See section 5.3 of
> https://www.mozilla.org/en-US/about/governance/policies/
> security-group/certs/policy/
>
> I would have a problem if Certinomis cross-signed with the old StartCom
> roots for which we do not trust certs issued after October 2016. In my
> opinion, that would be grounds for removing Certinomis' roots.
>
> However, I think it is fine for Certinomis to cross-sign with new StartCom
> subCA certs, as long as Certinomis ensures that Mozilla's Root Store Policy
> is being followed.
>
>

> > As far as the misissued certificates are concerned, here are a couple
> points to consider:
> >
> > 1) Many of the certificates are improperly validated “test”
> certificates, a practice that is extremely problematic and indicates a lack
> of or circumvention of technical controls.
>
>

> Yes, this is problematic. But other CAs have also had this problem, and
> for the other CAs we have worked with them to ensure the practice is
> stopped, tools/process put in place to prevent it in the future,
> problematic certs revoked, etc. But this type of mis-issuance has not yet
> been considered grounds for adding the relevant intermediate cert to OneCRL.
>
>

> > 2) StartCom's systems are apparently new, but they have failed to
> correctly implement simple aspects of the certificate profile such as
> keyUsage bits and character encodings. These issues are trivially detected
> by running tools like certlint and should have been caught well before the
> system made it into production.
>
>

> This is the part that particularly concerns me. I would expect StartCom to
> be on their best behavior and setting up an exemplary PKI. These types of
> mistakes should not be happening. This is the main reason I am considering
> adding the two intermediate certs to OneCRL and telling the CA to start
> over, and make sure they do it right this time.
>
> Thanks,
> Kathleen

Kurt Roeckx

unread,

Aug 3, 2017, 6:09:25 PM8/3/17

to Kathleen Wilson, mozilla-dev-s...@lists.mozilla.org

On Thu, Aug 03, 2017 at 01:43:08PM -0700, Kathleen Wilson via dev-security-policy wrote:
> On Thursday, August 3, 2017 at 9:49:41 AM UTC-7, Jonathan Rudenberg wrote:

> > Even absent the BR-violating certificates and disclosure timeline, I believe this cross-sign is problematic because it appears to circumvent the prerequisites and process described in https://bugzilla.mozilla.org/show_bug.cgi?id=1311832 for StartCom’s application for re-inclusion into the Mozilla root store. It’s not clear to me what the point of those requirements is if they can be avoided by obtaining cross-signatures from other CAs that are currently trusted by Mozilla.
>

> It is common practice for a CA to get cross-signed by a currently-included CA, so their cert chain is trusted while they are going through Mozilla's long inclusion process. This is OK, as long as the currently-included CA ensures that the subCA follows Mozilla's Root Store Policy.
> See section 5.3 of
> https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/

I would really like to see that they have at least opened a bug to
request the inclusion of that CA before it's cross-signed. It should
have already all the requirements that Mozilla has for including the
root CA certificate before it's cross signed.

I would prefer that it's even included in the Mozilla root store
before it's cross signed, or that it's been added to one of the
other root stores.

Kurt

Kathleen Wilson

unread,

Aug 3, 2017, 7:02:16 PM8/3/17

to mozilla-dev-s...@lists.mozilla.org

On Thursday, August 3, 2017 at 3:09:25 PM UTC-7, Kurt Roeckx wrote:
> I would really like to see that they have at least opened a bug to
> request the inclusion of that CA before it's cross-signed.

Here's StartCom's current root inclusion request:
https://bugzilla.mozilla.org/show_bug.cgi?id=1381406

> It should
> have already all the requirements that Mozilla has for including the
> root CA certificate before it's cross signed.

Correct. That should be true for all subCAs that are cross-signed by currently-included CAs.

>
> I would prefer that it's even included in the Mozilla root store
> before it's cross signed,

That might not be fair, given how long Mozilla's root inclusion process takes, and that we don't require this of other CAs who are new to our program.

Kathleen

Ryan Sleevi

unread,

Aug 3, 2017, 7:34:27 PM8/3/17

to mozilla-dev-s...@lists.mozilla.org

Kathleen,

Doesn't this depend on your perspective of whether or not "new CA" refers to the key or the organization?

There's no doubt StartCom is not a "new CA" - it is a CA that Mozilla entrusted with the ability to issue certificates, and the organization - and its management - egregiously and repeatedly violated that trust.

The decision to remove - and to instill new requirements - was to ensure that the organization meaningfully underwent change to ensure that it would not do the same if further keys operated by it were included as trusted by Mozilla products - whether directly or indirectly.

Further, the process of restricting it to going through the Mozilla process ensures that there is a sufficient level of community review of those remediation steps, so that the Mozilla community can be assured that StartCom, the organization, is both competent enough and trustworthy enough to be entrusted with keys to the Internet.

In this light, both the past actions of StartCom the organization AND the many technical failures are relevant in that evaluation. Cross-signing directly undermines that review process, and in doing so, undermines both trust in StartCom the organization and in the Mozilla process, if a removed CA need only to pay for a cross-sign and can thus bypass every remediation step required of them, as StartCom is effectively doing.

My understanding had been that when a remediation is imposed on an organization, due to failing to follow policies, then those remediation steps must be taken, and subsequently reviewed by the community, before any direct (root key inclusion) or indirect (cross-signing) restoration of trust in the Mozilla Root Store. Without that guarantee, any corrective actions are hollow, and with it, trust in the whole system itself is lost.

I do hope you can clarify whether remediations apply to keys operated by organizations, or whether they apply to the organization themselves. If they apply to the organization, one would naturally expect they apply to root inclusion or cross-signs, and the organization is no longer "treated like a new CA," because they are no longer a new CA - they are an existing one.

It is also worth noting that in the past, Mozilla directed other CAs that cross-signing of their (new) roots would be expressly forbidden until the corrective actions were taken and publicly reviewed. For example, allowing CNNIC to be cross-signed prior to remediation would have defeated the entire purpose of removal.

In this larger light, it would also seem that StartCom, having misissued a number of certificates already under their new hierarchy, which present a risk to Mozilla users (revocation is neither an excuse nor a mitigation for misissuance), should be required to take corrective steps and generate a new hierarchy that is not, out of the gate, presenting risk to the overall community due to its past misissuances. We can and should expect more of new keys being included, because the compatibility risk of expecting adherence to the Root Policy is non-existent.

Kathleen Wilson

unread,

Aug 3, 2017, 8:27:13 PM8/3/17

to mozilla-dev-s...@lists.mozilla.org

On Thursday, August 3, 2017 at 4:34:27 PM UTC-7, Ryan Sleevi wrote:
> I do hope you can clarify whether remediations apply to keys operated by organizations, or whether they apply to the organization themselves.

https://bugzilla.mozilla.org/show_bug.cgi?id=1311832
says: "StartCom may apply for inclusion of new (replacement) root certificates[1] following Mozilla's normal root inclusion/change process[2] (minus waiting in the queue for the discussion), after they have completed all of the following action items, and shown that WoSign has no control (people or code) over StartCom."

So, the remediations do apply to the organization.

> If they apply to the organization, one would naturally expect they apply to root inclusion or cross-signs, and the organization is no longer "treated like a new CA," because they are no longer a new CA - they are an existing one.
>

OK. Clearly I hadn't thought of it this way.

> It is also worth noting that in the past, Mozilla directed other CAs that cross-signing of their (new) roots would be expressly forbidden until the corrective actions were taken and publicly reviewed. For example, allowing CNNIC to be cross-signed prior to remediation would have defeated the entire purpose of removal.

In bug #1311832 there is a note about cross-signing:
"[1] The new (replacement) root certificates may be cross-signed by the Affected Roots. However, the Affected Roots may *not* be cross-signed by the new (replacement) root certificates, because that would bring the concerns about the Affected Roots into the scope of the new roots. Due to the way we are implementing the distrust, the new root certificates must have a Subject Distinguished Name that does not overlap with the Subject Distinguished Names listed above."

I don't see anything expressly forbidding cross-signing of new roots, but perhaps that was an oversight.

>
> In this larger light, it would also seem that StartCom, having misissued a number of certificates already under their new hierarchy, which present a risk to Mozilla users (revocation is neither an excuse nor a mitigation for misissuance), should be required to take corrective steps and generate a new hierarchy that is not, out of the gate, presenting risk to the overall community due to its past misissuances. We can and should expect more of new keys being included, because the compatibility risk of expecting adherence to the Root Policy is non-existent.

To me, this is very convincing that we should add the two StartCom intermediate certs to OneCRL.

Along this line of discussion, I have not felt comfortable with StartCom's current root inclusion request (bug #1381406), because Hanno raised a concern about the private key used by the new root is also used by two intermediate certificates, one of them revoked. This doesn't see like good practice to me, and I'm not sure that Inigo's response is sufficient. So, I'm also wondering if I should close Bug #1381406 and request StartCom to start completely over with their new CA Hierarchy, and get it right, before creating their next root inclusion request.

I will appreciate thoughtful and constructive feedback on this as well.

Thanks,
Kathleen

Matt Palmer

unread,

Aug 3, 2017, 9:13:34 PM8/3/17

to dev-secur...@lists.mozilla.org

On Thu, Aug 03, 2017 at 05:27:03PM -0700, Kathleen Wilson via dev-security-policy wrote:
> Along this line of discussion, I have not felt comfortable with StartCom's
> current root inclusion request (bug #1381406), because Hanno raised a
> concern about the private key used by the new root is also used by two
> intermediate certificates, one of them revoked. This doesn't see like
> good practice to me, and I'm not sure that Inigo's response is sufficient.
> So, I'm also wondering if I should close Bug #1381406 and request StartCom
> to start completely over with their new CA Hierarchy, and get it right,
> before creating their next root inclusion request.

I think it makes the most sense to ask StartCom to start "clean", as well.
Hierarchies that are to be globally trusted should not be used as
"experimental playgrounds", even if those experiments are revoked, because
as we all know revocation is not 100% effective.

Further, if Mozilla allows the existing hierarchy to be admitted, there's no
demonstration that StartCom is actually *capable* of doing things correctly.
A fully compliant hierarchy would be a demonstration that they *do* know
how, and got it wrong last time, as opposed to being incapable. Once
there's a fully-compliant setup in place, there's then no reason to use a
broken setup, given that it isn't currently used by anyone, anyway.

- Matt

Matt Palmer

unread,

Aug 3, 2017, 9:16:45 PM8/3/17

to dev-secur...@lists.mozilla.org

On Thu, Aug 03, 2017 at 01:43:08PM -0700, Kathleen Wilson via dev-security-policy wrote:
> However, I think it is fine for Certinomis to cross-sign with new StartCom
> subCA certs, as long as Certinomis ensures that Mozilla's Root Store
> Policy is being followed.

... which they didn't. So there's that.

> > 1) Many of the certificates are improperly validated “test”
> > certificates, a practice that is extremely problematic and indicates a
> > lack of or circumvention of technical controls.
>

> Yes, this is problematic. But other CAs have also had this problem, and
> for the other CAs we have worked with them to ensure the practice is
> stopped, tools/process put in place to prevent it in the future,
> problematic certs revoked, etc. But this type of mis-issuance has not yet
> been considered grounds for adding the relevant intermediate cert to
> OneCRL.

Those are CAs which have been operational for some time though, and which
have certificates "in the wild" which would be distrusted, correct? That's
a somewhat different case to this one, where nothing important is hanging
off the intermediate, so distrusting it doesn't hurt relying parties, only
the CA -- which is fine, because it's the CA's fault the distrust is
required, so it's entirely self-inflicted pain.

- Matt

Matt Palmer

unread,

Aug 3, 2017, 9:19:02 PM8/3/17

to dev-secur...@lists.mozilla.org

On Thu, Aug 03, 2017 at 11:20:19AM +0000, Inigo Barreira via dev-security-policy wrote:
> We´re revoking all those unrevoked certs to avoid any more problems.

Revoking problematic certificates doesn't avoid any problems. The problems
have already been created.

> Regarding the pre-certs, yes, I was aware of the discussion. As Gerv says
> there´s a binding statement of "intent" ... the problem with these is that
> we generated the pre-certs and logged in the CT log, where crt.sh looks or
> monitor, but those weren´t finally issued, so there are not such certs.

I don't understand how failing to issue a certificate corresponding to a
logged pre-certificate constitutes a "problem". You logged the pre-cert.
It was broken. There's the problem.

- Matt

Matt Palmer

unread,

Aug 3, 2017, 9:24:36 PM8/3/17

to dev-secur...@lists.mozilla.org

On Thu, Aug 03, 2017 at 08:47:17AM +0000, Inigo Barreira via dev-security-policy wrote:
> And what I don愒 understand are those comments of "very sloppy isuance

> practices" , "many non-BR compliants", "specially given the historic issues
> with StartCom" and consider them very unfair. These are subjective opinions
> which are very dangerous and not fair.

Fairness is not, as far as I'm aware, a criteria for inclusion in the
Mozilla root store.

> This is a totally new system that is not related with "the historic issues"
> at all, so whatever happened in the past is not related (and we could talk a
> lot of why StartCom was distrusted in the past), only the name is the same.

Systems are not limited to software and hardware. They also include the
people, and at least some of the people in "the system" are the same.
Further, if "the system" *were* completely new and improved, why is it still
producing problematic certificates and generally looking, from the outside,
like a complete and utter shambles?

> Finally I悲 like to understand also why has been asked to create OneCRL
> entries for these subCAs.

Because the intermediates chain to a CA certificate which represents a
demonstrably broken and untrustworthy system, and distrusting the
intermediates has no visible impact on the relying parties which the Mozilla
trust store represents.

- Matt

J.C. Jones

unread,

Aug 3, 2017, 9:40:36 PM8/3/17

to dev-secur...@lists.mozilla.org

On 8/3/17 5:27 PM, Kathleen Wilson via dev-security-policy wrote:
> On Thursday, August 3, 2017 at 4:34:27 PM UTC-7, Ryan Sleevi wrote:
> In bug #1311832 there is a note about cross-signing:
> "[1] The new (replacement) root certificates may be cross-signed by the Affected Roots. However, the Affected Roots may *not* be cross-signed by the new (replacement) root certificates, because that would bring the concerns about the Affected Roots into the scope of the new roots. Due to the way we are implementing the distrust, the new root certificates must have a Subject Distinguished Name that does not overlap with the Subject Distinguished Names listed above."
>
> I don't see anything expressly forbidding cross-signing of new roots, but perhaps that was an oversight.

The issue in my eyes isn't necessarily the cross-signature (though the
delay in disclosure is a problem), it's using a publicly-trusted CA to
fine-tune the issuance process.

It's perfectly understandable to sign invalid certificates that violate
aspects of root policies and the BRs while proceeding with your test
plan. I've certainly done that myself. It's not acceptable to me that
the test plan used a CA that either already was or subsequently became
publicly-trusted.

This is a professional PKI operation, being overseen by industry
veterans. If something as concrete as the issuance process had such a
glaring quality assurance methodology failure, why should anyone believe
that something much harder -- subscriber validation -- is going to be
done correctly?

I agree that Mozilla should add these intermediates to OneCRL.

> Along this line of discussion, I have not felt comfortable with StartCom's current root inclusion request (bug #1381406), because Hanno raised a concern about the private key used by the new root is also used by two intermediate certificates, one of them revoked. This doesn't see like good practice to me, and I'm not sure that Inigo's response is sufficient. So, I'm also wondering if I should close Bug #1381406 and request StartCom to start completely over with their new CA Hierarchy, and get it right, before creating their next root inclusion request.

I agree, it's not good practice, even if it's not prohibited. Since this
brings it up, we should consider prohibiting it going forward. Keys, and
space on HSMs for keys, are certainly not free, but they aren't so
expensive that the Web PKI should suffer the costs of ambiguous mappings
of keys to CAs.

CRLs and OneCRL revoke trust based on issuer/serial combinations. If
this key were compromised in some way, we would need to have knowledge
of, and revoke, all combinations. We've discussed before needing a way
through OneCRL (or OneCRL 2.0) to revoke based on public key hash; this
is part of the reason why.

J.C.

userwithuid

unread,

Aug 4, 2017, 8:16:05 AM8/4/17

to mozilla-dev-s...@lists.mozilla.org

On Friday, August 4, 2017 at 12:27:13 AM UTC, Kathleen Wilson wrote:
> Along this line of discussion, I have not felt comfortable with StartCom's current root inclusion request (bug #1381406), because Hanno raised a concern about the private key used by the new root is also used by two intermediate certificates, one of them revoked. This doesn't see like good practice to me, and I'm not sure that Inigo's response is sufficient.

OK, maybe I'm just not getting it but what exactly *is* the issue that was brought up in that bug?

I would think that this is, as the StartCom response seems to confirm, just the way you cross-sign new roots with existing ones? Old roots signing new roots is done by other CAs as well and I'm not aware of any talk or rule that this is discouraged or forbidden?

My understanding is this (please correct me if I'm wrong):

Hanno's crtsh link [0] just has a spkisha256 parameter. The list contains the root's self-sign and two instances where the root was signed by an old StartCom root.

I get similar output with the spkisha256 of other trusted roots, i.e.:

Amazon [1]: New roots signed with old Starfield Services Root that they bought
Commodo [2]: Newer roots signed with their old AddTrust Root
GlobalSign [3]: R3 root signed with their R1 root

How, if not like this, can roots be cross-signed at all, technically? How are the above 3 examples different from the way StartCom did it?

(If they aren't, I think it's important that the record is set straight and this specific thing doesn't appear in the real or mental list of things that new StartCom did wrong... If they are, disregard of course. :-D ).

[0] https://crt.sh/?spkisha256=d3b8136c20918725e848204735755a4fcce203d4c2eddcaa4013763b5a23d81f
[1] https://crt.sh/?spkisha256=fbe3018031f9586bcbf41727e417b7d1c45c2f47f93be372a17b96b50757d5a2
[2] https://crt.sh/?spkisha256=82b5f84daf47a59c7ab521e4982aefa40a53406a3aec26039efa6b2e0e7244c1
[3] https://crt.sh/?spkisha256=706bb1017c855c59169bad5c1781cf597f12d2cad2f63d1a4aa37493800ffb80

Inigo Barreira

unread,

Aug 4, 2017, 1:09:13 PM8/4/17

to Kathleen Wilson, mozilla-dev-s...@lists.mozilla.org

>
> In this larger light, it would also seem that StartCom, having misissued a
number of certificates already under their new hierarchy, which present a
risk to Mozilla users (revocation is neither an excuse nor a mitigation for
misissuance), should be required to take corrective steps and generate a new
hierarchy that is not, out of the gate, presenting risk to the overall
community due to its past misissuances. We can and should expect more of new
keys being included, because the compatibility risk of expecting adherence
to the Root Policy is non-existent.

To me, this is very convincing that we should add the two StartCom
intermediate certs to OneCRL.

Along this line of discussion, I have not felt comfortable with StartCom's
current root inclusion request (bug #1381406), because Hanno raised a
concern about the private key used by the new root is also used by two
intermediate certificates, one of them revoked. This doesn't see like good

practice to me, and I'm not sure that Inigo's response is sufficient. So,
I'm also wondering if I should close Bug #1381406 and request StartCom to
start completely over with their new CA Hierarchy, and get it right, before
creating their next root inclusion request.

I will appreciate thoughtful and constructive feedback on this as well.

Hi, I´ll try to clarify some of the comments here

1.- It´s true that we have miss-issued a very few number of certificates
under the new hierarchy as has been posted here and even all of them are
revoked, maybe it´s not enough according to some of the comments, which in
other cases, as also have been expressed here, was enough. But in any case,
for those mis-issued certificates, will try to explain:

a.- test certificates: those were mississued certificates issued directly
from the EJBCA system by the CA administrator for testing the CT log. Those
certificates were valid only some minutes, they were issued, the CT tested,
and then revoked. Don´t think they made any danger to the Mozilla community.
These were also reflected in the webtrust audit. And after the incidence we
put the countermeasures needed, not allowing to issue certificates directly.
This was indicated in the bugzilla, in a detailed document. Explained what
happened and why can´t happen any more. After that, none replied so assumed
that the explanation was enough.

b.- pre-certificates: there were 4 pre-certificates that were logged in the
CTs that finally weren´t issued. I still think it could be a
misinterpretation of the word intent and the binding according to the RFC
but as some posted here, it´s very clear and can´t be a misinterpreation, so
they were revoked inmediately when I was told it. Again, don´t think a
pre-certificate could be problematic for mozilla users, but it´s my opinion.

c.- mis-issuances related to the use of curves that are allowed by the BRs
but not in Mozilla. There´s been a discussion about it in both forums (CABF
and m.d.s.p),
which has not arrived any conclusion yet (seems that Mozilla is thinking on
allowing the same like the BRs if i´m not wrong). I asked personally Ryan in
the last CABF F2F meeting about it and he gave me clear instructions and
since then we are not allowing those curves. The countermeasure was applied
into production on June 21st. All certs with these curves were revoked. Also
in these ones there were some other errors related to some bit sets included
in the key usages, which according to 7.1.2.4 for which the CA shall not
issue unless is aware of a reason. The crt.sh tool can´t know if there´s a
reason as it only checks technical stuff. So, don´t see any issue with the
BRs.

d.- one mis-issuance certificate with the country code wrong, used the old
Zaire CC instead of the new one for the Democratic Republic of Congo. Well,
for this case we didn´t have our country DB updated, we did it yesterday and
also cross-checked if there were some others and realized that we had some
old ones like the french and dutch antilles and missing some others like
jersey and guernsey. I don´t think this is a big issue. The certificate was
revoked timely.

e.- misissuances related to Unicode. There were some certs with DNS not
valid due to the use of their own language, cyrilic, chinese, etc. There´s
been a discussion about it in the CABF, also a ballot 202 which has failed,
etc. We revoked all the certs involved and updated our system accordingly
adopting punnycode for all of them. Frankly don´t know if that´s the best
approach.

2.- Usage of the same key. Firstly, this is not prohibited in the BRs, it´s
one of the exceptions included as mentioned in the post. Maybe it´s unusual
or not desiderable, but I think we didn´t do anything wrong.
Our intention was to be the more accurate possible, we were allowed to
cross-sign the new ones with the old ones and to avoid differences we used
the same key for these cross-signatures for obtaining a certificate the most
similar to the new one. So initially, with that key we created the new root,
G3, for 25 years. And later on, when cross-signed with the old ones, we
generated another certificate, this is for 5 years, on april the 10th. We
realized that this certificate was not in UTF8, so decided to revoke the
cert and generate a new one coded correctly on april 11th. I don´t know what
else I can add to this. After posting this in the bugzilla, none replied so
again assumed the explanation was enough. If additional details are needed,
just ask me what you want me to provide.

Regarding the cross-signing with Certinomis, I didn´t know we couldn´t do it
as it´s been done for years for many CAs that want to start in the business.
This has been a very common practice performed by the new CAs that have come
to the market recently. When were distrusted, apart of be allowed to
cross-sign with the old roots, also were suggested different options to come
back to the business meanwhile the procedure for the inclusion is running,
and this was one, so I don´t understand the problem this has caused. As
Franck has explained they were waiting for our webtrust audit, which at
least takes 2 months according to WT, so don´t know how this could be
accomplished. Maybe a point in time WT audit was enough? And would be it
enough for others? I don´t know but I agree with Franck with the conditions
they required to us.

In summary, it´s true that we have mis-issued some certificates, I´m not
trying to hide anything, and just explaining what happened and what we have
done to remediate the issue. Some of them were valid certificates according
to the BRs but we decided to revoke them to avoid any issues, and I know
this is not enough, or at least not enough for Startcom, but it´s the only
thing we can do once they are issued. The majority of them were revoked
timely and for some others we were waiting for the outcome of the different
discussions. According to this list we´re talking about 46 certificates, and
again, not excusing for the issue, this is a long run as Kathleen said. I
know we don´t have the same credit than other CAs but it´s a new system with
new people and just ask to let us start the race.

Let me know if you have further questions or require additional information.

okaphone.e...@gmail.com

unread,

Aug 4, 2017, 1:13:22 PM8/4/17

to mozilla-dev-s...@lists.mozilla.org

On Friday, 4 August 2017 03:16:45 UTC+2, Matt Palmer wrote:
> On Thu, Aug 03, 2017 at 01:43:08PM -0700, Kathleen Wilson via dev-security-policy wrote:
> > However, I think it is fine for Certinomis to cross-sign with new StartCom
> > subCA certs, as long as Certinomis ensures that Mozilla's Root Store
> > Policy is being followed.
>
> ... which they didn't. So there's that.

Exactly.

I don't understand why this discussion seems to be about StartCom. Until they re-apply for the root program they have no direct obligation to conform to anything anymore. They may have to answer to Certinomis, depending on what was agreed with respect to the cross-signing. But that is really only relevant to Certinomis and StartCom themselves. Certinomis however, does have a root in Mozilla's root program and as such has to answer for any misissuance chaining up to their root certificate(s).

In my opinion it would make more sense for Certinomis to decide that they'd better revoke their cross-signings than for Mozzilla to add them to OneCRL.

Or am I missing something here?

CU Hans

mapno...@gmail.com

unread,

Aug 7, 2017, 2:23:28 AM8/7/17

to mozilla-dev-s...@lists.mozilla.org

+1 Support revocation!

Franck Leroy

unread,

Aug 7, 2017, 5:21:46 AM8/7/17

to mozilla-dev-s...@lists.mozilla.org

Hello
I see many reactions that are not in line with the reality because you don’t have all the history on the subject.
I’ll try to summarize.
Approximately one year ago Inigo was CTO of Izenpe (CA of the Basque Country) and he left this company in order to join StartCom.
Not long after he arrives at StartCom (days? Weeks? I don’t know) we discover the deal that has been made by the previous CEO (Eddy Nigg) with the Chinese’s guys and the relations with WoSign.
Inigo could have resign in regard to the trap the hiring was, but he decided to face, and to setup the remediation plan defined by Mozilla community.
As said Jon Snow in S07E01: “I will not punish a son for this father’s sins”.
So instead of denigrate Inigo’s work, as a community we should encourage and support him.
Setting up a new company, with a new datacentre, new pki software, NO client, NO revenue, with Chinese employees far away speaking not fluent English and with the pressure of the market, it is definitely not an easy task! Personally I would not have tried this, so bravo for Inigo’s bravery…
One of the major thing to solve in addition of the remediation plan was to be back in the business as soon as possible, because without any incomes a company cannot survive.
So Inigo contacted me to know if Certinomis could help him to be back in the business. As you can imagine we did not said yes immediately.
But Inigo is not an anonymous guy coming from a strange area of Spain. He has been for many years an active CABForum member. He is also an active expert at ESTI, where he was editing the CA policy for web authentication certificates. Inigo is known for his expertise, trustworthiness, honesty and not the least his sympathy. So when he said that he will build a new StartCom and engage the remediation plan, we could trust him.
We are a community, not only in order to do the “police” but also to support each other’s. So my answer was “yes, we will see how we can help you in respect with community expectations”.
There was different possibilities to be back in the business while waiting for a brand new CA root included in the browsers; reselling Certinomis certificates, becoming a Certinomis RA, creation a StartCom subCA under our root using our technology or a cross-signing operation on new StartCom sub certificates.
The cross signing was not the very first option we studied because it requires that the new StartCom infrastructure be ready. But the other options were not so trivial, in a technical point of view but also in the context of restoring StartCom trust.
Then in November 2016 I contacted Kathleen and Gerv to know if there was some stoppers to work with Inigo to help StartCom to be back in the business.
There was no opposition as long as we follow the requirements of the remediation plan. Gerv also answered that our plan was good to him.
So with Inigo we studied the different possibilities and finally, taking into account cost and delay, the more efficient solution was cross signing new sub certificates from a StartCom full new pki with CT log and get this audited.
On April 2017 the new StartCom datacentre with EJBCA pki software were ready, and StartCom was able to create a new root and some sub CAs.
Then Inigo work on the CT log activation in the EJBCA software, not an easy task, and the bugs to solve. He also engaged the webtrust audit with PwC.
On April it was also the mouth when Certinomis had planned to use its offline root in order to sign the authority revocation list (ARL, the root CRL). So there was an opportunity to save money by also cross signing StartCom CAs during this operation. But it was strictly clear with Inigo that we will not give him the certificates as long as the remediation plan is not completed.
By the end of June the audit was completed and so Inigo send me the report on July 3rd.
I sent it to Kathleen to be sure that the auditor company was an acceptable one and that the editor’s opinion was in line with the Mozilla policy. Kathleen pointed out some minor issues, so I asked Inigo to correct them.
By mid July Inigo had corrected the remaining minor issues, he published the updated policies and audit assessment report on StartCom web site (July 14/07) and updated the remediation bug (1311832) stating that all remediation steps where achieved (17/07).
Back from vacation on August first, I read Inigo’s emails, checked the StartCom policies, audit report, remediation bug progress, and it appears that every steps was done. So I asked a confirmation to my management to let StartCom having the cross signing certificates.
The day after I received the go from my management, so I filled the CCADB with all the corresponding information: policies, practices, audit dates, audit report, certificates fingerprints… And then sent to Inigo the cross signed certificates.

It appeared that some people think that there is a policy violation about the delay for CCADB disclosure.
Maybe I misunderstood the policy requirements on the subject. I really took care that no certificates could use the path to our root (by holding the certs in my safe) until all remediation steps were met. So I’m very sorry about this misunderstanding and I apologize to you.
But in my opinion, asking for a revocation without a full understanding of the situation, is not a balanced and a fair answer.

If you have any other question, I’ll do my best to answer (sorry for the basic English but I’m not English native nor English fluent).

Best regards
Franck Leroy

Itzhak Daniel

unread,

Aug 7, 2017, 2:31:24 PM8/7/17

to mozilla-dev-s...@lists.mozilla.org

Trust is something you *gain*.

I want to believe the internet has come a long way from PGP signing parties.

Matthew Hardeman

unread,

Aug 7, 2017, 2:59:05 PM8/7/17

to mozilla-dev-s...@lists.mozilla.org

To play the devil's advocate...

If everything is as Mr. Leroy of Certinomis points out, I don't see the problem with the cross-sign.

In that version of events, the vast majority of the issues in the new PKI (test certs, etc) had already been revoked and measures put in place to prevent that sort of issuance prior to Startcom being provided the cross-sign certificates.

They've committed to logging everything in CT and I can not recall any suggestion that any issuances have occurred which evaded CT.

At this point, why not let them sink or swim? Allow the cross-signs to stand. If Inigo has prior CA management experience and is running the technical picture at Startcom now, why not allow them to proceed under this new PKI infrastructure with past issues set aside and take a serious stance to any issues going forward.

As far as I know, the current manager of Startcom has not been previously accused of deception or bad action. Far more than has been problematic in this early testing phase of their new PKI has been forgiven by the root programs before.

Is it not possible that they're getting increased animus just for being called Startcom? I say "being called" because they have clearly undertaken a great deal of work to bring up an entirely new PKI infrastructure and have new and experienced management, according to Mr. Leroy's assertions.

Nothing disastrous or intentionally dishonest has been done in their new PKI. Why not grant them a gentleman's chance to proceed and address any further issues with great scrutiny?

Jakob Bohm

unread,

Aug 7, 2017, 4:03:27 PM8/7/17

to mozilla-dev-s...@lists.mozilla.org

As an "outside" observer, I would say that it would have been better if
the following additional things had been done:

1. At StartCom: The various tests should have been done with a dedicated
"test hierarchy" running on the production hardware but not chaining
to the intended production roots. This could also become a long term
test hierarchy for e.g. new software versions.

2. At StartCom: Remember to revoke certificates that get cancelled
between CT logging and actual issuance. I don't know if EJBCA can do
this without actually issuing the bad certificate, but hopefully there
is a way.

3. At Certinomis: Be more public up front about the "sealed" signing of
the cross certificate. Also, I am not sure how often your ARL signing
ceremonies happen, but if its every 3 months, the signing could have
happened in July instead.

4. At Certinomis: Maybe the cross-cert should have been post-dated to
a date when StartCom expected to be ready (as communicated privately).

5. At StartCom, Mozilla and Google: Should have been more clear if the
remediation plan did or did not require StartCom to seek
Mozilla/Google prior approval before getting cross-signed (if such
cross-signing were to occur before Mozilla approval of the new roots).

6. At StartCom: Be more clear if any of the "Chinese" staff is working
at, under or otherwise near WoSign and/or Richard Wang.

7. At Quihoo: Actually get rid of Richard Wang, not just change his
title from CEO to COO.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

Itzhak Daniel

unread,

Aug 7, 2017, 5:36:10 PM8/7/17

to mozilla-dev-s...@lists.mozilla.org

On Monday, August 7, 2017 at 11:03:27 PM UTC+3, Jakob Bohm wrote:
> 7. At Quihoo: Actually get rid of Richard Wang, not just change his
> title from CEO to COO.

I didn't map the new hierarchy of the "Spanish" StartCom CA ("StartCom CA Spain Sociedad Limitada"), having trouble registering to the Spanish company house and pull documents (I pulled from 3rd party, but they're garbage [1] [2]). I did mange to see that Mr. Barreira is the Directory but nothing on the share holders or parent company.

I took a quick look at StartCom UK (as the information there is free) and noticed Mr. Wang became a director again [3]... I wonder who is "StartCom CA Spain Sociedad Limitada" parent/share holder, maybe a disclosure?

Links:
1. https://www.letsphish.org/files/StartCom-CA-SPA-Appointment.pdf
2. https://www.letsphish.org/files/StartCom-CA-SPA-Profile.pdf
3. https://beta.companieshouse.gov.uk/company/09744347

Message has been deleted

Richard Wang

unread,

Aug 7, 2017, 11:07:15 PM8/7/17

to mozilla-dev-s...@lists.mozilla.org

For adding Richard Wang back to StartCom UK director is for the completion separation, this is a temporally adding as director for signing bank document to change the bank signer person from Richard Wang to New CEO Inigo. It will be removed soon once the bank signer change is done.

Mr. Jon Luk (Alliotts) is the accountant of StartCom UK that he processed this change, anyone can contact him to verify this change intention.

For StartCom, Richard Wang is 100% no any relationship after the separation, Qihoo 360 100% get rid of Richard Wang in StartCom case.

BTW, StartCom UK is the 100% shareholder of StartCom Spain.

Best Regards,

Richard

-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+richard=wosig...@lists.mozilla.org] On Behalf Of Itzhak Daniel via dev-security-policy
Sent: Tuesday, August 8, 2017 5:36 AM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: StartCom cross-signs disclosed by Certinomis

On Monday, August 7, 2017 at 11:03:27 PM UTC+3, Jakob Bohm wrote:

> 7. At Quihoo: Actually get rid of Richard Wang, not just change his
> title from CEO to COO.

I didn't map the new hierarchy of the "Spanish" StartCom CA ("StartCom CA Spain Sociedad Limitada"), having trouble registering to the Spanish company house and pull documents (I pulled from 3rd party, but they're garbage [1] [2]). I did mange to see that Mr. Barreira is the Directory but nothing on the share holders or parent company.

I took a quick look at StartCom UK (as the information there is free) and noticed Mr. Wang became a director again [3]... I wonder who is "StartCom CA Spain Sociedad Limitada" parent/share holder, maybe a disclosure?

Inigo Barreira

unread,

Aug 8, 2017, 4:39:39 AM8/8/17

to Jakob Bohm, mozilla-dev-s...@lists.mozilla.org

Hi,

1.- yes, I said many times that it was not a good decission and of course not the best way to start, but at all times these test certs were under control, lived only for some minutes. Everything was explained in bugzilla #1369359

2.- Those pre-certificates were related to these test certificates for the CT testing, and yes, technically they didn´t exist for EJBCA, so we were dealing with them on how to revoke them without issuing them wrongly and finally got a solution, and then inmediately revoked them.

So, these two issues were related to the same generic problem, the CT testing, which was only for Chrome compliance, but that were all the time controlled by us. It was a bad practice that it´s not happening again with the countermeasures set as explained.

3.- No, there´s nothing related to Wosign nor Richard Wang. As indicated in our remediation plan, 360 owns 100% of StartCom and we´ve performed all the changes proposed there to have nothing related to Wosign nor Richard. This is indicated in bugzilla #1311832 and also in the remediation plan delivered last year.
Also in our remediation plan, there was a chart in which was indicated how the Company is structured, and in the case of StartCom Spain, this is 100% owned by the StartCom UK, and I´m director in both offices.

And the reason why Richard Wang is director again in StartCom UK is due to some bank issues. Richard was removed as director of the StartCom UK at that time, but didn´t realize about his signatory rights in the UK bank account. We thought we could change that easily but was not possible. We´ve been dealing with our lawyers, Alliotts, and finally agreed that signing again Richard was going to be quicker and easier. I had planned to go to London for changing it end of july, beginning of august but due to some personal matters, it was imposible and have had to delay until September. Once we finish this bank issue, Richard will be removed again.
In any case these are internal issues that don´t affect the PKI itself, these are administrative tasks, but the fact that Richard is again director in StartCom UK is just for this matter and has no other responsibility or function within StartCom.

Let me know if you need more clarification or have some other doubts or questions

Best regards

Iñigo Barreira
CEO
StartCom CA Limited

-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+inigo=startco...@lists.mozilla.org] On Behalf Of Jakob Bohm via dev-security-policy
Sent: lunes, 7 de agosto de 2017 22:03
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: StartCom cross-signs disclosed by Certinomis

On 07/08/2017 11:21, Franck Leroy wrote:

As an "outside" observer, I would say that it would have been better if the following additional things had been done:

1. At StartCom: The various tests should have been done with a dedicated
"test hierarchy" running on the production hardware but not chaining
to the intended production roots. This could also become a long term
test hierarchy for e.g. new software versions.

2. At StartCom: Remember to revoke certificates that get cancelled
between CT logging and actual issuance. I don't know if EJBCA can do
this without actually issuing the bad certificate, but hopefully there
is a way.

3. At Certinomis: Be more public up front about the "sealed" signing of
the cross certificate. Also, I am not sure how often your ARL signing
ceremonies happen, but if its every 3 months, the signing could have
happened in July instead.

4. At Certinomis: Maybe the cross-cert should have been post-dated to
a date when StartCom expected to be ready (as communicated privately).

5. At StartCom, Mozilla and Google: Should have been more clear if the
remediation plan did or did not require StartCom to seek
Mozilla/Google prior approval before getting cross-signed (if such
cross-signing were to occur before Mozilla approval of the new roots).

6. At StartCom: Be more clear if any of the "Chinese" staff is working
at, under or otherwise near WoSign and/or Richard Wang.

7. At Quihoo: Actually get rid of Richard Wang, not just change his
title from CEO to COO.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors.

WiseMo - Remote Service Management for PCs, Phones and Embedded _______________________________________________

Inigo Barreira

unread,

Aug 8, 2017, 4:51:09 AM8/8/17

to Itzhak Daniel, mozilla-dev-s...@lists.mozilla.org

Hi,

In the remediation plan that was published in October there was a chart in
which was indicate how the group was going to change, from WoSign management
to be under 360 management. I can provide the information again if you wish.

StartCom Spain is 100% owned by Startcom UK, which is also 100% owned by the
Startcom HK which is the head of the group. Over this, there´s 360 managing
it all. I don´t think I have to share here the notary papers but this is how
it´s structured now.
About the incorporation of Richard, I already explained the reason, it´s
just a bank issue and it´s just temporary until we can change the signatory.

Best regards

Iñigo Barreira
CEO
StartCom CA Limited

-----Original Message-----
From: dev-security-policy
[mailto:dev-security-policy-bounces+inigo=startco...@lists.mozilla.org]
On Behalf Of Itzhak Daniel via dev-security-policy
Sent: lunes, 7 de agosto de 2017 23:36
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: StartCom cross-signs disclosed by Certinomis

On Monday, August 7, 2017 at 11:03:27 PM UTC+3, Jakob Bohm wrote:

> 7. At Quihoo: Actually get rid of Richard Wang, not just change his
> title from CEO to COO.

I didn't map the new hierarchy of the "Spanish" StartCom CA ("StartCom CA
Spain Sociedad Limitada"), having trouble registering to the Spanish company
house and pull documents (I pulled from 3rd party, but they're garbage [1]
[2]). I did mange to see that Mr. Barreira is the Directory but nothing on
the share holders or parent company.

I took a quick look at StartCom UK (as the information there is free) and
noticed Mr. Wang became a director again [3]... I wonder who is "StartCom CA
Spain Sociedad Limitada" parent/share holder, maybe a disclosure?

Links:
1. https://www.letsphish.org/files/StartCom-CA-SPA-Appointment.pdf
2. https://www.letsphish.org/files/StartCom-CA-SPA-Profile.pdf
3. https://beta.companieshouse.gov.uk/company/09744347

Inigo Barreira

unread,

Aug 8, 2017, 5:01:30 AM8/8/17

to Percy, mozilla-dev-s...@lists.mozilla.org

Hi Percy,

StartCom Spain exists since september last year. And it was included in the
remediation plan set in October last year, but at the time Gerv wrote that
email it didn´t exist officially, it took a while to be registered
officially in the "equivalent" spanish companies house.
The process started in august, it´s typically a holiday month, and it took
longer than expected because there were some organizations closed or with
few people working.

Best regards

Iñigo Barreira
CEO
StartCom CA Limited

-----Original Message-----
From: dev-security-policy
[mailto:dev-security-policy-bounces+inigo=startco...@lists.mozilla.org]
On Behalf Of Percy via dev-security-policy
Sent: martes, 8 de agosto de 2017 2:39
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: StartCom cross-signs disclosed by Certinomis

StartCom CA Spain Sociedad Limitada was not in the original hierarchy [1] or
the proposed hierarchy [2] . I request that StartCom makes a full disclosure
of the ownership information.

In addition, in the WoSign remediation plan, WoSign Stated[3] that

Due to the severity of issues noted within, the decision has been made to
address the above three areas as they fall under the areas of 1)
leadership/authority in WoSign and StartCom, 2) operational/business process
and 3) technology.

If WoSign/Startcom has determined leadership/authority is the No.1 cause of
issues, why is Richard Wang appointed as a director of StartCom merely 6
month after his removal? This doesn't even mention his COO role at WoSign.

[1]
https://groups.google.com/forum/#!searchin/mozilla.dev.security.policy/start
com%7Csort:relevance/mozilla.dev.security.policy/0pqpLJ_lCJQ/z69lmZ88DwAJ
[2]https://en.wikipedia.org/wiki/StartCom#cite_note-structure201612-9
[3]https://www.wosign.com/report/WoSign_Incident_Report_Update_07102016.pdf

uri...@gmail.com

unread,

Aug 8, 2017, 9:43:06 AM8/8/17

to mozilla-dev-s...@lists.mozilla.org

Can this be responded to more directly and comprehensively please?

Are there any staff or personnel being shared between WoSign and Startcom?
This includes any staff from (or paid by) Qihoo 360 its subsidiaries, contractors, or affiliates--does anyone do any work (paid or unpaid) for both Wosign and Startcom? Are there any personnel switching between WoSign and Startcom?

Inigo Barreira

unread,

Aug 8, 2017, 10:01:14 AM8/8/17

to uri...@gmail.com, mozilla-dev-s...@lists.mozilla.org

Can this be responded to more directly and comprehensively please?

Are there any staff or personnel being shared between WoSign and Startcom?

No

This includes any staff from (or paid by) Qihoo 360 its subsidiaries, contractors, or affiliates--does anyone do any work (paid or unpaid) for both Wosign and Startcom?

No

Are there any personnel switching between WoSign and Startcom?

No

On Tuesday, August 8, 2017 at 4:39:39 AM UTC-4, Inigo Barreira wrote:

Gervase Markham

unread,

Sep 11, 2017, 5:59:24 AM9/11/17

to mozilla-dev-s...@lists.mozilla.org

Getting back to this very late... I am studying this situation today.

On 07/08/17 10:21, Franck Leroy wrote:
> Then in November 2016 I contacted Kathleen and Gerv to know if there was some stoppers to work with Inigo to help StartCom to be back in the business.
> There was no opposition as long as we follow the requirements of the remediation plan. Gerv also answered that our plan was good to him.

The plan I approved was the following (quoting you):

"May be the safer solution for the interim period they have their new
root trusted, is that ;

+ we create a new subCAs with startssl names (signed by Certinomis
root) in our pki systems on a dedicated HSM.

+ startcom will only have RA access, and we can control all certs
issuance as the HSM is under our control.

+ when startssl new root is publicly trusted by Mozilla, we give the
HSM and an export of the database to Startcom.

+ then startcom cross sign the dedicated CAs with their new root, and
then they are autonomous to use the CAs their own pki system."

This seems to be very different to the plan you implemented.

In that email exchange, you asked if a cross-sign was permitted.
Kathleen replied:

"It would have to be for their new root(s) only. Definitely not allowed
for their old roots.

As always, the CA with the root cert in Mozilla's program is responsible
for ensuring that their subCAs fully comply with the CA Browser Forum's
Baseline Requirements and Mozilla's CA Certificate Policy.

I think the following from
https://bugzilla.mozilla.org/show_bug.cgi?id=1311832 would still apply:
<list of action items from that bug>"

Various bugs have been filed since then to suggest that StartCom has not
been following those two documents. In addition, StartCom's first
attempt at demonstrating they had met that list of action items (leaving
aside the question of whether they, in fact, have) was in mid-July:
https://bugzilla.mozilla.org/show_bug.cgi?id=1311832#c12

This was long after you did your cross-sign.

I am continuing to consider what the best next steps are in this situation.

Gerv

Gervase Markham

unread,

Sep 11, 2017, 7:27:16 AM9/11/17

to Franck Leroy

Hi Franck,

On 03/08/17 08:59, Franck Leroy wrote:
> On end of June the audit report form PwC was available but with still some minor issues. I asked StartCom to correct them.
>
> On July 14th the audit report and the policy were updated and published on StartCom website.

The audit reports on StartCom's website
<https://www.startcomca.com/policy> are dated at the end of June, and
have significant qualifications. E.g.:
https://www.startcomca.com/pwc-webtrust-ca-2017.pdf

What updates to the audit reports were made on July 14th?

Do you consider these audit reports sufficient to say the StartCom has
passed these audits, despite the qualifications therein?

Gerv

Inigo Barreira

unread,

Sep 11, 2017, 10:00:21 AM9/11/17

to Gervase Markham, Franck Leroy, mozilla-dev-s...@lists.mozilla.org

Hi Gerv,

Those updates are referred basically to the format of the report in which
Franck asked to include specific information such as the serial number,
names, etc. according to your instructions. The report itself has not been
changed (that´s forbidden).

Regarding the qualifications or findings, the majority of them were fixed at
that time as the auditors explain in the section "other questions". There
were only 2 pending, the BCP and the TSA, which have been finished and do
not affect to the validation&issuance processes.
I can provide responses to all those findings, as I did to the auditors,
with evidences.

Best regards

Iñigo Barreira
CEO
StartCom CA Limited

-----Original Message-----
From: dev-security-policy
[mailto:dev-security-policy-bounces+inigo=startco...@lists.mozilla.org]
On Behalf Of Gervase Markham via dev-security-policy
Sent: lunes, 11 de septiembre de 2017 13:27
To: Franck Leroy <fr.l...@gmail.com>;
mozilla-dev-s...@lists.mozilla.org
Subject: Re: StartCom cross-signs disclosed by Certinomis