Following up on Trustico: reseller practices and accountability

955 views
Skip to first unread message

Eric Mill

unread,
Mar 4, 2018, 4:06:23 PM3/4/18
to mozilla-dev-s...@lists.mozilla.org
Last week, Trustico (a reseller, formerly for Symantec and now for Comodo)
sent 23,000 private keys to DigiCert, to force their revocation. This
showed that Trustico had been storing customer keys generated through one
or more CSR/key generation forms on their website.

Though Trustico disagrees, this appears to be a clear case of routine key
compromise for subscribers who obtained their key from Trustico. The
security of Trustico's systems, which are not audited or accountable to
root program requirements, were storing large amounts of key material whose
compromise could have led to the subsequent compromise of connections to
tens of thousands of online services.

It was also noted that Trustico was exposing key material to interception
by a number of third parties through client-side JavaScript embeds, and
that Trustico's website had functionality that allowed remote code
execution as root on one of their web servers.

These m.d.s.p threads document/link to those things:

*
https://groups.google.com/d/topic/mozilla.dev.security.policy/wxX4Yv0E3Mk/discussion
*
https://groups.google.com/d/topic/mozilla.dev.security.policy/BLvabFwcJqo/discussion

As part of the second thread, Comodo noted:

We also asked Trustico to cease offering any tools to generate and/or
retain customer private keys. They have complied with this request and
have confirmed that they do not intend to offer any such tools again in the
future.


That is good to hear, but a "we won't do it again" response, if accepted by
Comodo as sufficient, seems disproportionate to the severity of the issue,
given Trustico's unfamiliarity with norms around private key management,
and with basic security practices.

It's also clear from the experience that rules of the road for resellers
are unclear, and that accountability is limited. It seems possible, or
likely, that other resellers may also be mishandling customer keys

So, what would useful next steps be to improve security and accountability
for resellers?

One thought: Mozilla could ask CAs to obtain a written response from all
contracted resellers about if/how they interact with customer key material,
including the level of isolation/security given their key generation
environment (if they have one), and whether any third-party JavaScript is
given access to generated key material.

Any other ideas?

Also -- Comodo noted:

Trustico have also confirmed to us that they were not, and are not, in
possession of the private keys that correspond to any of the certificates
that they have requested for their customers through Comodo CA.


Since there appears to have been a significant overlap period, between the
time Trustico switched to Comodo and when Trustico was asked by Comodo to
cease key storage practices, it's a little hard to take at face value the
assurance that Trustico was never in possession of any Comodo keys. It
would be nice to hear something from Comodo about whether they've verified
this in any more detail.

-- Eric

--
konklone.com | @konklone <https://twitter.com/konklone>

Paul Kehrer

unread,
Mar 4, 2018, 4:44:26 PM3/4/18
to mozilla-dev-s...@lists.mozilla.org
On March 4, 2018 at 5:06:41 PM, Eric Mill via dev-security-policy (
dev-secur...@lists.mozilla.org) wrote:

<snip>

It's also clear from the experience that rules of the road for resellers
are unclear, and that accountability is limited. It seems possible, or
likely, that other resellers may also be mishandling customer keys

So, what would useful next steps be to improve security and accountability
for resellers?


As you already suggested an official communication requesting information
from the CAs about the way their reseller networks manage subscriber key
material would be a good start. Eventually I think it's likely that
resellers need to be subject to some limited form of audit (maybe as
simplistic as a PCI self-assessment questionnaire?). While that doesn't
prevent bad behavior it would generate an evidence trail for termination of
relationships with incompetent/malicious actors.

Of course, CAs are likely to be reluctant to share a complete list of their
resellers since they probably consider that competitive information. So, it
would be nice if we could just make it part of the CA's audits...

One way to do that would be that the baseline requirements could be updated
to create a section defining requirements placed upon resellers (especially
around subscriber key management). This way CAs would be incentivized to
manage their business relationships more carefully since their business
partners could cause them audit issues. This has some precedent since in
the past some resellers acted as RAs and had their own subordinates -- a
practice that was terminated as they came under scrutiny and demands for
audits.

Mozilla, of course, cannot amend the BRs itself. However, past evidence
suggests that if the Mozilla program introduces their own requirements
around reseller management and disclosure then the probability of a CABF
ballot with similar restrictions passing is relatively high (thus getting
it into the audit regime).

-Paul

Anis

unread,
Mar 4, 2018, 5:11:26 PM3/4/18
to mozilla-dev-s...@lists.mozilla.org
It is essential to have the reseller contract draft presented as well as to check the procedures followed by the reseller in order to provide a reliable and standards compliant service.
And also, I would like to know how COMODO had the confirmation that Trustico will no longer produce this kind of act in the future.

Ryan Sleevi

unread,
Mar 4, 2018, 6:04:15 PM3/4/18
to Eric Mill, mozilla-dev-s...@lists.mozilla.org
On Sun, Mar 4, 2018 at 4:04 PM, Eric Mill via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> So, what would useful next steps be to improve security and accountability
> for resellers?
>

It depends - do you view resellers as the user's delegated agent - that is,
much like one might contract out IT services or janitorial services - or do
you view them as the CA's agent - outsourced marketing and technical
support?

Answering this question seems key to understanding what, if any, meaningful
reforms can be offered. And I would not look to the discounts offered
resellers as a way of arguing that they are the CA's agent, since CA's
regularly offer discounts to customers who purchase in bulk.

okaphone.e...@gmail.com

unread,
Mar 5, 2018, 7:52:05 AM3/5/18
to mozilla-dev-s...@lists.mozilla.org
On Sunday, 4 March 2018 22:44:26 UTC+1, Paul Kehrer wrote:
> On March 4, 2018 at 5:06:41 PM, Eric Mill via dev-security-policy (
> dev-secur...@lists.mozilla.org) wrote:
>
> <snip>
>
> It's also clear from the experience that rules of the road for resellers
> are unclear, and that accountability is limited. It seems possible, or
> likely, that other resellers may also be mishandling customer keys
>
> So, what would useful next steps be to improve security and accountability
> for resellers?
>
>
> As you already suggested an official communication requesting information
> from the CAs about the way their reseller networks manage subscriber key
> material would be a good start. Eventually I think it's likely that
> resellers need to be subject to some limited form of audit (maybe as
> simplistic as a PCI self-assessment questionnaire?). While that doesn't
> prevent bad behavior it would generate an evidence trail for termination of
> relationships with incompetent/malicious actors.

I'm not sure that that would be reasonable. After all resellers can have resellers, and so on so where would that end? With the end user having to accept a "general license agreement"? And distrusting a reseller could also be difficult.

I think it will have to be/remain the responsibility of the CA to choose their reselllers in such a way that "not too many questions are being asked" about them.

James Burton

unread,
Mar 5, 2018, 8:27:06 AM3/5/18
to okaphone.e...@gmail.com, mozilla-dev-security-policy
Currently, resellers are allowed to submit CSRs on behalf of their
customers and as we've seen this is open to abuse. Maybe it's time to stop
this practice and restrict submission of CSRs to CA portals only.

On Mon, Mar 5, 2018 at 12:51 PM, okaphone.elektronika--- via
dev-security-policy <dev-secur...@lists.mozilla.org> wrote:

> On Sunday, 4 March 2018 22:44:26 UTC+1, Paul Kehrer wrote:
> > On March 4, 2018 at 5:06:41 PM, Eric Mill via dev-security-policy (
> > dev-secur...@lists.mozilla.org) wrote:
> >
> > <snip>
> >
> > It's also clear from the experience that rules of the road for resellers
> > are unclear, and that accountability is limited. It seems possible, or
> > likely, that other resellers may also be mishandling customer keys
> >
> > So, what would useful next steps be to improve security and
> accountability
> > for resellers?
> >
> >
> > As you already suggested an official communication requesting information
> > from the CAs about the way their reseller networks manage subscriber key
> > material would be a good start. Eventually I think it's likely that
> > resellers need to be subject to some limited form of audit (maybe as
> > simplistic as a PCI self-assessment questionnaire?). While that doesn't
> > prevent bad behavior it would generate an evidence trail for termination
> of
> > relationships with incompetent/malicious actors.
>
> I'm not sure that that would be reasonable. After all resellers can have
> resellers, and so on so where would that end? With the end user having to
> accept a "general license agreement"? And distrusting a reseller could also
> be difficult.
>
> I think it will have to be/remain the responsibility of the CA to choose
> their reselllers in such a way that "not too many questions are being
> asked" about them.
>
>
> > Of course, CAs are likely to be reluctant to share a complete list of
> their
> > resellers since they probably consider that competitive information. So,
> it
> > would be nice if we could just make it part of the CA's audits...
> >
> > One way to do that would be that the baseline requirements could be
> updated
> > to create a section defining requirements placed upon resellers
> (especially
> > around subscriber key management). This way CAs would be incentivized to
> > manage their business relationships more carefully since their business
> > partners could cause them audit issues. This has some precedent since in
> > the past some resellers acted as RAs and had their own subordinates -- a
> > practice that was terminated as they came under scrutiny and demands for
> > audits.
> >
> > Mozilla, of course, cannot amend the BRs itself. However, past evidence
> > suggests that if the Mozilla program introduces their own requirements
> > around reseller management and disclosure then the probability of a CABF
> > ballot with similar restrictions passing is relatively high (thus getting
> > it into the audit regime).
> >
> > -Paul
>
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

Ryan Sleevi

unread,
Mar 5, 2018, 9:16:22 AM3/5/18
to James Burton, mozilla-dev-security-policy, OKAPHONE Elektronika
Considering that the Baseline Requirements explicitly acknowledge that the
Applicant may delegate the obtaining of their certificates to a third-party
(Applicant Representative), how would you propose that the applicant's
agents (which, in a legal sense, is the name for their employees - that is,
those with legal authority to represent the company) and resellers?

What would stop someone from offering a "CSR-as-a-service" in which they
generate CSRs for users, and then users take that generated CSR to the CA?
What role are you suggesting that the CA has to play in policing 'how' the
CSR was generated - since a CSR is-a CSR is-a CSR?

On Mon, Mar 5, 2018 at 8:26 AM, James Burton via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Currently, resellers are allowed to submit CSRs on behalf of their
> customers and as we've seen this is open to abuse. Maybe it's time to stop
> this practice and restrict submission of CSRs to CA portals only.
>
> On Mon, Mar 5, 2018 at 12:51 PM, okaphone.elektronika--- via
> dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
>
> > On Sunday, 4 March 2018 22:44:26 UTC+1, Paul Kehrer wrote:
> > > On March 4, 2018 at 5:06:41 PM, Eric Mill via dev-security-policy (
> > > dev-secur...@lists.mozilla.org) wrote:
> > >
> > > <snip>
> > >
> > > It's also clear from the experience that rules of the road for
> resellers
> > > are unclear, and that accountability is limited. It seems possible, or
> > > likely, that other resellers may also be mishandling customer keys
> > >
> > > So, what would useful next steps be to improve security and
> > accountability
> > > for resellers?
> > >
> > >
> > > As you already suggested an official communication requesting
> information
> > > from the CAs about the way their reseller networks manage subscriber
> key
> > > material would be a good start. Eventually I think it's likely that
> > > resellers need to be subject to some limited form of audit (maybe as
> > > simplistic as a PCI self-assessment questionnaire?). While that doesn't
> > > prevent bad behavior it would generate an evidence trail for
> termination
> > of
> > > relationships with incompetent/malicious actors.
> >
> > I'm not sure that that would be reasonable. After all resellers can have
> > resellers, and so on so where would that end? With the end user having to
> > accept a "general license agreement"? And distrusting a reseller could
> also
> > be difficult.
> >
> > I think it will have to be/remain the responsibility of the CA to choose
> > their reselllers in such a way that "not too many questions are being
> > asked" about them.
> >
> >

James Burton

unread,
Mar 5, 2018, 9:43:14 AM3/5/18
to Ryan Sleevi, mozilla-dev-security-policy, OKAPHONE Elektronika
It wouldn't stop someone from offering such a service and it also wouldn't
prevent users from using that CSR as it is their choice in the end. This
was just an idea.
CAs shouldn't be policing users. CAs should be instead enforcing best
practices on resellers as practices like this shaken user confidence to a
degree which affects all.




On Mon, Mar 5, 2018 at 2:15 PM, Ryan Sleevi <ry...@sleevi.com> wrote:

> Considering that the Baseline Requirements explicitly acknowledge that the
> Applicant may delegate the obtaining of their certificates to a third-party
> (Applicant Representative), how would you propose that the applicant's
> agents (which, in a legal sense, is the name for their employees - that is,
> those with legal authority to represent the company) and resellers?
>
> What would stop someone from offering a "CSR-as-a-service" in which they
> generate CSRs for users, and then users take that generated CSR to the CA?
> What role are you suggesting that the CA has to play in policing 'how' the
> CSR was generated - since a CSR is-a CSR is-a CSR?
>
> On Mon, Mar 5, 2018 at 8:26 AM, James Burton via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
>
>> Currently, resellers are allowed to submit CSRs on behalf of their
>> customers and as we've seen this is open to abuse. Maybe it's time to stop
>> this practice and restrict submission of CSRs to CA portals only.
>>
>> On Mon, Mar 5, 2018 at 12:51 PM, okaphone.elektronika--- via
>> dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
>>
>> > On Sunday, 4 March 2018 22:44:26 UTC+1, Paul Kehrer wrote:
>> > > On March 4, 2018 at 5:06:41 PM, Eric Mill via dev-security-policy (
>> > > dev-secur...@lists.mozilla.org) wrote:
>> > >
>> > > <snip>
>> > >
>> > > It's also clear from the experience that rules of the road for
>> resellers
>> > > are unclear, and that accountability is limited. It seems possible, or
>> > > likely, that other resellers may also be mishandling customer keys
>> > >
>> > > So, what would useful next steps be to improve security and
>> > accountability
>> > > for resellers?
>> > >
>> > >
>> > > As you already suggested an official communication requesting
>> information
>> > > from the CAs about the way their reseller networks manage subscriber
>> key
>> > > material would be a good start. Eventually I think it's likely that
>> > > resellers need to be subject to some limited form of audit (maybe as
>> > > simplistic as a PCI self-assessment questionnaire?). While that
>> doesn't
>> > > prevent bad behavior it would generate an evidence trail for
>> termination
>> > of
>> > > relationships with incompetent/malicious actors.
>> >
>> > I'm not sure that that would be reasonable. After all resellers can have
>> > resellers, and so on so where would that end? With the end user having
>> to
>> > accept a "general license agreement"? And distrusting a reseller could
>> also
>> > be difficult.
>> >
>> > I think it will have to be/remain the responsibility of the CA to choose
>> > their reselllers in such a way that "not too many questions are being
>> > asked" about them.
>> >
>> >

Eric Mill

unread,
Mar 5, 2018, 9:44:20 AM3/5/18
to Ryan Sleevi, James Burton, mozilla-dev-security-policy, OKAPHONE Elektronika
I think it probably makes more sense to focus on sensitive key material
than non-sensitive CSRs.

As a starting point, how reasonable would it be for CAs to question their
resellers, or to disseminate additional language to add to their reseller
agreements to prohibit non-transactional/ephemeral key storage?

-- Eric

On Mon, Mar 5, 2018 at 9:15 AM, Ryan Sleevi via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Considering that the Baseline Requirements explicitly acknowledge that the
> Applicant may delegate the obtaining of their certificates to a third-party
> (Applicant Representative), how would you propose that the applicant's
> agents (which, in a legal sense, is the name for their employees - that is,
> those with legal authority to represent the company) and resellers?
>
> What would stop someone from offering a "CSR-as-a-service" in which they
> generate CSRs for users, and then users take that generated CSR to the CA?
> What role are you suggesting that the CA has to play in policing 'how' the
> CSR was generated - since a CSR is-a CSR is-a CSR?
>
> On Mon, Mar 5, 2018 at 8:26 AM, James Burton via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
>
> > Currently, resellers are allowed to submit CSRs on behalf of their
> > customers and as we've seen this is open to abuse. Maybe it's time to
> stop
> > this practice and restrict submission of CSRs to CA portals only.
> >
> > On Mon, Mar 5, 2018 at 12:51 PM, okaphone.elektronika--- via
> > dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
> >
> > > On Sunday, 4 March 2018 22:44:26 UTC+1, Paul Kehrer wrote:
> > > > On March 4, 2018 at 5:06:41 PM, Eric Mill via dev-security-policy (
> > > > dev-secur...@lists.mozilla.org) wrote:
> > > >
> > > > <snip>
> > > >
> > > > It's also clear from the experience that rules of the road for
> > resellers
> > > > are unclear, and that accountability is limited. It seems possible,
> or
> > > > likely, that other resellers may also be mishandling customer keys
> > > >
> > > > So, what would useful next steps be to improve security and
> > > accountability
> > > > for resellers?
> > > >
> > > >
> > > > As you already suggested an official communication requesting
> > information
> > > > from the CAs about the way their reseller networks manage subscriber
> > key
> > > > material would be a good start. Eventually I think it's likely that
> > > > resellers need to be subject to some limited form of audit (maybe as
> > > > simplistic as a PCI self-assessment questionnaire?). While that
> doesn't
> > > > prevent bad behavior it would generate an evidence trail for
> > termination
> > > of
> > > > relationships with incompetent/malicious actors.
> > >
> > > I'm not sure that that would be reasonable. After all resellers can
> have
> > > resellers, and so on so where would that end? With the end user having
> to
> > > accept a "general license agreement"? And distrusting a reseller could
> > also
> > > be difficult.
> > >
> > > I think it will have to be/remain the responsibility of the CA to
> choose
> > > their reselllers in such a way that "not too many questions are being
> > > asked" about them.
> > >
> > >
> > > _______________________________________________
> > > dev-security-policy mailing list
> > > dev-secur...@lists.mozilla.org
> > > https://lists.mozilla.org/listinfo/dev-security-policy
> > >
> > _______________________________________________
> > dev-security-policy mailing list
> > dev-secur...@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-security-policy
> >
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>



okaphone.e...@gmail.com

unread,
Mar 5, 2018, 12:10:17 PM3/5/18
to mozilla-dev-s...@lists.mozilla.org
On Monday, 5 March 2018 15:44:20 UTC+1, Eric Mill wrote:
> I think it probably makes more sense to focus on sensitive key material
> than non-sensitive CSRs.
>
> As a starting point, how reasonable would it be for CAs to question their
> resellers, or to disseminate additional language to add to their reseller
> agreements to prohibit non-transactional/ephemeral key storage?
>
> --
> konklone.com | @konklone <https://twitter.com/konklone>

I remember I read somewhere in the discussion about this incident that Trustico also may have taken normal CSR's from their customers and replaced them with CSR's they generated. Which would typically go unnoticed as they then after singing provided a file (pem or whatever) that could be put on the users webserver "without any further hassle". That is, they private key in that file would be the one Trustico used for that replacement CSR and not the one the customer provided. So, ehm... it worked. ;-)

I do not know if that is truly what happend. But if it is, it would explain why some people on reddit persisted in saying their key could not have been compromised as they never gave it to Trustico at all, but they still got the notification E-Mail that their certificatie was about to be revoked.

I don't know if it's worth investigating that angle any further, but it's perhaps a scenario worth considering in this context. For the reseller would strictly not be storing private keys of such subscribers.

CU Hans

okaphone.e...@gmail.com

unread,
Mar 5, 2018, 12:29:59 PM3/5/18
to mozilla-dev-s...@lists.mozilla.org
Ah, found it. It was tialaramex who suggested that this could be how Trustico got the private keys. https://www.reddit.com/r/sysadmin/comments/80uaq3/digicert_certificates_being_revoked/duyg6pn/

Just speculation then. But still worth keeping in mind as something a reseller could be doing. I can just see some programmer coming up with this idea to workaround the problem of not having the private key. ;-)

CU Hans

Ryan Sleevi

unread,
Mar 5, 2018, 12:51:25 PM3/5/18
to okaphone.e...@gmail.com, mozilla-dev-s...@lists.mozilla.org
On Mon, Mar 5, 2018 at 12:10 PM okaphone.elektronika--- via
dev-security-policy <dev-secur...@lists.mozilla.org> wrote:

> On Monday, 5 March 2018 15:44:20 UTC+1, Eric Mill wrote:
> > I think it probably makes more sense to focus on sensitive key material
> > than non-sensitive CSRs.
> >
> > As a starting point, how reasonable would it be for CAs to question their
> > resellers, or to disseminate additional language to add to their reseller
> > agreements to prohibit non-transactional/ephemeral key storage?
> >
> > --
> > konklone.com | @konklone <https://twitter.com/konklone>
>
> I remember I read somewhere in the discussion about this incident that
> Trustico also may have taken normal CSR's from their customers and replaced
> them with CSR's they generated. Which would typically go unnoticed as they
> then after singing provided a file (pem or whatever) that could be put on
> the users webserver "without any further hassle". That is, they private key
> in that file would be the one Trustico used for that replacement CSR and
> not the one the customer provided. So, ehm... it worked. ;-)
>

I think unfounded speculation such as this is actively harmful.
Extraordinary claims deserve some evidence. Otherwise, it makes discussion
no better than an angry mob, particularly because of the technical
challenges.

I do not know if that is truly what happend. But if it is, it would explain
> why some people on reddit persisted in saying their key could not have been
> compromised as they never gave it to Trustico at all, but they still got
> the notification E-Mail that their certificatie was about to be revoked.


A much simpler explanation is one already echoed on the list. Trustico
requested revocation for all the certs they sold, but could only provide
the keys for the certs they generated. Yet because they requested it for
all, and stated compromise, an email was sent to those customers - even
though no evidence of compromise had been provided.


>
> I don't know if it's worth investigating that angle any further, but it's
> perhaps a scenario worth considering in this context. For the reseller
> would strictly not be storing private keys of such subscribers.
>
> CU Hans

Ryan Sleevi

unread,
Mar 5, 2018, 12:56:46 PM3/5/18
to Eric Mill, James Burton, Ryan Sleevi, mozilla-dev-security-policy, OKAPHONE Elektronika
I think it would be very unreasonable, unfortunately.

Such a discussion requires defining what a reseller is - and a whole
spectrum exists in which Amazon Certificate Manager, Venafi, Dreamhost, and
more (all very different services) could unduly get swept up into such a
definition.

Further, I think in a world of automated issuance, it would be even more
harmful towards automation, by suggesting more restrictions on who can get
certificates and how.

I understand the frustration that such resellers exist. I think it
highlights failures of the CAs that partner with such resllers to educate
them (and their customers) as to best practices. But I think beyond that,
restrictions by CAs or censure by browsers will be actively harmful to the
ecosystem as a whole, and that is why I push back on such proposals.

Is it ideal? No. Should we actively call out insecurity - absolutely. But I
hesitate to believe that more can be done without causing more harm than
good, sadly.

On Mon, Mar 5, 2018 at 9:42 AM Eric Mill <er...@konklone.com> wrote:

> I think it probably makes more sense to focus on sensitive key material
> than non-sensitive CSRs.
>
> As a starting point, how reasonable would it be for CAs to question their
> resellers, or to disseminate additional language to add to their reseller
> agreements to prohibit non-transactional/ephemeral key storage?
>
> -- Eric
>
> On Mon, Mar 5, 2018 at 9:15 AM, Ryan Sleevi via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
>
>> Considering that the Baseline Requirements explicitly acknowledge that the
>> Applicant may delegate the obtaining of their certificates to a
>> third-party
>> (Applicant Representative), how would you propose that the applicant's
>> agents (which, in a legal sense, is the name for their employees - that
>> is,
>> those with legal authority to represent the company) and resellers?
>>
>> What would stop someone from offering a "CSR-as-a-service" in which they
>> generate CSRs for users, and then users take that generated CSR to the CA?
>> What role are you suggesting that the CA has to play in policing 'how' the
>> CSR was generated - since a CSR is-a CSR is-a CSR?
>>
>> On Mon, Mar 5, 2018 at 8:26 AM, James Burton via dev-security-policy <
>> dev-secur...@lists.mozilla.org> wrote:
>>
>> > Currently, resellers are allowed to submit CSRs on behalf of their
>> > customers and as we've seen this is open to abuse. Maybe it's time to
>> stop
>> > this practice and restrict submission of CSRs to CA portals only.
>> >
>> > On Mon, Mar 5, 2018 at 12:51 PM, okaphone.elektronika--- via
>> > dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
>> >
>> > > On Sunday, 4 March 2018 22:44:26 UTC+1, Paul Kehrer wrote:
>> > > > On March 4, 2018 at 5:06:41 PM, Eric Mill via dev-security-policy (
>> > > > dev-secur...@lists.mozilla.org) wrote:
>> > > >
>> > > > <snip>
>> > > >
>> > > > It's also clear from the experience that rules of the road for
>> > resellers
>> > > > are unclear, and that accountability is limited. It seems possible,
>> or
>> > > > likely, that other resellers may also be mishandling customer keys
>> > > >
>> > > > So, what would useful next steps be to improve security and
>> > > accountability
>> > > > for resellers?
>> > > >
>> > > >
>> > > > As you already suggested an official communication requesting
>> > information
>> > > > from the CAs about the way their reseller networks manage subscriber
>> > key
>> > > > material would be a good start. Eventually I think it's likely that
>> > > > resellers need to be subject to some limited form of audit (maybe as
>> > > > simplistic as a PCI self-assessment questionnaire?). While that
>> doesn't
>> > > > prevent bad behavior it would generate an evidence trail for
>> > termination
>> > > of
>> > > > relationships with incompetent/malicious actors.
>> > >
>> > > I'm not sure that that would be reasonable. After all resellers can
>> have
>> > > resellers, and so on so where would that end? With the end user
>> having to
>> > > accept a "general license agreement"? And distrusting a reseller could
>> > also
>> > > be difficult.
>> > >
>> > > I think it will have to be/remain the responsibility of the CA to
>> choose
>> > > their reselllers in such a way that "not too many questions are being
>> > > asked" about them.
>> > >
>> > >
>> > > _______________________________________________
>> > > dev-security-policy mailing list
>> > > dev-secur...@lists.mozilla.org
>> > > https://lists.mozilla.org/listinfo/dev-security-policy
>> > >
>> > _______________________________________________
>> > dev-security-policy mailing list
>> > dev-secur...@lists.mozilla.org
>> > https://lists.mozilla.org/listinfo/dev-security-policy
>> >
>> _______________________________________________
>> dev-security-policy mailing list
>> dev-secur...@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-security-policy
>>
>
>
>

Ryan Hurst

unread,
Mar 5, 2018, 2:17:39 PM3/5/18
to mozilla-dev-s...@lists.mozilla.org
I agree with Sleevi on this, the real question on what can and should be done here is dependent on who the reseller is an agent of and what role they play in the overall ecosystem.

While it is easy to say that resellers are pure marketers with no vested interest in security outcomes, and there is some truth to this, the reality is far more complex. For one there is no one size fits all definition for a reseller, for example:

- Hosting “reseller” - As a hosting provider, for example one that utilizes CPANEL, you may be responsible for enrolling for certificates and generating keys for users as well as managing the lifecycle, you are clearly acting “as a reseller” if you are selling “a certificate” but you are also acting as a delegate of the user if you are configuring and managing SSL for them.
- SaaS “reseller” - As a SaaS provider, for example one that hosts Wordpress, you may be responsible for enrolling for certificates and generating keys for users as well as managing the lifecycle, you are clearly acting “as a reseller” if you are selling “a certificate” but you are also acting as a delegate of the user if you are configuring and managing SSL for them.
- Marketing “resellers” - As a pure reseller, for example one that offers regional sales and marketing support, again you are clearly acting as a delegate of the CA by providing marketing and sales support for a vertical, region or market segment, but you could very well be providing value added services to the user (such as simplfying enrollment and/or SSL configuration) and as such are again a delagate of both parties.

As I look at this non-exhaustive list, it seems to me the difference between the reseller and a and the more typical SaaS service provider where SSL is possibly a paid feature is the sale of a certificate.

With that said, since there are so many different types of “other parties” it is probably better to avoid discussing resellers directly and focus on responsibilities of “other parties” instead.

For example, today the BRs require that CAs and RAs:
- Require consent to subscriber key archival (section 6.1.2),
- Require the encryption of the subscribers private in transport (section 6.1.2).

In no particular order here are some questions I have for myself on this topic:

- Should we provide a definition of “other parties”, and “reseller” and make sure they are clear so responsibilities of parties are unambiguous?
- In the BRs we currently say “Parties other than the Subscriber SHALL NOT archive the Subscriber Private Key” (in Section 6.1.2) should we also say that the CAs should be required to demonstrate they have communicated this requirement to the other party and get affirmative acknowledgement from the “other party” during their audits?
- The BRs currently state subscriber authorization is required for archival but there is no text covering what minimal level of authorization expressed. I would have thought this not necessary but TrustIco has been arguing users should have implicitly known they had this practice even though it was not disclosed and there was no explicit consent for archival. While I think that is a irresponsible position the text could be made clearer.
- The current BR text talks about RAs generating keys on behalf of the subscriber (6.1.2 but it says nothing about other parties?
- Should the BRs be revised to require CAs to have the “other parties” publicly disclose if they generate keys for users, how they protect them at rest and in transport, and how they capture consent for these practices if at all?
- Though the concept of key archival for RAs and CAs is allowed in the BRs (section 6.1.2) it does not require keys be encrypted while in archive, Should this be changed? At the same time should we mandate some minimal level of protection that would prevent all user keys being accessed without user consent like was done here?
- Today the BRs inconsistently discuss private key archival, for example section 6.2.5 talks about CA key archival but not subscriber archival. Should we fix this?
- Should we formalize a proof proof of possession mechanism, such as what is done in ACME, as an alternative to sharing the actual key as to encourage this approach to be used instead of distribution of the actual key?

One thing for us to keep in mind while looking at these issues is we are moving to a world where SSL is the default and for that to be true we need automation and permissionless SSL deployment (e.g. automation) is necessary for that to be a reality.

This discussion is probably better for the CABFORUM public list but since the thread started here I thought it best to share my thoughts here.

Ryan Hurst
Google Trust Services

Ryan Sleevi

unread,
Mar 5, 2018, 2:38:31 PM3/5/18
to Ryan Hurst, mozilla-dev-s...@lists.mozilla.org
While these are interesting questions, I think it gets to the heart of
policy questions, which is how is policy maintained and enforced. Today,
there’s only one method - distrust.

So are you suggesting the CA should be distrusted if these “other parties”
(which may have no observable relationship with the CA) don’t adhere to
this policy? Are you suggesting the certificates these “other parties” are
involved with get distrusted? Or something else?

Because without teeth, the policy suggestions themselves are hollow.


>
> One thing for us to keep in mind while looking at these issues is we are
> moving to a world where SSL is the default and for that to be true we need
> automation and permissionless SSL deployment (e.g. automation) is necessary
> for that to be a reality.
>
> This discussion is probably better for the CABFORUM public list but since
> the thread started here I thought it best to share my thoughts here.


I disagree on that venue suggestion, since here we can actually have
widespread public participation. I would also suggest that Section 1.3 of
the Bylaws would no doubt be something constantly having to be pointed out
in such discussions.


>
> Ryan Hurst
> Google Trust Services

Matthew Hardeman

unread,
Mar 5, 2018, 2:39:05 PM3/5/18
to James Burton, mozilla-dev-security-policy, okaphone.e...@gmail.com
On Mon, Mar 5, 2018 at 7:26 AM, James Burton via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Currently, resellers are allowed to submit CSRs on behalf of their
> customers and as we've seen this is open to abuse. Maybe it's time to stop
> this practice and restrict submission of CSRs to CA portals only.
>
>
Wouldn't that become prohibitively specific in light of automation, APIs
for submission, etc? Does ACME count as "submission of CSRs to CA portals
only".

Also, as many CAs effectively utilize only the CSR's public key payload and
signature performed by the private key, there are lots of non-CSR ways to
communicate those elements outside a formal CSR document.

Matthew Hardeman

unread,
Mar 5, 2018, 2:47:38 PM3/5/18
to Eric Mill, Ryan Sleevi, James Burton, mozilla-dev-security-policy, OKAPHONE Elektronika
In terms of an "opportunity" to insert regulation into the reseller
ecosystem, as Mr. Sleevi points out, there is the question of whether the
reseller is just a third party agent acting under contract by the end-user
client vs a party with a special relationship with one or more CAs and
their own end-user sales channel soliciting customers.

Ultimately, the reality is that the resellers all have interfaces offering
some level of automated processing for bulk ordering. These resellers
aren't acting as literal extra hands on behalf of the end user.

They have specialized access to bulk ordering interfaces, generally
pursuant to a reseller contract. They're not literally hand-entering data
as though they themselves were the end-user.

It's the existence of that relationship and access which provides an
opportunity for the community to decide "If the CA has relationships in
this nature, the CA has an affirmative duty to ensure as a condition of the
relationship that _______ (insert rules)".



On Mon, Mar 5, 2018 at 8:42 AM, Eric Mill via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> I think it probably makes more sense to focus on sensitive key material
> than non-sensitive CSRs.
>
> As a starting point, how reasonable would it be for CAs to question their
> resellers, or to disseminate additional language to add to their reseller
> agreements to prohibit non-transactional/ephemeral key storage?
>
> -- Eric
>
> On Mon, Mar 5, 2018 at 9:15 AM, Ryan Sleevi via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
>
> > Considering that the Baseline Requirements explicitly acknowledge that
> the
> > Applicant may delegate the obtaining of their certificates to a
> third-party
> > (Applicant Representative), how would you propose that the applicant's
> > agents (which, in a legal sense, is the name for their employees - that
> is,
> > those with legal authority to represent the company) and resellers?
> >
> > What would stop someone from offering a "CSR-as-a-service" in which they
> > generate CSRs for users, and then users take that generated CSR to the
> CA?
> > What role are you suggesting that the CA has to play in policing 'how'
> the
> > CSR was generated - since a CSR is-a CSR is-a CSR?
> >
> > On Mon, Mar 5, 2018 at 8:26 AM, James Burton via dev-security-policy <
> > dev-secur...@lists.mozilla.org> wrote:
> >
> > > Currently, resellers are allowed to submit CSRs on behalf of their
> > > customers and as we've seen this is open to abuse. Maybe it's time to
> > stop
> > > this practice and restrict submission of CSRs to CA portals only.
> > >
> > > On Mon, Mar 5, 2018 at 12:51 PM, okaphone.elektronika--- via
> > > dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
> > >
> > > > On Sunday, 4 March 2018 22:44:26 UTC+1, Paul Kehrer wrote:
> > > > I'm not sure that that would be reasonable. After all resellers can
> > have
> > > > resellers, and so on so where would that end? With the end user
> having
> > to
> > > > accept a "general license agreement"? And distrusting a reseller
> could
> > > also
> > > > be difficult.
> > > >
> > > > I think it will have to be/remain the responsibility of the CA to
> > choose
> > > > their reselllers in such a way that "not too many questions are being
> > > > asked" about them.
> > > >
> > > >
> > > > _______________________________________________
> > > > dev-security-policy mailing list
> > > > dev-secur...@lists.mozilla.org
> > > > https://lists.mozilla.org/listinfo/dev-security-policy
> > > >
> > > _______________________________________________
> > > dev-security-policy mailing list
> > > dev-secur...@lists.mozilla.org
> > > https://lists.mozilla.org/listinfo/dev-security-policy
> > >
> > _______________________________________________
> > dev-security-policy mailing list
> > dev-secur...@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-security-policy
> >
>
>
>

Ryan Hurst

unread,
Mar 5, 2018, 2:55:57 PM3/5/18
to mozilla-dev-s...@lists.mozilla.org
On Monday, March 5, 2018 at 11:38:31 AM UTC-8, Ryan Sleevi wrote:
> While these are interesting questions, I think it gets to the heart of
> policy questions, which is how is policy maintained and enforced. Today,
> there’s only one method - distrust.
>
> So are you suggesting the CA should be distrusted if these “other parties”
> (which may have no observable relationship with the CA) don’t adhere to
> this policy? Are you suggesting the certificates these “other parties” are
> involved with get distrusted? Or something else?
>
> Because without teeth, the policy suggestions themselves are hollow.

That is a very valid point.

Well since I do not have a concrete proposal it is hard to say at this point if a CA should be kicked out for non-conformance to a given critera. With that said today there are over 20 SHOULDs in the BRs and I can imagine failure to meet those should would be considered in aggregate when looking at a distrust event.

If nothing else addressing any potential ambiguity would be useful.

>
> I disagree on that venue suggestion, since here we can actually have
> widespread public participation. I would also suggest that Section 1.3 of
> the Bylaws would no doubt be something constantly having to be pointed out
> in such discussions.
>

Fair enough, as I am on the plane to CA/Browser Forum event maybe, as a result, I had this venue on my mind, I agree this is a fine venue for this discussion.

Ryan Sleevi

unread,
Mar 5, 2018, 3:51:30 PM3/5/18
to Matthew Hardeman, James Burton, Ryan Sleevi, mozilla-dev-security-policy, Eric Mill, OKAPHONE Elektronika
On Mon, Mar 5, 2018 at 2:47 PM Matthew Hardeman <mhar...@gmail.com> wrote:

> In terms of an "opportunity" to insert regulation into the reseller
> ecosystem, as Mr. Sleevi points out, there is the question of whether the
> reseller is just a third party agent acting under contract by the end-user
> client vs a party with a special relationship with one or more CAs and
> their own end-user sales channel soliciting customers.
>
> Ultimately, the reality is that the resellers all have interfaces offering
> some level of automated processing for bulk ordering. These resellers
> aren't acting as literal extra hands on behalf of the end user.
>

Many do.


> They have specialized access to bulk ordering interfaces, generally
> pursuant to a reseller contract.
>

Not always.

They're not literally hand-entering data as though they themselves were
> the end-user.
>


Some do.

It's the existence of that relationship and access which provides an
> opportunity for the community to decide "If the CA has relationships in
> this nature, the CA has an affirmative duty to ensure as a condition of the
> relationship that _______ (insert rules)".
>

Sure, and when CAs switch the relationships to not be of that nature, but
one slightly different - or when it’s the user’s agents that have failed -
we have accomplished nothing but introduce more complexity for no gain,
because the underlying problems remain. And I worry that such complexity
will limit the flexibility to solve real problems.

I don’t hear calls for suggesting that browsers should not connect if the
domain was registered through a registrar with less than stellar domain
hygeine, or refusing to connect to domains hosted by hosting providers that
may store user passwords unsalted, but that is really the degree of “basic
infrastructure” we are talking about here.

I do deeply worry that the amount of energy spent here trying to
contemplate policy on this will drain energy from the far more important
efforts, such as ensuring reliable, interoperable automation, such that the
very need to generate a CSR by hand evaporates. If we can get there, we
have solved the problems resellers are trying to solve, better, more
securely, and without having to invent paper tigers so we can waggle our
fingers when they fail.
>> > > > I'm not sure that that would be reasonable. After all resellers can
>> > have
>> > > > resellers, and so on so where would that end? With the end user
>> having
>> > to
>> > > > accept a "general license agreement"? And distrusting a reseller
>> could
>> > > also
>> > > > be difficult.
>> > > >
>> > > > I think it will have to be/remain the responsibility of the CA to
>> > choose
>> > > > their reselllers in such a way that "not too many questions are
>> being
>> > > > asked" about them.
>> > > >
>> > > >

Nick Lamb

unread,
Mar 5, 2018, 5:30:45 PM3/5/18
to dev-secur...@lists.mozilla.org, okaphone.e...@gmail.com
On Mon, 5 Mar 2018 09:29:47 -0800 (PST)
"okaphone.elektronika--- via dev-security-policy"
<dev-secur...@lists.mozilla.org> wrote:

> On Monday, 5 March 2018 18:10:17 UTC+1, okaphone.e...@gmail.com
> wrote:
> Ah, found it. It was tialaramex who suggested that this could be how
> Trustico got the private keys.
> https://www.reddit.com/r/sysadmin/comments/80uaq3/digicert_certificates_being_revoked/duyg6pn/

I wrote this comment in response to a redditor who claimed they'd
received an email about this mass revocation although they were sure
they'd used best practices in issuing a CSR.

Now that we know in fact the reseller tried to have all certificates
revoked regardless of whether they had the private keys (and DigiCert
not unreasonably balked at doing this) it is likely the redditor in
question had got an email from their reseller and their cert was not
eventually revoked.


> Just speculation then. But still worth keeping in mind as something a
> reseller could be doing. I can just see some programmer coming up
> with this idea to workaround the problem of not having the private
> key. ;-)

I'm pretty sure I have seen this sort of practice, but I don't have any
hard evidence and it may be another of the bad ideas that has died out
as the market reforms.


In terms of the larger topic of this thread, I don't think we're going
to get very far putting pressure on CAs to fix resellers for reasons
several people have already mentioned. We can however encourage three
things that will help even though they can't overnight forbid
undesirable retention of other people's keys:


1. Education. Let's make sure material from the Trust Store owners,
from CAs, and from other entities we come into contact with describes
processes that are secure by default, such as the use of CSRs. Got a
document that skips the CSR "just for the example" ? Fix that, the same
way you'd show a normal family wearing seatbelts in a car in a movie
even though obviously for the movie they might be on a sound stage so
the seatbelts do nothing.

2. Implementation. Software vendors including Trust Store owners (such
as Microsoft and Apple) have an opportunity to "bake in" secure
approaches. The easier it is to do things the safer way, the less
likely users are to look for a shortcut from a reseller. Nobody is
offering a key generation feature so as to make the sales journey more
complicated and harder to use - if "just use a CSR" was the easy
option, that's all resellers would offer.

3. Customer focused standards. Rather than try to push from the CAs,
groups like PCI get to set demand, if the PCI compliance document
explicitly says that your private keys mustn't come from somebody else
then that's another reason somebody is going to get that right. I'm
sure there are other appropriate groups that mandate SSL and could
explicitly specify this as a requirement.

Jakob Bohm

unread,
Mar 6, 2018, 7:49:52 AM3/6/18
to mozilla-dev-s...@lists.mozilla.org
How about something simple like:

(Rephrase terminology etc. as necessary)

If a CA has any arrangements with any 3rd parties to act as
intermediaries between the subscriber and the CA, while not being the
party that operates the normal uses of the private key on the
subscribers behalf, the CA must ensure that any activities by such 3rd
parties and related to certificates chaining to the CA comply fully with
the applicable BR and CPS requirements applicable to the CA performing
similar activities.

Additionally, subscribers must not be forced or encouraged to have 3rd
parties operate the normal uses of the private key on the subscribers
behalf.
Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

Eric Mill

unread,
Mar 11, 2018, 1:17:01 PM3/11/18
to mozilla-dev-s...@lists.mozilla.org
Though I'm not a GlobalSign customer, I'm told that GlobalSign sent the
following email to its partner ecosystem:

Dear Partner,

In reaction to current events related to the private key exposure and mass
revocation by Trustico/Digicert, GlobalSign is reaching out to its trusted
Partners and Resellers to survey how they approach customer private key and
CSR generation. The most secure method is to generate the keys on the
server and then use the CSR when requesting the certificate. If you do
generate private keys for any of your customers outside of the web server
environment where the certificate will be hosted, we request that you stop
this practice immediately.

We ask that all Partners and Resellers complete the following short
questionnaire as soon as possible or by: Friday, March 16, 2018.

Compliance questionnaire : [REDACTED]
Note: Only one questionnaire needs to be completed per company.

Thank you in advance for your cooperation and attention to this important
topic.

Kind regards,
GlobalSign Security and Compliance


So it's nice to see that at least one CA is taking action on this topic
without being ordered to (that I'm aware of). Obviously not all resellers
are like Trustico and perform only a straight certificate pass-through, as
Ryan Sleevi pointed out, and key escrow is a necessary part of acting as a
host, or CDN, or other authorized representative.

But surely there is still some way to responsibly look through the
ecosystem for resellers that do not perform an authorized function that
requires key escrow. Are any other CAs willing to do something like
GlobalSign has done?

Also, it would be very helpful if GlobalSign could share some information
relating to the responses they get, even if they are aggregated or
anonymized.

-- Eric
> It's also clear from the experience that rules of the road for resellers
> are unclear, and that accountability is limited. It seems possible, or
> likely, that other resellers may also be mishandling customer keys
>
> So, what would useful next steps be to improve security and accountability
> for resellers?
>
> One thought: Mozilla could ask CAs to obtain a written response from all
> contracted resellers about if/how they interact with customer key material,
> including the level of isolation/security given their key generation
> environment (if they have one), and whether any third-party JavaScript is
> given access to generated key material.
>
> Any other ideas?
>
> Also -- Comodo noted:
>
> Trustico have also confirmed to us that they were not, and are not, in
> possession of the private keys that correspond to any of the certificates
> that they have requested for their customers through Comodo CA.
>
>
> Since there appears to have been a significant overlap period, between the
> time Trustico switched to Comodo and when Trustico was asked by Comodo to
> cease key storage practices, it's a little hard to take at face value the
> assurance that Trustico was never in possession of any Comodo keys. It
> would be nice to hear something from Comodo about whether they've verified
> this in any more detail.
>
> -- Eric
>

Tim Hollebeek

unread,
Mar 12, 2018, 1:59:31 PM3/12/18
to Eric Mill, mozilla-dev-s...@lists.mozilla.org
You might be interested in the following blog post:

https://www.digicert.com/blog/keeping-subscribers-safe-partner-best-practice
s/

We are continuing to look into whether there are any additional partners or
resellers that are misbehaving. This may indeed be an area where stricter
requirements are necessary, though it's a complicated problem.

-Tim
Reply all
Reply to author
Forward
0 new messages