Suspicious test.com Cert Issued By GlobalSign

[Andrew Ayer]

I found another certificate for www.test.com that I believe was
mis-issued by GlobalSign:

https://crt.sh/?sha256=9d503e7c6c4fb6e6d7436c07ff445b95214871ea13ac1cb3b0d7abbce9be6cfb

This certificate was issued on 2015-09-11 and is not yet expired. I
was not paying close attention to mozilla.dev.security.policy back
then, but I can't find any mention of this certificate in the archives.

The certificate was revoked on 2015-09-11. It is also present in
Chrome's CRLSets, although it might have been added automatically
since it's an EV cert.

Reasons I think this certificate is mis-issued:

1. The subject organization is "GMO GlobalSign Ltd" and there are
DNS SANs for globalsign-support.com and www.globalsign-support.com.
However, test.com does not appear affiliated with GlobalSign in any way.

2. The certificate has not been detected in the wild by Censys. The
live certificate for www.test.com was issued by Network Solutions and
judging from CT, they have been using Network Solutions since at least
July 2015.

3. The same public key can also be found in the following certificate,
which has DNS SANs for globalsign-support.com and
www.globalsign-support.com but NOT www.test.com:

https://crt.sh/?sha256=7f2c6c5d4b0f0e4f1f3b41e5c3354968b1f38350fa3c24820389b566db619b01

Regards,
Andrew

[Gervase Markham]

On 25/01/17 17:36, Andrew Ayer wrote:
> I found another certificate for www.test.com that I believe was
> mis-issued by GlobalSign:
>
> https://crt.sh/?sha256=9d503e7c6c4fb6e6d7436c07ff445b95214871ea13ac1cb3b0d7abbce9be6cfb

Yes, that looks mis-issued. I realise this was some time ago now, but do
the Globalsign reps on the list have any comment?

Gerv

[Doug Beattie]

Hi Gerv,

We've researched the audit events around the certificate:

https://crt.sh/?sha256=9d503e7c6c4fb6e6d7436c07ff445b95214871ea13ac1cb3b0d7abbce9be6cfb

The domain test.com was inadvertently used in a certificate request and issuance - here are the audit events for the managed service account:

9/11/2015 11:41:20 - test.com added as a prevetted domains
9/11/2015 11:50 - Order received by CA
9/11/2015 11:51:02 - Certificate issued
9/11/2015 11:52:48 - Certificate revoked
9/11/2015 14:24:03 - test.com removed as a prevetted domain

Back in 2015, there were some GlobalSign testing in which users thought it was acceptable to use domains like test.com and example.com for testing purposes. Since this time, GlobalSign has implemented procedures to avoid any similar situations in the future. We've purchased domains like globalsign-demo.com, globalsign-support.com and aeg-test.com for testing purposes

The issuance of certificates from production CAs always uses domains which have been properly verified in accordance with the BRs and our vetting policies and the use of "testing" domains is only permitted if the domains are properly vetted in accordance with our CPS. Certainly, the reported misissuance over the past year have highlighted this to all CAs.

As part of researching this reported misissuance, we've reviewed all orders and certificates we've issued since this time to test.com and example.com and found several orders in the pending or cancelled state, but none of them were ever issued. We continue to stress the importance of proper testing within our development, QA and production environments to avoid future misissuances.

Doug

> -----Original Message-----
> From: dev-security-policy [mailto:dev-security-policy-
> bounces+doug.beattie=globals...@lists.mozilla.org] On Behalf Of Gervase
> Markham
> Sent: Thursday, January 26, 2017 4:20 AM
> To: mozilla-dev-s...@lists.mozilla.org
> Subject: Re: Suspicious test.com Cert Issued By GlobalSign
>
> On 25/01/17 17:36, Andrew Ayer wrote:

> > I found another certificate for www.test.com that I believe was
> > mis-issued by GlobalSign:
> >
> >
> >
> https://crt.sh/?sha256=9d503e7c6c4fb6e6d7436c07ff445b95214871ea13ac1c
> b
> > 3b0d7abbce9be6cfb
>

> Yes, that looks mis-issued. I realise this was some time ago now, but do the
> Globalsign reps on the list have any comment?
>
> Gerv

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy

[Nick Lamb]

Thank you for undertaking this investigation Doug and for sharing what you found. I am glad to hear that GlobalSign had taken action to make similar issuances less likely in the future even before Andrew reported this.

In hindsight probably it would have been helpful to suggest to all members of Mozilla's root programme that they consider whether they needed one or more such "test domains" as the rules on DNS name validation have gradually tightened.

The existence of lists of "prevetted domains" for managed service accounts doubtless streamlines things considerably for valuable large corporate customers, but it does open up some additional vulnerability compared to a simpler model in which everything is vetted each time. I hope GlobalSign has policies in place to mitigate that vulnerability.

[Doug Beattie]

Hi Nick,

Yes, we have controls in place that trigger domain re-vetting in accounts prior to the max allowed by the BRs to assure that domains are not used beyond the 13/39 month limits.

Doug

> -----Original Message-----
> From: dev-security-policy [mailto:dev-security-policy-
> bounces+doug.beattie=globals...@lists.mozilla.org] On Behalf Of Nick
> Lamb
> Sent: Wednesday, February 1, 2017 5:13 PM
> To: mozilla-dev-s...@lists.mozilla.org
> Subject: Re: Suspicious test.com Cert Issued By GlobalSign
>

[Gervase Markham]

On 01/02/17 19:47, Doug Beattie wrote:
> 9/11/2015 11:41:20 - test.com added as a prevetted domains

Who added this - a customer, or a GlobalSign employee?

Were customers permitted to add domains to the prevetted list in their
enterprise accounts without GlobalSign confirming that they actually own
it? Are they still?

Gerv

[Doug Beattie]

Gerv,

This was for GlobalSign account used for testing, so it was a GlobalSIgn employee. Customers are not, nor have they ever been, permitted to add domains without GlobalSign enforcing the domain verification process.

> -----Original Message-----
> From: Gervase Markham [mailto:ge...@mozilla.org]
> Sent: Wednesday, February 8, 2017 4:19 AM
> To: Doug Beattie <doug.b...@globalsign.com>; mozilla-dev-security-
> pol...@lists.mozilla.org
> Subject: Re: Suspicious test.com Cert Issued By GlobalSign
>

> On 01/02/17 19:47, Doug Beattie wrote:

> > 9/11/2015 11:41:20 - test.com added as a prevetted domains
>

[Gervase Markham]

On 13/02/17 14:34, Doug Beattie wrote:
> This was for GlobalSign account used for testing, so it was a
> GlobalSIgn employee. Customers are not, nor have they ever been,
> permitted to add domains without GlobalSign enforcing the domain
> verification process.

OK, then I'm a bit confused. You say this was a "managed service
account"; does such an account belong to a customer? If so, let's call
them company Foo.

> 9/11/2015 11:41:20 - test.com added as a prevetted domains

i.e. a GlobalSign employee added test.com to the account of company Foo.

> 9/11/2015 11:50 - Order received by CA

i.e. Company Foo places an order for a test.com cert. (You say
"received", so it didn't come from internally?)

How did Company Foo know it was OK to order such a cert? And why would
they do so?

Gerv

[Doug Beattie]

Company Foo was a GlobalSign "test" account which we set up to verify proper issuance.

> -----Original Message-----
> From: Gervase Markham [mailto:ge...@mozilla.org]
> Sent: Monday, February 13, 2017 8:57 AM
> To: Doug Beattie <doug.b...@globalsign.com>; mozilla-dev-security-
> pol...@lists.mozilla.org
> Subject: Re: Suspicious test.com Cert Issued By GlobalSign
>

> On 13/02/17 14:34, Doug Beattie wrote:

> > This was for GlobalSign account used for testing, so it was a
> > GlobalSIgn employee. Customers are not, nor have they ever been,
> > permitted to add domains without GlobalSign enforcing the domain
> > verification process.
>

> OK, then I'm a bit confused. You say this was a "managed service account";
> does such an account belong to a customer? If so, let's call them company Foo.
>

> > 9/11/2015 11:41:20 - test.com added as a prevetted domains
>

> i.e. a GlobalSign employee added test.com to the account of company Foo.
>

> > 9/11/2015 11:50 - Order received by CA
>

[Gervase Markham]

On 13/02/17 14:34, Doug Beattie wrote:

> This was for GlobalSign account used for testing, so it was a
> GlobalSIgn employee. Customers are not, nor have they ever been,
> permitted to add domains without GlobalSign enforcing the domain
> verification process.

But currently GlobalSign employees still are?

If so, can you help us understand why that's necessary? Given that you
control the domains used for testing, you should be able to set them up
to auto-pass some form of automated validation, so imposing a validation
requirement for every addition would not, at least on a superficial
understanding, lead to increased friction in testing.

Gerv

[Gervase Markham]

Hi Doug,

On 15/02/17 17:09, Gervase Markham wrote:
> But currently GlobalSign employees still are?
>
> If so, can you help us understand why that's necessary? Given that you
> control the domains used for testing, you should be able to set them up
> to auto-pass some form of automated validation, so imposing a validation
> requirement for every addition would not, at least on a superficial
> understanding, lead to increased friction in testing.

It's been quite some time since this question was posed; when might you
have a response?

Thanks,

Gerv

[douglas...@gmail.com]

Sorry, I missed the last request. As outlined above, this domain was added to this account for only a very short period of time and then it was removed, so it's no longer being used. Further, we've educated the groups involved that they must use real domains that are then properly verified in accordance with the CPS and BRs.

> Thanks,
>
> Gerv

[Gervase Markham]

Hi Doug,

On 28/02/17 12:44, douglas...@gmail.com wrote:
> Sorry, I missed the last request. As outlined above, this domain was
> added to this account for only a very short period of time and then
> it was removed, so it's no longer being used. Further, we've
> educated the groups involved that they must use real domains that are
> then properly verified in accordance with the CPS and BRs.

That's lovely, but it doesn't answer my question. Let me restate it: why
does GlobalSign believe it is necessary to give employees the power to
add arbitrary domains to accounts without going through ownership
validation?

Gerv

[Gervase Markham]

Hi Doug,

On 03/03/17 11:17, Gervase Markham wrote:
> That's lovely, but it doesn't answer my question. Let me restate it: why
> does GlobalSign believe it is necessary to give employees the power to
> add arbitrary domains to accounts without going through ownership
> validation?

You are getting various pings from me this morning :-) This question is
still outstanding, and has been for a month now.

Thanks,

Gerv

[douglas...@gmail.com]

On Thursday, March 16, 2017 at 6:59:41 AM UTC-4, Gervase Markham wrote:
> Hi Doug,
>
> On 03/03/17 11:17, Gervase Markham wrote:
> > That's lovely, but it doesn't answer my question. Let me restate it: why
> > does GlobalSign believe it is necessary to give employees the power to
> > add arbitrary domains to accounts without going through ownership
> > validation?
>

Hi Gerv,

For the record, we don't think it's necessary (or permissible) to give employees (RAs) the power to add arbitrary domains to accounts without proper vetting.

This was a breakdown in the vetting process whereby this "test" domain was added in order to issue a certificate in production. When this was done the cert was revoked and the vetting for the domain was disabled.

After this happened back in 2015 all of the RAs were instructed to follow production vetting procedures in production (obviously) and to not bend or break them when doing "testing".

[Gervase Markham]

On 16/03/17 11:25, douglas...@gmail.com wrote:
> For the record, we don't think it's necessary (or permissible) to
> give employees (RAs) the power to add arbitrary domains to accounts
> without proper vetting.

I guess I'm still not being clear - sorry :-( Let me try one more time:

Why does GlobalSign believe it is necessary for employees to have the
technical capability to add arbitrary domains to accounts without going
through ownership validation?

I mean, clearly they did back in 2015, because that's exactly what
happened. Do they still have the technical capability (ignoring policy
and set procedures for a moment) or not?

Gerv

[douglas...@gmail.com]

On Thursday, March 16, 2017 at 11:49:44 AM UTC-4, Gervase Markham wrote:

> Why does GlobalSign believe it is necessary for employees to have the
> technical capability to add arbitrary domains to accounts without going
> through ownership validation?
>
> I mean, clearly they did back in 2015, because that's exactly what
> happened. Do they still have the technical capability (ignoring policy
> and set procedures for a moment) or not?

Yes, RAs (trusted role employees) need to have the technical ability to manually add domains to accounts. They can verify domains in one of the 10 different methods and some of those involve manually looking in who-is for registrant info, using a DAD or in calling the contact. When one of these is used, we collect the vetting data then the RA can add/approve that domain.

> Gerv

[Nick Lamb]

This makes sense. How I imagine this in my head is if Mozilla were to come to you asking about some particular certificate issued by GlobalSign you'd always have records showing either that some sort of mechanical process validated the Subject FQDNs, or that a specific human individual (employee of GlobalSign) added something like a scan of a physical document which was the document used to validate. Is that roughly correct?

[Nio]

>Back in 2015, there were some GlobalSign testing in which users thought it was acceptable to use domains like test.com and example.com for testing purposes. Since this time, GlobalSign has implemented procedures to avoid any similar situations in the future.

Does it mean that GlobalSign allow RAs (trusted role employees) to add domain without necessary domain validation? Isn’t there any technical restriction that prevent this if the domain has not been validated, like having two different steps, and domain validation is the former step which cannot be overridden, than “adding a domain”?

>As part of researching this reported misissuance, we've reviewed all orders and certificates we've issued since this time to test.com and example.com and found several orders in the pending or cancelled state.

Is there any other domain that was thought to be allowed for testing purposes?

Nio

[Nio]

[Gervase Markham]

On 16/03/17 17:20, douglas...@gmail.com wrote:
> Yes, RAs (trusted role employees) need to have the technical ability
> to manually add domains to accounts. They can verify domains in one
> of the 10 different methods and some of those involve manually
> looking in who-is for registrant info, using a DAD or in calling the
> contact. When one of these is used, we collect the vetting data then
> the RA can add/approve that domain.

But is the addition of the domain gated on the
uploading/attachment/submission of what could plausibly be vetting data?

I mean, I understand you can't programmatically check that a person has
made a phone call. But you can require them to write a report of the
results of that phone call and not allow addition of the domain until
they've done it. Yes, they could just put "flibbertigibbet" into the
text box, but that at least shows they are deliberately bypassing the
process.

If the addition is so gated, what did the employee in this case do? Did
they upload bogus data?

Gerv

[douglas...@gmail.com]

On Friday, March 17, 2017 at 5:37:38 AM UTC-4, Gervase Markham wrote:

> On 16/03/17 17:20, douglas beattie wrote:
> > Yes, RAs (trusted role employees) need to have the technical ability
> > to manually add domains to accounts. They can verify domains in one
> > of the 10 different methods and some of those involve manually
> > looking in who-is for registrant info, using a DAD or in calling the
> > contact. When one of these is used, we collect the vetting data then
> > the RA can add/approve that domain.
>
> But is the addition of the domain gated on the
> uploading/attachment/submission of what could plausibly be vetting data?

It's gated on the RA following the correct process which is uploading the documents in one system and then approving the domain in a different system.

> I mean, I understand you can't programmatically check that a person has
> made a phone call. But you can require them to write a report of the
> results of that phone call and not allow addition of the domain until
> they've done it. Yes, they could just put "flibbertigibbet" into the
> text box, but that at least shows they are deliberately bypassing the
> process.
>
> If the addition is so gated, what did the employee in this case do? Did
> they upload bogus data?

No bogus data was uploaded.

Doug

[okaphone.e...@gmail.com]

The suspense is killing... What "non bogus" data was uploaded then? Can't have been any "plausible vetting data" can it?

CU Hans

[ta...@symantec.com]

> > > If the addition is so gated, what did the employee in this case do? Did
> > > they upload bogus data?
> >
> > No bogus data was uploaded.
> >
> > Doug
>
> The suspense is killing... What "non bogus" data was uploaded then? Can't have been any "plausible vetting data" can it?

I'm also interested. Form validation demands that it not be entirely insanely random data; certain constraints exist that mean when entering information (such as dropdowns for states, etc) that will require someone to make up plausible data. Can we call it "plausible data" instead of "bogus data"?

[Gervase Markham]

On 17/03/17 16:28, douglas...@gmail.com wrote:
>> If the addition is so gated, what did the employee in this case do? Did
>> they upload bogus data?
>
> No bogus data was uploaded.

I spoke about this with Doug at the CAB Forum meeting. The system which
collects the data is not integrated with the system to which the domains
are added. The validation specialist concerned, contrary to policy
("it's just a test"), simply did not add any data to the data recording
system relating to the addition of this domain to the authorized list
for the account.

This raises the question of whether Mozilla should attempt to require
such linkage by policy, so that domains cannot be added unless domain
validation has been performed. It would not be a totally unprecendented
mandate (at the moment we e.g. require 2-factor auth, which is a direct
mandate for how CA systems operate). However, not all methods of domain
validation are automated. Given that if someone wants to work around the
system, it's not really possible to programmatically check that e.g.
someone's write-up of a phone conversation is genuine or not, I am
minded not to attempt this.

Comments?

Gerv

[Nick Lamb]

On Friday, 24 March 2017 10:11:36 UTC, Gervase Markham wrote:
> I spoke about this with Doug at the CAB Forum meeting. The system which
> collects the data is not integrated with the system to which the domains
> are added. The validation specialist concerned, contrary to policy
> ("it's just a test"), simply did not add any data to the data recording
> system relating to the addition of this domain to the authorized list
> for the account.
>
> This raises the question of whether Mozilla should attempt to require
> such linkage by policy

I agree with you Gerv that it seems impractical to fix this with Mozilla policy.

There are a number of things CAs can and should be doing here to reduce the chance that they're put in the same position as GlobalSign, but I don't know if any of them are amenable to being specified by policy.

Good software engineering principles can help. There should be a simple, documented process for testing things, including certificate issuance. If there are ad hoc processes, it's really easy for those to end up infringing policy. But I can't see Mozilla legislating some particular software engineering process.

Good HR can help too. "Don't walk past" rules are appropriate, every employee is entitled to identify something as a security problem, and to see it get solved, covering up problems has to be what gets you in trouble, not telling people about them. But this needs morale, and it's just not practical for Mozilla to control the morale at a programme member CA.

Finally though, and maybe there is an opportunity to write policy here but I'm not sure how, the CA needs to be proactively monitoring its own systems for anomalies and outliers. The test.com certificate was an outlier because there was no data in the data recording system about it. That had a perfectly sensible, but BR defying, explanation. It doesn't need a team of people, or even a whole full-time employee, but _somebody_ at the CA should be looking for this type of problem so they find it before everybody else does.