GlobalSign BR violation

[Roland Bracewell Shoemaker]

It appears GlobalSign has issued an EV certificate containing dNSNames
which include spaces which are non-valid DNS characters. This is a
violation of CABF Baseline Regulations Sections 7.1.4.2.1. and
presumably 3.2.2.4. since there is no way to confirm control of a
non-valid DNS name.

Pre-certificate:
https://crt.sh/?q=2d935bf09230c5ba9552c4ac5f0e6dd85e44fa2755819ade9a6f54beff7555de
Certificate:
https://crt.sh/?q=7b64ea5a8f0572c99e63cc36939163ff80ea9cd62d03d1fa661aeb0627ef8633

[Matt Palmer]

On Sat, Feb 25, 2017 at 11:22:18AM -0800, Roland Bracewell Shoemaker via dev-security-policy wrote:
> It appears GlobalSign has issued an EV certificate containing dNSNames
> which include spaces which are non-valid DNS characters. This is a
> violation of CABF Baseline Regulations Sections 7.1.4.2.1. and
> presumably 3.2.2.4. since there is no way to confirm control of a
> non-valid DNS name.

While this is certainly an extremely facepalm-worthy issuance, it's almost
certainly not a DCV failure, because the domain for which control was
validated is almost certainly the eTLD+1 (`vietnamairlines.com`), and not
the FQDN in the sAN.

Still... oy gevalt. Also, `cablint` already picks this up
(https://crt.sh/?id=10570720&opt=cablint), so yeah...

- Matt

[Itzhak Daniel]

How those lines are parsed? what happens when a client reaches a whitespace? Will this allow 'vietnamairlines.com' to use 'owa', 'mail' and 'autodiscover' in their internal infrastructure?

[Nick Lamb]

On Monday, 27 February 2017 00:53:46 UTC, Itzhak Daniel wrote:
> How those lines are parsed? what happens when a client reaches a whitespace? Will this allow 'vietnamairlines.com' to use 'owa', 'mail' and 'autodiscover' in their internal infrastructure?

Because they're dnsNames a correctly implemented TLS client needn't "parse" them further at all, either they are an exact case-insensitive match for the server's DNS name or they aren't.

On the other hand apparently we can't even rely on a CA to get this right, so who knows if any clients get it wrong.

The naming on this certificate suggests it was requested for use with Microsoft's Exchange product, so assuming Microsoft's Internet Explorer / Edge web browser, and their Outlook mail client get this right the certificate was probably simply useless as issued. It /is/ interesting that the subscriber had a previous certificate for the valid set of names, and today the https://owa.vietnamairlines.com/ server (OWA stands for "Outlook Web Access" ie it's a web UI for the mail service) is serving a wildcard from someone else, so perhaps the subscriber concluded GlobalSign were hopeless and stopped using them? I hope they got their money back.

[Jakob Bohm]

On 27/02/2017 01:53, Itzhak Daniel wrote:
> How those lines are parsed? what happens when a client reaches a whitespace? Will this allow 'vietnamairlines.com' to use 'owa', 'mail' and 'autodiscover' in their internal infrastructure?
>

Programs don't parse the text lines from the crt.sh website. They
parse the clearly delimited binary (DER encoded) certificate.

Looking at a more detailed dump of the actual certificate shows that
those are indeed spaces and not a parsing error by crt.sh though.

I don't know if owa.m.vietnamairlines.com ever existed, but I happen to
know that GlobalSign provides the 3 exchange-related DNS SANs at no
extra charge when ordering an EV certificate, so maybe those invalid
DNS SAN entries were simply never used or discovered.

From all of this, it looks like a technical processing error when
generating the SAN attribute, as the primary name
(m.vietnamairlines.com) was emitted correctly. The certificate is
*currently* used by https://m.vietnamairlines.com and may never have
been intended (by the airline) for use with the 3 exchange names that
were incorrectly encoded (such as owa.m.vietnamairlines.com, not
owa.vietnamairlines.com).

But really, GlobalSign should reach out to the airline and reissue the
certificate without the stray space characters, then revoke the broken
cert after the webserver has switched (within reasonable time).

GlobalSign should also check if other EV certificates were issued with
the same processing bug and arrange to have any such certificates
similarly fixed.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

[douglas...@gmail.com]

On Monday, February 27, 2017 at 4:05:09 PM UTC-5, Jakob Bohm wrote:
> On 27/02/2017 01:53, Itzhak Daniel wrote:
> > How those lines are parsed? what happens when a client reaches a whitespace? Will this allow 'vietnamairlines.com' to use 'owa', 'mail' and 'autodiscover' in their internal infrastructure?
> >
>
> Programs don't parse the text lines from the crt.sh website. They
> parse the clearly delimited binary (DER encoded) certificate.
>
> Looking at a more detailed dump of the actual certificate shows that
> those are indeed spaces and not a parsing error by crt.sh though.
>
> I don't know if owa.m.vietnamairlines.com ever existed, but I happen to
> know that GlobalSign provides the 3 exchange-related DNS SANs at no
> extra charge when ordering an EV certificate, so maybe those invalid
> DNS SAN entries were simply never used or discovered.
>
> From all of this, it looks like a technical processing error when
> generating the SAN attribute, as the primary name
> (m.vietnamairlines.com) was emitted correctly. The certificate is
> *currently* used by https://m.vietnamairlines.com and may never have
> been intended (by the airline) for use with the 3 exchange names that
> were incorrectly encoded (such as owa.m.vietnamairlines.com, not
> owa.vietnamairlines.com).
>
> But really, GlobalSign should reach out to the airline and reissue the
> certificate without the stray space characters, then revoke the broken
> cert after the webserver has switched (within reasonable time).

Yes, we're working to do just this now.

[Ryan Sleevi]

On Tue, Feb 28, 2017 at 8:53 AM, douglas.beattie--- via dev-security-policy
<dev-secur...@lists.mozilla.org> wrote:

>
> Yes, we're working to do just this now.

While that's good and well, I do hope GlobalSign will produce an incident
report regarding this matter, as to how the situation in
https://groups.google.com/d/msg/mozilla.dev.security.policy/luxlU5TL2ew/qkL1ZdThAQAJ
came to be in the first place.

I think GoDaddy's recent explanation -
https://groups.google.com/d/msg/mozilla.dev.security.policy/Htujoyq-pO8/uRBcS2TmBQAJ
- may provide a model for GlobalSign here.

The intent is that we don't just deal with the symptom, but that we work to
understand the root cause, the scope of impact, and the opportunities for
improvement.

I look forward to reading GlobalSign's analysis of both this and other
recent issues.

[douglas...@gmail.com]

Ryan,

GlobalSign certificate issuance has been referenced in several different threads recently and I think most of them are closed; however, if you feel otherwise, let me know.

Suspicious Test certificate
https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/-gaS1p3vrXc
I provided a formal response in that thread that I believe closes this issue.

Incapusla certifciates with SANs for domains that don't currently exist
https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/l1mDdCL8LlU
This is not an issue as all SANs were validated in accordance with the BRs when the domain was verified, thus no formal response is needed.

And lastly this ticket. The Domain name was validated in accordance with the BRs, but there was a bug that allowed a user entered space to be included in some of the SAN values. While the value is not compliant with RFC 5280 or the BRs, there was no security issue with the certificate that was issued (it was likely not able to secure the intended subdomains). We'll provide an incident report for this.

If this isn't sufficient for some reason, I'm sure you will let us know.

Doug

[Ryan Sleevi]

On Tue, Feb 28, 2017 at 12:02 PM, douglas.beattie--- via
dev-security-policy <dev-secur...@lists.mozilla.org> wrote:

> Ryan,
>
> GlobalSign certificate issuance has been referenced in several different
> threads recently and I think most of them are closed; however, if you feel
> otherwise, let me know.
>

Hi Doug,

Right, I realize there were several threads - You've addressed some of the
scenarios for both Incapsula and the test certificates - however, I haven't
seen an explanation as to how the spaces were introduced into these SANs,
the scope of how many GlobalSign certs this affected, how long the duration
of affect was, and what GlobalSign is doing to correct that.

While I understand you plan to reach out to Vietnam Airlines regarding this
specific cert, it's understanding both the root cause and the steps
GlobalSign is taking to redress those that I think are relevant here.

> And lastly this ticket. The Domain name was validated in accordance with
> the BRs, but there was a bug that allowed a user entered space to be
> included in some of the SAN values. While the value is not compliant with
> RFC 5280 or the BRs, there was no security issue with the certificate that
> was issued (it was likely not able to secure the intended subdomains).
> We'll provide an incident report for this.
>
> If this isn't sufficient for some reason, I'm sure you will let us know.

Right, I think an incident report on this would be useful. I think I would
be quite cautious to suggest "there is no security issue with the
certificate that was issued" - I think many a CA would have said that about
encoding, say, a null byte (\0) within a SAN, prior to realizing the issues.

For example, as a systemic issue, it seems this suggests that GlobalSign
does not validate what appears in the SAN, so long as the validated domain
appears within it. This could range from a SERIOUS security issue (for
example, if GlobalSign's systems are themselves not robust against NULL
bytes) to a benign one. Understanding the root cause, scope, and
remediation plans is useful here to assure the relying parties of
GlobalSign's committment to security.

[Gervase Markham]

On 28/02/17 20:02, douglas...@gmail.com wrote:
> Suspicious Test certificate
> https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/-gaS1p3vrXc
>
> I provided a formal response in that thread that I believe closes
> this issue.

I still have an outstanding question.

> And lastly this ticket. The Domain name was validated in accordance
> with the BRs, but there was a bug that allowed a user entered space
> to be included in some of the SAN values. While the value is not
> compliant with RFC 5280 or the BRs, there was no security issue with
> the certificate that was issued (it was likely not able to secure the
> intended subdomains). We'll provide an incident report for this.

Lovely. :-)

Gerv

[Gervase Markham]

On 28/02/17 20:02, douglas...@gmail.com wrote:

> And lastly this ticket. The Domain name was validated in accordance
> with the BRs, but there was a bug that allowed a user entered space
> to be included in some of the SAN values. While the value is not
> compliant with RFC 5280 or the BRs, there was no security issue with
> the certificate that was issued (it was likely not able to secure the
> intended subdomains). We'll provide an incident report for this.

Hi Doug,

Any news on when we might see this incident report?

Thanks,

Gerv

[Doug Beattie]

Hi Gerv,

Here is the incident report for this reported issue.

Doug

> -----Original Message-----
> From: dev-security-policy [mailto:dev-security-policy-
> bounces+doug.beattie=globals...@lists.mozilla.org] On Behalf Of Gervase
> Markham via dev-security-policy
> Sent: Thursday, March 16, 2017 6:57 AM
> To: D B <douglas...@gmail.com>; mozilla-dev-security-
> pol...@lists.mozilla.org
> Subject: Re: GlobalSign BR violation
>
> On 28/02/17 20:02, douglas...@gmail.com wrote:

> > And lastly this ticket. The Domain name was validated in accordance
> > with the BRs, but there was a bug that allowed a user entered space to
> > be included in some of the SAN values. While the value is not
> > compliant with RFC 5280 or the BRs, there was no security issue with
> > the certificate that was issued (it was likely not able to secure the
> > intended subdomains). We'll provide an incident report for this.
>

> Hi Doug,
>
> Any news on when we might see this incident report?
>
> Thanks,
>
> Gerv
>

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy

[dbo...@fastmail.com]

On Tuesday, April 4, 2017 at 8:19:28 AM UTC-7, Doug Beattie wrote:
> Here is the incident report for this reported issue.

I don't see anything attached or linked?

[douglas...@gmail.com]

Attachment was stripped, here it the content:

GlobalSign BR violation: EV Certificate with dNSName containing a space

On February 26, 2017, we received a report that there were multiple SANs in an EV SSL Certificate that contained a space within it. Spaces are not permitted characters, per RFC 5280. We notified the customer on March 1, 2017 and revoked the certificate the next day on March 2, 2017.

How this happened:

A customer ordered a certificate with CN of m.vietnamairlines.com and requested 3 additional SANs as listed below. The base domain, “vietnamairlines.com”, was vetted and the certificate was then approved and issued. When they ordered this 2 year certificate in August 2015 they entered a space (likely via copy/paste) in the SAN fields and this was not obvious the vetting team when they were reviewing the request. Our ordering application did not properly strip out the spaces and the vetting team did not notice the space in the middle of these SANs. The result was this set of invalid SAN values:
* owa. m.vietnamairlines.com
* mail. m.vietnamairlines.com
* autodiscover. m.vietnamairlines.com

Scope of the problem:

As part of this problem investigation, we went back and searched for DNSNames with characters that don’t comply with RFC 5280 in active certificates. We found 11 orders (each with 1 active certificate) that contained values non-compliant with the RFC and we found one order with 34 active certificates (due to many certificate updates/reissuances for this order). All have been revoked.

In February 2016, we put in more comprehensive data validation for SANs and CNs and have not issued any non-compliant dNSName values in SSL Certificates since that time, until last week on March 21, 2017. We issued a Wildcard OV SSL Certificate with a SAN of the format www.*.domain.com. It has been since revoked. The issue here is that the updated logic released in February 2016 was not applied to a certain case for a certain language specific validation routines (we have unique server side validation routines for different languages and one was missed).

Plans for Remediation:

GlobalSign is continuing to improve automated compliance checking services to catch issues prior to issuance in a centralized location as a standard compliance process that all certificates will need to pass. In addition to checking data when we receive new orders, we will be adding independent, redundant checks as follows:

* By the end of May 2017, we’ll be adding an independent post issuance check for 100% of issued SSL Certificates. This centralized check can be more robust and will proactively detect errors so we can address them immediately.
* By the end of July 2017, we’ll be adding the same independent pre-issuance check in-line with the issuance process at the very last step prior to issuance to check issues before issuance.

We feel that adding these 2 additional levels of checking we will eliminate issuance of certificates with CommonName and SAN values that do not comply with RFC 5280. We plan to add redundant pre- and post-issuance validation for additional fields later this year.

[Gervase Markham]

On 04/04/17 16:31, douglas...@gmail.com wrote:
> Attachment was stripped, here it the content:

Thanks Doug.

Unless anyone sees something particularly problematic here, I think we
can call this incident closed.

Gerv

[Nick Lamb]

On Tuesday, 4 April 2017 16:31:10 UTC+1, douglas...@gmail.com wrote:
> How this happened:

Thanks Doug,

I have a question: These certificates appear to be not only forbidden by the BRs but also technically unlikely to function as desired by the subscriber. Did any customers report problems which (perhaps in hindsight now that the problem is understood at GlobalSign) show they'd noticed the certificates they obtained did not work ?

[Doug Beattie]

> -----Original Message-----
> From: dev-security-policy [mailto:dev-security-policy-

> bounces+doug.beattie=globals...@lists.mozilla.org] On Behalf Of Nick
> Lamb via dev-security-policy
>
> I have a question: These certificates appear to be not only forbidden by the BRs
> but also technically unlikely to function as desired by the subscriber. Did any
> customers report problems which (perhaps in hindsight now that the problem is
> understood at GlobalSign) show they'd noticed the certificates they obtained
> did not work ?

I'm not aware of any communications to us about certificates they received but didn't work. If they did call support then I would have assumed the order would have been cancelled/revoked or otherwise updated to fix the problem, but none of that happened. All I can assume is that they didn't actually use it to secure those SANs and only used it on the CN.

[Jakob Bohm]

Just a tip: How about the account with 34 reissues, this may have been
a failed attempt to self-service the bad certificate by requesting
reissue when it didn't work? Maybe you need to check your history with
this customer to see if some kind of reach out would be appropriate
from a customer service perspective.