Proposed limited exception to SHA-1 issuance

2369 views
Skip to first unread message

Gervase Markham

unread,
Feb 23, 2016, 1:58:19 PM2/23/16
to mozilla-dev-s...@lists.mozilla.org, Kathleen Wilson, Richard Barnes
Mozilla and other browsers have been approached by Worldpay, a large
payment processor, via Symantec, their CA. They have been transitioning
to SHA-2 but due to an oversight have failed to do so in time for a
portion of their infrastructure, and failed to renew some SHA-1 server
certificates before the issuance deadline of 31st December 2015.

They now need 9 SHA-1 certificates issued before 28th February 2015, or
approximately 10,000+ payment terminals around the world will stop
working. This equipment was created some time ago, and unfortunately
only supports publicly-trusted roots. Using roots removed from browser
root programs is also not a complete solution to the program; these
10,000 do not trust any of those roots. This equipment does not support
SHA-256 and cannot be replaced in time. The data travels over the public
internet but the servers are not accessed by browsers. Due to the short
timelines involved, a change in the BRs by the CAB Forum is also not
possible. Therefore, they are seeking to get browser acknowledgement
that a qualified audit, qualified by the existence of these certs, will
be acceptable.

The payment industry is moving towards SHA-256 but their timeline does
not line up with the CAB Forum one. Our understanding is that Worldpay
is not the only payment processor in this position. (We are not sure how
to match this information with Worldpay's assertion that this was an
oversight on their part, unless such oversights are unusually common at
payment processors.)

Our proposal, which we have sent to Symantec, Worldpay and the other
browsers, is as follows:

Symantec may issue certificates to Worldpay if the following things are
true:

1. You immediately give copies to Mozilla (and other vendors who desire
them) for us to immediately add them to OneCRL, as if they had been
mis-issued.

2. Symantec's OCSP server MUST present a response of Revoked to any
request for these certificates from, at a minimum, Firefox (based on
User-Agent). Other browsers may wish to be added to this list. Sending
Revoked to everyone would be easiest, but that depends on your testing
as to whether it will break the intended clients.

3. Certificates issued under this exception MUST be logged to CT, and
Symantec MUST disclose which logs they will be published in.

4. On issuance of any such certificate(s), the issuer MUST send mail to
cabfpub announcing the event, including references to the CT entries.

5. The auditor's qualification MUST actively attest that the extent of
SHA-1 issuance is no greater than that disclosed in CT. (Otherwise the
qualification will be deemed unacceptable.)

6. The lifetime of the issued SHA-1 certificates MUST be no more than 90
days. Reissuance is permitted, but Mozilla reserves the right to decide
in the future that the conditions for further issuance of such
certificates may vary, including deeming them unacceptable under any
circumstances. Mozilla is very likely to not permit validity to extend
beyond the SHA-1 deadline of 31st December 2016.

7. This exception applies to Worldpay only; you need to come back and
ask, presenting the circumstances, for other cases. If the impact is
similar, similar terms may be extended.


Mozilla is very keen to see SHA-1 eliminated, but understands that for
historical reasons poor decisions were made in private PKIs about which
roots to trust, and such decisions are not easily remedied.

Please comment on whether this proposal seems reasonable, being aware of
the short timelines involved.

Gerv

Andrew Ayer

unread,
Feb 23, 2016, 3:06:30 PM2/23/16
to mozilla-dev-s...@lists.mozilla.org, Gervase Markham, Kathleen Wilson, Richard Barnes
On Tue, 23 Feb 2016 18:57:41 +0000
Gervase Markham <ge...@mozilla.org> wrote:

> Please comment on whether this proposal seems reasonable, being aware
> of the short timelines involved.

I am opposed. There is no telling how many other organizations are in a
similar situation due to poor planning or "oversights" on their part,
and who will also want special treatment. Granting this exception will
set an expectation that exceptions will be granted in the future and
therefore that deadlines and deprecations need not be taken seriously.
That would have a negative effect on efforts to move the security of
the Internet forward.

Multiple mistakes were made by Worldpay (using public roots, leaving
the transition to the last minute, and then forgetting to renew before
the sunset) and Symantec (failing to make sure their customer was
prepared). They had ample opportunity to avoid a crisis. It is not
Mozilla's responsibility to dig them out of the hole they have dug for
themselves, and doing so is contrary to Mozilla's interest in keeping
the Internet secure.

Additionally, none of the stipulations in the proposal mitigate the
risk of SHA-1 issuance. Disclosure and revocation do no good if an
undisclosed, unrevoked certificate (possibly with CA:TRUE) can be
collided with the disclosed and revoked certificate.

Regards,
Andrew

Yuhong Bao

unread,
Feb 23, 2016, 4:13:00 PM2/23/16
to Andrew Ayer, mozilla-dev-s...@lists.mozilla.org, Gervase Markham, Kathleen Wilson, Richard Barnes

If OneCRL always used the same hash algorithm as the certificate, then any colliding certificate would also be treated as revoked.

Charles Reiss

unread,
Feb 23, 2016, 4:45:03 PM2/23/16
to mozilla-dev-s...@lists.mozilla.org
On 02/23/16 18:57, Gervase Markham wrote:
[snip]
> Symantec may issue certificates to Worldpay if the following things are
> true:

Based on what's happened with MD5 certificates, it seems the main risk
of harm comes from something like a chosen-prefix collision attack using
a specially constructed CSR. In this case, the serial number of the
resulting colliding certificate can be forged, so Mozilla's
revocation/OneCRL requirement wouldn't seem to do much unless the OneCRL
entries are keyed on the tbsCertificate SHA1 hash or similar (and not
the issuer+serial number or the whole certificate hash).

If Mozilla is to allow this SHA1 issuance, it should also consider
requiring steps to limit the impact of such attacks, such as one or more of:

- Issuing the certificates from a subCA with a pathlen constraint that
prevents that subCA from signing subsubCA certificates, which I gather
is the already the case for most (all?) of Symantec's subCAs that
directly issue TLS server certificates.

- Including >80 bits of entropy in the serial numbers of these
certificates. (The BRs recommend but do not require 20 bits, and
Symantec does not follow this recommendation under some of their subCAs:
https://crt.sh/?cablint=38&iCAID=1559)

- Issuing the certificates via an internally operated (SHA-1) subCA that
is technically constrained within the meaning of Mozilla's policy.

Andrew Ayer

unread,
Feb 23, 2016, 4:48:05 PM2/23/16
to Yuhong Bao, mozilla-dev-s...@lists.mozilla.org
On Tue, 23 Feb 2016 13:12:27 -0800
Yuhong Bao <yuhong...@hotmail.com> wrote:

> If OneCRL always used the same hash algorithm as the certificate,
> then any colliding certificate would also be treated as revoked.

OneCRL would need to use the hash of the TBS, not the certificate. The
TBS is what's collided, but once the signature is added, the hashes are
not identical anymore (unless the length of the TBS happens to be a
multiple of the SHA-1 block size).

Andrew

David E. Ross

unread,
Feb 23, 2016, 4:56:25 PM2/23/16
to mozilla-dev-s...@lists.mozilla.org
Overall, I am opposed to accommodating negligence, which is what is
proposed here. If such accommodation must be made nevertheless, the
"not later" dates of the affected certificates should be not later than
60 days after their "not before" dates. Surely, 60 days should be
sufficient to comply with eliminating SHA-1 certificates.

--
David E. Ross

While many tributes to the late Supreme Court Associate Justice
Antonin Scalia now fill the news media, his legacy was not
necessarily positive. See my "What Price Order, Mr. Justice Scalia?"
at <http://www.rossde.com/editorials/edtl_scalia_wrong.html>.

tech...@gmail.com

unread,
Feb 23, 2016, 5:32:53 PM2/23/16
to mozilla-dev-s...@lists.mozilla.org
Gerv -- I was at the same meeting as you and heard the same request from these financial service companies. You (and others) are correct that they should have started preparing to deal with this earlier, but in the real world, people and companies make forgivable mistakes.

One comment stuck with me -- they said if they had realized their problem in time (i.e., in December 2015) they could have gotten one year replacement SHA-1 certificates and had another 12 months to solve their problem. Because they didn't figure out their problem in time, now they only have days or weeks until their current SHA-1 certs expire. I think it's right to give them the extra

I think your proposal is the right thing to do -- even though Trend Micro has no similar customers and will not be issuing SHA-1 certs to anyone under this proposal. The safeguards you are implementing should be sufficient to protect users, and this will allow thousands of terminals used by consumers to stay in operation and serve their needs.

I think we all need to remember what's important - the security and the needs of users. You have done a good job of balancing needs in your plan, and Trend Micro commends you for it.

Kirk Hall

Richard Barnes

unread,
Feb 23, 2016, 5:54:15 PM2/23/16
to Charles Reiss, mozilla-dev-s...@lists.mozilla.org
On Tue, Feb 23, 2016 at 1:44 PM, Charles Reiss <wogg...@gmail.com> wrote:

> On 02/23/16 18:57, Gervase Markham wrote:
> [snip]
> > Symantec may issue certificates to Worldpay if the following things are
> > true:
>
> Based on what's happened with MD5 certificates, it seems the main risk
> of harm comes from something like a chosen-prefix collision attack using
> a specially constructed CSR. In this case, the serial number of the
> resulting colliding certificate can be forged, so Mozilla's
> revocation/OneCRL requirement wouldn't seem to do much unless the OneCRL
> entries are keyed on the tbsCertificate SHA1 hash or similar (and not
> the issuer+serial number or the whole certificate hash).
>
> If Mozilla is to allow this SHA1 issuance, it should also consider
> requiring steps to limit the impact of such attacks, such as one or more
> of:
>
> - Issuing the certificates from a subCA with a pathlen constraint that
> prevents that subCA from signing subsubCA certificates, which I gather
> is the already the case for most (all?) of Symantec's subCAs that
> directly issue TLS server certificates.
>

I don't think it makes sense to require Symantec to stand up a new subCA,
given the time horizon. However, it would be good to see if they could use
an existing subCA that is constrained in this way.



> - Including >80 bits of entropy in the serial numbers of these
> certificates. (The BRs recommend but do not require 20 bits, and
> Symantec does not follow this recommendation under some of their subCAs:
> https://crt.sh/?cablint=38&iCAID=1559)
>

This is an excellent idea.



> - Issuing the certificates via an internally operated (SHA-1) subCA that
> is technically constrained within the meaning of Mozilla's policy.
>

As above, I don't think making a new subCA makes sense. This could be a
reasonable approach for longer-range issues, though, should any arise.

--Richard



>
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

Richard Barnes

unread,
Feb 23, 2016, 5:54:16 PM2/23/16
to Andrew Ayer, Yuhong Bao, mozilla-dev-s...@lists.mozilla.org
Unfortunately, given the latency of deploying code to Firefox, it won't be
possible to make any changes to OneCRL in this case.

However, OneCRL is based on issuer and serial number, so forged
certificates will not be able to use the serial number in a chosen-prefix
attack.



>
> Andrew

Richard Barnes

unread,
Feb 23, 2016, 5:55:54 PM2/23/16
to David E. Ross, mozilla-dev-s...@lists.mozilla.org
On Tue, Feb 23, 2016 at 1:55 PM, David E. Ross <nob...@nowhere.invalid>
wrote:

> On 2/23/2016 10:57 AM, Gervase Markham wrote:
> Overall, I am opposed to accommodating negligence, which is what is
> proposed here. If such accommodation must be made nevertheless, the
> "not later" dates of the affected certificates should be not later than
> 60 days after their "not before" dates. Surely, 60 days should be
> sufficient to comply with eliminating SHA-1 certificates.
>

Hi David,

I certainly agree that we want to continue to apply upgrade pressure. The
plan above proposes 90 days, which we chose in part because there is a fair
bit of deployment experience in the web with certs of that length. (For
example, https://google.com uses 90-day certs.) Do you think there's a
material difference between 60 days and 90 days?

Thanks,
--Richard



>
> --
> David E. Ross
>
> While many tributes to the late Supreme Court Associate Justice
> Antonin Scalia now fill the news media, his legacy was not
> necessarily positive. See my "What Price Order, Mr. Justice Scalia?"
> at <http://www.rossde.com/editorials/edtl_scalia_wrong.html>.

Richard Barnes

unread,
Feb 23, 2016, 6:42:30 PM2/23/16
to Andrew Ayer, Gervase Markham, mozilla-dev-s...@lists.mozilla.org, Kathleen Wilson
On Tue, Feb 23, 2016 at 12:05 PM, Andrew Ayer <ag...@andrewayer.name> wrote:

> On Tue, 23 Feb 2016 18:57:41 +0000
> Gervase Markham <ge...@mozilla.org> wrote:
>
> > Please comment on whether this proposal seems reasonable, being aware
> > of the short timelines involved.
>
> I am opposed. There is no telling how many other organizations are in a
> similar situation due to poor planning or "oversights" on their part,
> and who will also want special treatment. Granting this exception will
> set an expectation that exceptions will be granted in the future and
> therefore that deadlines and deprecations need not be taken seriously.
> That would have a negative effect on efforts to move the security of
> the Internet forward.
>
> Multiple mistakes were made by Worldpay (using public roots, leaving
> the transition to the last minute, and then forgetting to renew before
> the sunset) and Symantec (failing to make sure their customer was
> prepared). They had ample opportunity to avoid a crisis. It is not
> Mozilla's responsibility to dig them out of the hole they have dug for
> themselves, and doing so is contrary to Mozilla's interest in keeping
> the Internet secure.
>
> Additionally, none of the stipulations in the proposal mitigate the
> risk of SHA-1 issuance. Disclosure and revocation do no good if an
> undisclosed, unrevoked certificate (possibly with CA:TRUE) can be
> collided with the disclosed and revoked certificate.
>

You're correct that disclosure and revocation requirements don't directly
address issues of collisions. In fact, the lifetime constraint could
exacerbate the issue by causing more certs to be issued if the transition
takes too long. However, I think there are a few factors that mitigate
risk here.

First, we should keep in mind that all known attacks on the PKI (AFAIK) by
way of hash functions have been via attacks on collision resistance, via
chosen prefix attacks. Constraining this to cases where we know precisely
who the applicants are significantly reduces the risk of that avenue of
attack, making it mostly a question of second preimage resistance, and
AFAIK, there's been no progress on that dimension of SHA-1.

Second, as pointed out elsewhere in this thread, even if we don't trust the
certificate applicant, we can reduce the chosen prefix risk by requiring a
higher minimum serial number entropy. The usage of the serial number for
OneCRL precludes is reuse anyway.

And finally, I think the lifetime constraint is worthwhile because it
provides a concrete incentive to upgrade in a timely fashion.

Net of the above considerations, I think we have a acceptable mitigations
to the collision risks.

--Richard



>
> Regards,
> Andrew
>

tech...@gmail.com

unread,
Feb 23, 2016, 7:42:18 PM2/23/16
to mozilla-dev-s...@lists.mozilla.org
On Tuesday, February 23, 2016 at 10:58:19 AM UTC-8, Gervase Markham wrote:
Gerv, I just re-read your plan, and maybe point 6 is too restrictive -- allowing only 90 days for these emergency SHA-1 replacement certs. Remember, through Dec. 2015, these same financial service companies COULD have purchased SHA-1 certificates that would have been valid through 12/31/2016 - a full year.

Those SHA-1 certs would NOT have been subject to ANY of the restrictions in your plan - and so they would not have been as secure for users as the emergency SHA-1 certs being issued under your plan.

So why not allow these emergency SHA-1 certs that are subject to your safeguards expire as late as 12/31/2016? (As I have said, there are already many of these machines with SHA-1 certs that expire 12/31/2016 that are NOT subject to your safeguards, so this is only an improvement...)

Steve

unread,
Feb 23, 2016, 7:45:55 PM2/23/16
to Richard Barnes, Andrew Ayer, mozilla-dev-s...@lists.mozilla.org, Gervase Markham, Kathleen Wilson
Large quantities of SHA-1 certificates were issued in the weeks prior to
the deadline as operators of systems not intended for primarily browser
based consumption maximized their remaining compliant lifespan, Embedded
physical deployment of devices that are not updated at browser speed runs
the gamut from car dashboard systems for aGPS to media set top boxes to
other purpose built appliances in closed communities. This device
certificate lifespan extension flood created far more collision surface
than the 9 certificates in question that will be produced by an invested
applicant with presumably no malicious intent to create intentionally
collision prone requests or public keys designed to produce specific
hashes. Statistically the actual risk to the public trust community of
these 9 certs is a fraction of the certs issued in the last two months of
2015. The CA's client missed one part of their environment or else these
renewals would have occurred late last year and this thread wouldn't
exist. In fact, they would have had certificates valid later in 2016 than
the Mozilla proposal offers.

Publicly trusted roots were often used in non-browser situations because
you put two quotes in front of the customer. Here's the cost to create and
manage your own dedicated multitier PKI. Here's the cost to leverage our
existing infra. Many customers chose to live within the existing public
trust PKI as a simple financial situation.

Kind regards,
Steve

On Tue, Feb 23, 2016, 6:42 PM Richard Barnes <rba...@mozilla.com> wrote:

> On Tue, Feb 23, 2016 at 12:05 PM, Andrew Ayer <ag...@andrewayer.name>
> wrote:
>
> > On Tue, 23 Feb 2016 18:57:41 +0000
> > Gervase Markham <ge...@mozilla.org> wrote:
> >
> > > Please comment on whether this proposal seems reasonable, being aware
> > > of the short timelines involved.
> >

Eric Mill

unread,
Feb 23, 2016, 9:31:12 PM2/23/16
to Gervase Markham, mozilla-dev-s...@lists.mozilla.org, Kathleen Wilson, Richard Barnes
On Tue, Feb 23, 2016 at 1:57 PM, Gervase Markham <ge...@mozilla.org> wrote:

>
> Our proposal, which we have sent to Symantec, Worldpay and the other
> browsers, is as follows:
>

Thank you for bringing this to the list for public input, even with a tight
timeline and under immense pressure. It really speaks to how sincerely
Mozilla and its root program are rooted in public review and participation.

I agree fully with Andrew Ayer's comments, and don't believe Mozilla should
allow this exception. While it would certainly cause pain and a loss of
business for some of WorldPay's customers, WorldPay should take alternative
drastic action, such as buying all of their affected customers a new
terminal, paying the shipping costs, and providing human on-site support
wherever possibly practical. If this is expensive to them, fine -- it's
more expensive to the web PKI to establish this kind of precedent.

It would also be worth learning what segment of the market these 10,000
terminals would affect. I've seen these terminals before:

https://www.google.com/search?q=worldpay&espv=2&biw=1168&bih=783&site=webhp&source=lnms&tbm=isch&sa=X&ved=0ahUKEwj91arSqo_LAhUBOSYKHZ4_AtYQ_AUICCgD#tbm=isch&q=worldpay+terminal

Many of them are used by small businesses and sole proprietors. Many of
them are used by large businesses. Which kind of customer is being affected
should factor into Mozilla's decision.

All that said, if Mozilla does decide to proceed, some suggestions inline
below.

Symantec may issue certificates to Worldpay if the following things are
> true:
>
> 1. You immediately give copies to Mozilla (and other vendors who desire
> them) for us to immediately add them to OneCRL, as if they had been
> mis-issued.
>
> 2. Symantec's OCSP server MUST present a response of Revoked to any
> request for these certificates from, at a minimum, Firefox (based on
> User-Agent). Other browsers may wish to be added to this list. Sending
> Revoked to everyone would be easiest, but that depends on your testing
> as to whether it will break the intended clients.
>

Has testing of whether it will break the intended clients been done yet?
Or, will it be done before the certificates are issued?

If tests show that the intended clients will not affected by Symantec's
OCSP responder generically returning Revoked for all user agents, then the
proposal should not allow an exception for the Firefox user agent. While
Mozilla can't speak for what other browsers want, neither the Baseline
Requirements or the Mozilla root program allow CAs to only comply with
requirements insofar as they affect Firefox user agents.

If the testing can't be completed in time, then the proposal should include
a conditional that only allows user agent discretion if testing shows,
after the fact, that the terminals will not work if Symantec's OCSP
responders show them a Revoked response.

Further, if testing shows that the terminals won't work, that testing
should produce a fingerprint that can identify the machines to Symantec's
OCSP responders. So, if user agent testing is allowed, the preferred
approach should be for Worldpay to only produce non-revoked OCSP responses
to those terminals, rather than whitelisting Firefox.



> 6. The lifetime of the issued SHA-1 certificates MUST be no more than 90
> days. Reissuance is permitted, but Mozilla reserves the right to decide
> in the future that the conditions for further issuance of such
> certificates may vary, including deeming them unacceptable under any
> circumstances. Mozilla is very likely to not permit validity to extend
> beyond the SHA-1 deadline of 31st December 2016.
>

This seems much too vague. WorldPay should commit to a date by which their
systems will be updated to support SHA-256 certificates, and publish this
date to cabfpub. Mozilla should not permit issuance of certificates with
notAfter dates after that date.

If the committed-to timeline is > 90 days, Mozilla and Symantec should
encourage WorldPay to beat their deadline. Mozilla can do that by notifying
this list (or requiring Symantec or WorldPay to do so) in advance of a
renewal taking place, in case there is relevant public input.

If WorldPay turns out to need an extension to their committed date, the
process we're doing right now should repeat in full (with more notice), and
the extension should be treated as a newly proposed exception.


>
> 7. This exception applies to Worldpay only; you need to come back and
> ask, presenting the circumstances, for other cases. If the impact is
> similar, similar terms may be extended.
>

I would add a requirement that further applicants for exceptions do so
here, on the mozilla.dev.security.policy list, and that they must provide
more than a few days' notice. This would ensure that neither Mozilla nor
the community get jammed like this again.


Mozilla is very keen to see SHA-1 eliminated, but understands that for
> historical reasons poor decisions were made in private PKIs about which
> roots to trust, and such decisions are not easily remedied.
>

As Andrew Ayer said, the biggest danger here is allowing a poor precedent.
Anyone getting an exception should have dire needs that affect the general
public, and the experience of getting that exception should not be pleasant.

This seems quite fair, in the face of the rather audacious security
exception they are asking the web PKI to create for them, and Mozilla is
asking the community to accept, especially after so many others with dire
use cases have been turned down.

And again, some questions Mozilla should get prompt answers to (and share
with this list) if at all possible before making a decision include:

* What customer segment is affected by this issue?
* Will a universal Revoked response from Symantec's OCSP servers result in
the terminals failing to operate correctly?
* What date is WorldPay willing to commit to, by which they will have fixed
these terminals?

-- Eric


>
> Please comment on whether this proposal seems reasonable, being aware of
> the short timelines involved.
>
> Gerv
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>



--
konklone.com | @konklone <https://twitter.com/konklone>

Peter Gutmann

unread,
Feb 23, 2016, 9:38:55 PM2/23/16
to Gervase Markham, mozilla-dev-s...@lists.mozilla.org, Kathleen Wilson, Richard Barnes
Gervase Markham <ge...@mozilla.org> writes:

>Mozilla is very keen to see SHA-1 eliminated, but understands that for
>historical reasons poor decisions were made in private PKIs about which roots
>to trust, and such decisions are not easily remedied.

I'm curious about what's going on here, as you say this is a private PKI, so
why do they need certs from a public CA? Presumably Worldpay is doing this
for B2B comms, so why don't they issue their own certs, and they can keep
using SHA-1 for as long as required? It seems like Worldpay's mistake wasn't
failing to update SHA-1 only devices, it was using a public CA for a private
PKI.

Peter.

Eric Mill

unread,
Feb 23, 2016, 9:44:59 PM2/23/16
to Peter Gutmann, mozilla-dev-s...@lists.mozilla.org, Gervase Markham, Kathleen Wilson, Richard Barnes
On Tue, Feb 23, 2016 at 9:38 PM, Peter Gutmann <pgu...@cs.auckland.ac.nz>
wrote:
Peter's note reminded me that WorldPay doesn't necessarily have to update
the *code* on each of its terminals (to support SHA-2) -- they could also
just update the contents of the root store to include one of the roots
Symantec operates that is capable of issuing SHA-1 certificates. It doesn't
even have to be a root that was ever publicly trusted.

I'm not trying to trivialize the difficulty of doing even that -- just
noting that, since this is an emergency interim request, WorldPay has
simpler emergency interim options than adding SHA-256 support.

-- Eric


>
> Peter.

Richard Barnes

unread,
Feb 24, 2016, 2:07:00 AM2/24/16
to Eric Mill, mozilla-dev-s...@lists.mozilla.org, Gervase Markham, Kathleen Wilson
I got some feedback privately that encouraging "split-horizon" OCSP is
really architecturally a bad idea, and I have to say I agree. I think we
got caught up in the technical aspects of simulating a revocation for the
web as faithfully as possible, but on further thought, it's important that
a certificate's status have a single, global value that is reflected in
OCSP.

However, Symantec and WorldPay say that the affected devices do check OCSP,
so a globally uniform revocation would not allow these devices to continue
to operate. So we would have to rely on OneCRL. I can probably live with
this; OneCRL has been in Firefox for long enough (since 37) that the base
of old browsers that don't get OneCRL information should be fairly small.

--Richard


>> 6. The lifetime of the issued SHA-1 certificates MUST be no more than 90
>> days. Reissuance is permitted, but Mozilla reserves the right to decide
>> in the future that the conditions for further issuance of such
>> certificates may vary, including deeming them unacceptable under any
>> circumstances. Mozilla is very likely to not permit validity to extend
>> beyond the SHA-1 deadline of 31st December 2016.
>>
>
> This seems much too vague. WorldPay should commit to a date by which their
> systems will be updated to support SHA-256 certificates, and publish this
> date to cabfpub. Mozilla should not permit issuance of certificates with
> notAfter dates after that date.
>
> If the committed-to timeline is > 90 days, Mozilla and Symantec should
> encourage WorldPay to beat their deadline. Mozilla can do that by notifying
> this list (or requiring Symantec or WorldPay to do so) in advance of a
> renewal taking place, in case there is relevant public input.
>
> If WorldPay turns out to need an extension to their committed date, the
> process we're doing right now should repeat in full (with more notice), and
> the extension should be treated as a newly proposed exception.
>
>
>>
>> 7. This exception applies to Worldpay only; you need to come back and
>> ask, presenting the circumstances, for other cases. If the impact is
>> similar, similar terms may be extended.
>>
>
> I would add a requirement that further applicants for exceptions do so
> here, on the mozilla.dev.security.policy list, and that they must provide
> more than a few days' notice. This would ensure that neither Mozilla nor
> the community get jammed like this again.
>
>
> Mozilla is very keen to see SHA-1 eliminated, but understands that for
>> historical reasons poor decisions were made in private PKIs about which
>> roots to trust, and such decisions are not easily remedied.
>>
>
> As Andrew Ayer said, the biggest danger here is allowing a poor precedent.
> Anyone getting an exception should have dire needs that affect the general
> public, and the experience of getting that exception should not be pleasant.
>
> This seems quite fair, in the face of the rather audacious security
> exception they are asking the web PKI to create for them, and Mozilla is
> asking the community to accept, especially after so many others with dire
> use cases have been turned down.
>
> And again, some questions Mozilla should get prompt answers to (and share
> with this list) if at all possible before making a decision include:
>
> * What customer segment is affected by this issue?
> * Will a universal Revoked response from Symantec's OCSP servers result in
> the terminals failing to operate correctly?
> * What date is WorldPay willing to commit to, by which they will have
> fixed these terminals?
>
> -- Eric
>
>
>>
>> Please comment on whether this proposal seems reasonable, being aware of
>> the short timelines involved.
>>
>> Gerv

Rob Stradling

unread,
Feb 24, 2016, 5:16:50 AM2/24/16
to Gervase Markham, mozilla-dev-s...@lists.mozilla.org, Kathleen Wilson, Richard Barnes
On 23/02/16 18:57, Gervase Markham wrote:
> Mozilla and other browsers have been approached by Worldpay, a large
> payment processor, via Symantec, their CA. They have been transitioning
> to SHA-2 but due to an oversight have failed to do so in time for a
> portion of their infrastructure, and failed to renew some SHA-1 server
> certificates before the issuance deadline of 31st December 2015.
>
> They now need 9 SHA-1 certificates issued before 28th February 2015, or
> approximately 10,000+ payment terminals around the world will stop
> working. This equipment was created some time ago, and unfortunately
> only supports publicly-trusted roots. Using roots removed from browser
> root programs is also not a complete solution to the program; these
> 10,000 do not trust any of those roots.

Gerv, I would really like to see more technical details about the PKI
software in WorldPay's terminals before offering an opinion on whether
or not an limited exception should be granted. Perhaps we can find an
alternative technical solution for WorldPay that would have less (or
even no) impact on the Web PKI. With that in mind, some questions...

Which roots do these WorldPay terminals trust?

Do these terminals support signature hash algorithms that are even less
secure than sha1WithRSAEncryption (i.e. md5WithRSAEncryption,
md4WithRSAEncryption, md2WithRSAEncryption) that modern browsers will
reject? (If so, perhaps Symantec could be permitted to issue new MD5
certs to WorldPay from a publicly trusted root).

Do these terminals support key sizes that modern browsers will reject?
(If so, perhaps Symantec could be permitted to issue new (for example)
1024-bit RSA certs to WorldPay from a publicly trusted root).

Do these terminals have any certificate path building/verification bugs
that could be exploited? For example:
- Has anyone actually checked that these terminals will reject an
expired certificate? (If not, then WorldPay could merrily continue to
use the very-soon-to-expire SHA-1 certs that they already have).
- Do these terminals correctly handle the Basic Constraints
extension? (If not, then Symantec could issue new SHA-1 end-entity
certs from a non-CA cert).
- Do these terminals barf if they encounter an unrecognized critical
extension? (If not, then perhaps Symantec could be permitted to issue
new SHA-1 end-entity certs that contain a random critical extension,
which modern browsers should reject).

What software do these terminals use to build and verify certificate
chains? If it's proprietary code developed by WorldPay, then the
community probably wouldn't be able to help find an exploitable bug.
But if it's an old version of NSS or OpenSSL, then the community could
help find an exploitable bug.

<snip>

--
Rob Stradling
Senior Research & Development Scientist
COMODO - Creating Trust Online

Peter Gutmann

unread,
Feb 24, 2016, 5:21:37 AM2/24/16
to Rob Stradling, Gervase Markham, mozilla-dev-s...@lists.mozilla.org, Kathleen Wilson, Richard Barnes
Rob Stradling <rob.st...@comodo.com> writes:

>But if it's an old version of NSS or OpenSSL, then the community could help
>find an exploitable bug.

If it's a remote-code-exec we could patch their firmware for them to support
SHA-256. Think of it as an undocumented remote admin capability.

(Something like this has been done in the past to fix a commercial vendor's
gear).

Peter.

Rob Stradling

unread,
Feb 24, 2016, 6:24:44 AM2/24/16
to Peter Gutmann, mozilla-dev-s...@lists.mozilla.org, Gervase Markham, Kathleen Wilson, Richard Barnes
True, although engineering and deploying that to 10,000+ terminals
within the next 4 days could be a bit of a challenge! ;-)

Steve

unread,
Feb 24, 2016, 9:15:50 AM2/24/16
to Rob Stradling, Peter Gutmann, Gervase Markham, mozilla-dev-s...@lists.mozilla.org, Kathleen Wilson, Richard Barnes
Just as important as browser users are the people who rely on payment
terminals to enjoy their daily life. Here, the affected customer states no
intent to put these certificates into browser accessible space. They state
no business case where the 9 payment gateways are accessible by browsers or
that any business case exists on the gateways that uses any client other
than the payment terminal.

Their path to avoid disruption to consumers on Sunday is the 9 gateways,
not the 10,000+ terminals. Pushing firmware to devices that handle money
in a hurry would show very poor security and privacy posture. I don't want
yesterday's build in my wallet.

If the 9 certificates were issued in December, this conversation wouldn't
exist and the certificates would be valid likely into December this year.
But a mark on the calendar was crossed when we all agreed this would end.
The situation puts people who rely on payment terminals out of service and
is claimed to be oversight.

The potential for a relying party of the trust in the root store cultivated
by Mozilla to be harmed is quite low, there is no browser use case.
Contrast that with 10,000 merchants and how many people per day, per hour,
are denied card swipes to live their daily lives.

Everyone agrees there are significant technical issues with SHA-1 hashes.
However, once the certificates are issued, the status of the certificate is
not significant even if there are collision blocks in the public key
because the serial numbers would differ in the collision pair. Once the
certificates exist, the strength of their use depends on server cipher
suite configuration.

So it comes down to precedent and causing certificate users to take
requirements seriously. While I can't speak authoritatively, there appears
to be no malicious intent here. The assertion is that many other payment
terminals were upgraded on time. So Johnny got a 98 on the exam, but we
send him to his room.

This isn't a case of an attacker submitting keys with manipulated collision
blocks. This is a case of a major business that looks back in horror at a
mistake that is about to cost them untold customer relationships.

There are four days left to solve this. The customer would have had 10
months to solve this if they caught it in December. The same 9
certificates would exist, the same potential collision surface, and 10 more
months of closed community operation.

As we look toward barriers, constraints, and performance expectations, I
suggest that a compromise that maintains security across this community is
to permit careful yet accelerated deployment of SHA256-capable devices to
solve this problem at a pace that avoids accidents and oversights that come
from haste and could lead to PII exposure. I suggest we shift from
prevention to duration, the lifespan of the SHA-1 certificates to be
deployed in this case.

Kind regards,
Steve



On Wed, Feb 24, 2016 at 6:24 AM Rob Stradling <rob.st...@comodo.com>
wrote:

Peter Gutmann

unread,
Feb 24, 2016, 9:25:18 AM2/24/16
to Steve, Rob Stradling, Gervase Markham, mozilla-dev-s...@lists.mozilla.org, Kathleen Wilson, Richard Barnes
Steve <steve...@gmail.com> writes:

>They state no business case where the 9 payment gateways are accessible by
>browsers or that any business case exists on the gateways that uses any
>client other than the payment terminal.

So these things will never see access by a browser enforcing the SHA-1
restrictions? Where is the restriction coming from then, is it that the
browser vendors have told the CAs (via the CAB Forum) they can't issue SHA-1
certs of any kind, even if they're only used in a private PKIs?

I'd really like to get some comment from WorldPay on precisely what's going on
here. It'd be a lot easier to sort out if we knew exactly what was happening,
Rob Stradling posted a good base set of questions that need to be answered.
If we knew more details it seems like there could be an easier solution than
having to break the no-SHA1-any-more rule.

Peter.

Gervase Markham

unread,
Feb 24, 2016, 9:32:13 AM2/24/16
to Eric Mill, Kathleen Wilson, Richard Barnes
On 24/02/16 02:26, Eric Mill wrote:
> It would also be worth learning what segment of the market these 10,000
> terminals would affect. I've seen these terminals before:
>
> https://www.google.com/search?q=worldpay&espv=2&biw=1168&bih=783&site=webhp&source=lnms&tbm=isch&sa=X&ved=0ahUKEwj91arSqo_LAhUBOSYKHZ4_AtYQ_AUICCgD#tbm=isch&q=worldpay+terminal
>
> Many of them are used by small businesses and sole proprietors. Many of
> them are used by large businesses. Which kind of customer is being affected
> should factor into Mozilla's decision.

I don't agree:

"Do not pervert justice; do not show partiality to the poor or
favouritism to the great, but judge your neighbour fairly." -- Leviticus
19:15

> Has testing of whether it will break the intended clients been done yet?
> Or, will it be done before the certificates are issued?

Turns out, these clients do do revocation checking, and various people
are against "split horizon" OCSP, so we will rely on OneCRL only.

Gerv

Gervase Markham

unread,
Feb 24, 2016, 9:34:26 AM2/24/16
to Peter Gutmann, Kathleen Wilson, Richard Barnes
On 24/02/16 02:38, Peter Gutmann wrote:
> I'm curious about what's going on here, as you say this is a private PKI, so
> why do they need certs from a public CA? Presumably Worldpay is doing this
> for B2B comms, so why don't they issue their own certs, and they can keep
> using SHA-1 for as long as required? It seems like Worldpay's mistake wasn't
> failing to update SHA-1 only devices, it was using a public CA for a private
> PKI.

Indeed, but that mistake may have been made 15 or 20 years ago. No CA
today would recommend this course of action. However, once hardware is
in the field, it's often very hard to change. The fact that this sucks
from a security perspective, we would both agree on - but it's true.

Gerv

Steve

unread,
Feb 24, 2016, 9:43:25 AM2/24/16
to Gervase Markham, Eric Mill, mozilla-dev-s...@lists.mozilla.org, Kathleen Wilson, Richard Barnes
Given OCSP support in the terminal software, this isn't likely to be
archaic firmware open to ignoring criticality. Since money is flowing here,
audits would scream at even older hash options or intentional defect
exploitation.

>From experience securing an application that moved 30% of all cash that
changed hands in a business day, I can state that no financial services
company of this scale will expose their network to an untested certificate
chain. Four days are not enough time to test alternate chains or
certificate designs.

Kind regards,
Steve

Gervase Markham

unread,
Feb 24, 2016, 9:58:46 AM2/24/16
to Andrew Ayer, mozilla-dev-s...@lists.mozilla.org, Kathleen Wilson, Richard Barnes
On 23/02/16 20:05, Andrew Ayer wrote:
> Multiple mistakes were made by Worldpay (using public roots, leaving
> the transition to the last minute, and then forgetting to renew before
> the sunset) and Symantec (failing to make sure their customer was
> prepared).

I think it's unreasonable to blame Symantec for this; Worldpay were
clearly aware of the deadline (or at least, part of the business was).

> They had ample opportunity to avoid a crisis. It is not
> Mozilla's responsibility to dig them out of the hole they have dug for
> themselves,

It is not our responsibility; on the other hand, the damage which may
happen if we do not (i.e. if we refuse, Symantec will not issue to
Worldpay, and it's Worldpay's merchant customers who will be unable to
take payments) does not accrue to Worldpay but to others.

Gerv

Gervase Markham

unread,
Feb 24, 2016, 9:59:14 AM2/24/16
to Andrew Ayer, mozilla-dev-s...@lists.mozilla.org, Kathleen Wilson, Richard Barnes
On 23/02/16 20:05, Andrew Ayer wrote:
> Multiple mistakes were made by Worldpay (using public roots, leaving
> the transition to the last minute, and then forgetting to renew before
> the sunset) and Symantec (failing to make sure their customer was
> prepared).

I think it's unreasonable to blame Symantec for this; Worldpay were
clearly aware of the deadline (or at least, part of the business was).

> They had ample opportunity to avoid a crisis. It is not
> Mozilla's responsibility to dig them out of the hole they have dug for
> themselves,

Tim Hollebeek

unread,
Feb 24, 2016, 10:59:31 AM2/24/16
to Steve, Rob Stradling, Peter Gutmann, mozilla-dev-s...@lists.mozilla.org, Gervase Markham, Kathleen Wilson, Richard Barnes
For those who are unaware, payment terminals, especially older ones, generally do not have remote update functionality anyway. Even for modern ones that do, I've heard from point of sale vendors that maybe 25% at most of their terminals in the field are reachable, often because of not enabling the feature to avoid breaking tested systems at random times, segmented networks, and firewalls, and so on.

Getting rid of older terminal software generally involves replacing the terminal, because shipping a USB securely under dual control to a pizza shop franchise owner and expecting him to update his terminal's firmware successfully generally doesn't work. The people involved aren't technical experts.

Also, for those suggesting modifications to the terminal systems themselves, be aware that there are extensive audit or validation requirements for any software or configuration changes on payment terminals. A new PCI PA-DSS audit is not going to be completed in 4 days.

The financial services industry is extremely complicated, and has all sorts of warts that I'm not even going to try to defend, but for those unfamiliar with it, I'd caution against proposing impractical solutions that could very well do far more harm than good. Yes, these businesses should have done a far better job avoiding this problem, but the risk related to issuing a limited number of carefully controlled certificates must be balanced against the reality of denying thousands of small, medium and large businesses the ability to accept cardholder transactions.

-Tim

-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+thollebeek=trustw...@lists.mozilla.org] On Behalf Of Steve
Sent: Wednesday, February 24, 2016 9:16 AM
To: Rob Stradling; Peter Gutmann
Cc: Gervase Markham; mozilla-dev-s...@lists.mozilla.org; Kathleen Wilson; Richard Barnes
Subject: Re: Proposed limited exception to SHA-1 issuance

Their path to avoid disruption to consumers on Sunday is the 9 gateways, not the 10,000+ terminals. Pushing firmware to devices that handle money in a hurry would show very poor security and privacy posture. I don't want yesterday's build in my wallet.


________________________________

This transmission may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is strictly prohibited. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.

Eric Mill

unread,
Feb 24, 2016, 11:04:16 AM2/24/16
to Gervase Markham, mozilla-dev-s...@lists.mozilla.org, Kathleen Wilson, Richard Barnes
On Wed, Feb 24, 2016 at 9:31 AM, Gervase Markham <ge...@mozilla.org> wrote:

> On 24/02/16 02:26, Eric Mill wrote:
> > It would also be worth learning what segment of the market these 10,000
> > terminals would affect. I've seen these terminals before:
> >
> >
> https://www.google.com/search?q=worldpay&espv=2&biw=1168&bih=783&site=webhp&source=lnms&tbm=isch&sa=X&ved=0ahUKEwj91arSqo_LAhUBOSYKHZ4_AtYQ_AUICCgD#tbm=isch&q=worldpay+terminal
> >
> > Many of them are used by small businesses and sole proprietors. Many of
> > them are used by large businesses. Which kind of customer is being
> affected
> > should factor into Mozilla's decision.
>
> I don't agree:
>
> "Do not pervert justice; do not show partiality to the poor or
> favouritism to the great, but judge your neighbour fairly." -- Leviticus
> 19:15
>

Clearly, Mozilla is making a value judgment that this SHA-1 exception is
more merited than other public and private requests for exceptions. It
doesn't sound like Mozilla is potentially supporting this exception based
on a calculation of economic impact, but rather an assessment of its
potential to affect human consumers and business owners.

In either case, I'm not suggesting devaluing rich people - I'm saying that
large businesses have a greater capacity to absorb disruption and ephemeral
economic loss than small businesses and sole proprietors, and that's what
Mozilla should take into account.

-- Eric

Gervase Markham

unread,
Feb 24, 2016, 11:12:01 AM2/24/16
to Jeremy Rowley, Rob Stradling, Kathleen Wilson, Richard Barnes
On 24/02/16 15:45, Jeremy Rowley wrote:
> I think Rob's questions are great and should be answered before deciding.
> Many CAs have roots and can issue certs that browsers will simply reject.
> There may be a simple way to provide them certs without issuing a ton of
> SHA1s that are placed on OneCRL.

As noted during the CAB Forum meeting where this was discussed: they
have 200,000+ devices affected, and the "use an old or decommissioned or
otherwise non-BR root" plan works with 90% of them, but not all. That
was plan A, and it didn't work. We are now on plan B.

Gerv

Gervase Markham

unread,
Feb 24, 2016, 11:12:10 AM2/24/16
to mozilla-dev-s...@lists.mozilla.org
On 24/02/16 16:03, Eric Mill wrote:
> Clearly, Mozilla is making a value judgment that this SHA-1 exception is
> more merited than other public and private requests for exceptions. It
> doesn't sound like Mozilla is potentially supporting this exception based
> on a calculation of economic impact, but rather an assessment of its
> potential to affect human consumers and business owners.
>
> In either case, I'm not suggesting devaluing rich people - I'm saying that
> large businesses have a greater capacity to absorb disruption and ephemeral
> economic loss than small businesses and sole proprietors, and that's what
> Mozilla should take into account.

Regardless, we are unlikely to be able to get an analysis of the nature
of the customers affected before we have to make a decision.

Gerv


benjami...@gmail.com

unread,
Feb 24, 2016, 1:15:17 PM2/24/16
to mozilla-dev-s...@lists.mozilla.org
On Tuesday, February 23, 2016 at 6:58:19 PM UTC, Gervase Markham wrote:
> Mozilla and other browsers have been approached by Worldpay, a large
> payment processor, via Symantec, their CA. They have been transitioning
> to SHA-2 but due to an oversight have failed to do so in time for a
> portion of their infrastructure, and failed to renew some SHA-1 server
> certificates before the issuance deadline of 31st December 2015.
>
> They now need 9 SHA-1 certificates issued before 28th February 2015, or
> approximately 10,000+ payment terminals around the world will stop
> working. This equipment was created some time ago, and unfortunately
> only supports publicly-trusted roots. Using roots removed from browser
> root programs is also not a complete solution to the program; these
> 10,000 do not trust any of those roots. This equipment does not support
> SHA-256 and cannot be replaced in time. The data travels over the public
> internet but the servers are not accessed by browsers. Due to the short
> timelines involved, a change in the BRs by the CAB Forum is also not
> possible. Therefore, they are seeking to get browser acknowledgement
> that a qualified audit, qualified by the existence of these certs, will
> be acceptable.
>
> The payment industry is moving towards SHA-256 but their timeline does
> not line up with the CAB Forum one. Our understanding is that Worldpay
> is not the only payment processor in this position. (We are not sure how
> to match this information with Worldpay's assertion that this was an
> oversight on their part, unless such oversights are unusually common at
> payment processors.)
>
> Our proposal, which we have sent to Symantec, Worldpay and the other
> browsers, is as follows:
>
> Symantec may issue certificates to Worldpay if the following things are
> true:
>
> 1. You immediately give copies to Mozilla (and other vendors who desire
> them) for us to immediately add them to OneCRL, as if they had been
> mis-issued.
>
> 2. Symantec's OCSP server MUST present a response of Revoked to any
> request for these certificates from, at a minimum, Firefox (based on
> User-Agent). Other browsers may wish to be added to this list. Sending
> Revoked to everyone would be easiest, but that depends on your testing
> as to whether it will break the intended clients.
>
> 3. Certificates issued under this exception MUST be logged to CT, and
> Symantec MUST disclose which logs they will be published in.
>
> 4. On issuance of any such certificate(s), the issuer MUST send mail to
> cabfpub announcing the event, including references to the CT entries.
>
> 5. The auditor's qualification MUST actively attest that the extent of
> SHA-1 issuance is no greater than that disclosed in CT. (Otherwise the
> qualification will be deemed unacceptable.)
>
> 6. The lifetime of the issued SHA-1 certificates MUST be no more than 90
> days. Reissuance is permitted, but Mozilla reserves the right to decide
> in the future that the conditions for further issuance of such
> certificates may vary, including deeming them unacceptable under any
> circumstances. Mozilla is very likely to not permit validity to extend
> beyond the SHA-1 deadline of 31st December 2016.
>
> 7. This exception applies to Worldpay only; you need to come back and
> ask, presenting the circumstances, for other cases. If the impact is
> similar, similar terms may be extended.
>
>
> Mozilla is very keen to see SHA-1 eliminated, but understands that for
> historical reasons poor decisions were made in private PKIs about which
> roots to trust, and such decisions are not easily remedied.
>
> Please comment on whether this proposal seems reasonable, being aware of
> the short timelines involved.
>
> Gerv

I wonder if Worldpay are making their customers who are using those payment terminals aware of this issue so that they can make contingency plans themselves.

In any case this exception should probably go ahead as there doesn't seem to be a security risk here for trust store users, just a risk of setting a bad precedent if organisations are allowed to make mistakes, and then use the scale of their mistakes to force Mozilla to bail them out.

Andrew Ayer

unread,
Feb 24, 2016, 2:23:50 PM2/24/16
to Gervase Markham, mozilla-dev-s...@lists.mozilla.org
On Wed, 24 Feb 2016 14:58:37 +0000
Gervase Markham <ge...@mozilla.org> wrote:

> > They had ample opportunity to avoid a crisis. It is not
> > Mozilla's responsibility to dig them out of the hole they have dug
> > for themselves,
>
> It is not our responsibility; on the other hand, the damage which may
> happen if we do not (i.e. if we refuse, Symantec will not issue to
> Worldpay, and it's Worldpay's merchant customers who will be unable to
> take payments) does not accrue to Worldpay but to others.

If Mozilla declines to make this exception, it will not be that choice
which harms Worldpay's customers. It will be the numerous bad choices
made by Worldpay. Worldpay has significant resources at their
disposal: in 2014, their revenue was 3.6 billion GBP and their EBITDA
was 375 million GBP[1]. If Worldpay cannot get new payment terminals to
their affected merchants in time, then they can overnight credit card
imprinters and their merchants can accept cards the old fashioned way.
If Worldpay does not want to take care of their customers, Mozilla
should feel no guilt for not doing that for them.

It takes far too long to deprecate insecure practices on the Internet:
consider 1024-bit RSA roots, weak DH, RC4, and MD5. How many times has
Firefox (and other browsers) been forced to delay a deprecation in
order to accommodate the needs of large companies? These companies have
the resources to do better, but they don't because they do not see a
cost to continuing their insecure practices. Instead, they are happy
to have the cost be borne by the hundreds of millions of Firefox users,
who are forced to use a browser that is less secure than it should be.
Bailing Worldpay out of the predicament they put themselves in will
reinforce the perception that there are no economic consequences for
failing to properly invest in security that affects the entire Internet.

No Firefox users will be adversely affected if an exception is not made.
But if an exception is made, companies will continue under-investing in
the migrations that are necessary to move the security of the Internet
forward. That will harm Firefox users in the long run.

Regards,
Andrew

[1] http://www.worldpay.com/us/about/financial-results

Gervase Markham

unread,
Feb 24, 2016, 2:36:02 PM2/24/16
to mozilla-dev-s...@lists.mozilla.org
On 24/02/16 19:27, Jeremy Rowley wrote:
> I believe the concern is that Worldpay is asking for an exception by saying,
> "We've tried 'things' and they didn't work - can we please have a SHA1
> cert?" We don't know what these 'things' they've tried are or whether there
> is an alternative. Lots of customers have asked for SHA1 certs on the
> premises that they need them because of old devices. Is this one special?
> Perhaps, but the alternatives should first be considered.

I believe that large chunks of Worldpay got this right; one part of the
business "didn't get the memo" and missed the deadline for cert renewal
at the end of the year. Once they found out, a short time ago, they
tried the "non-BR root" method but that only covered 90% of devices due
to divergent root stores. (200,000 devices use these servers, so 20,000
would still be affected if they went that way - that's where the numbers
we have been using come from.)

> When creating OneCRL, Mozilla expressed concerns about the potential size of
> the CRL if end entity certs were included. Now, they are being asked to
> include 10,000 end-entity certs in OneCRL (which are not even revoked).

Sorry if this wasn't clear originally; they don't need one cert per
terminal, they need one cert per receiving server. The number of certs
concerned is nine (9).

Gerv

Steve

unread,
Feb 24, 2016, 2:36:21 PM2/24/16
to Jeremy Rowley, Gervase Markham, Eric Mill, mozilla-dev-s...@lists.mozilla.org, Kathleen Wilson, Richard Barnes
It's 9 certs. They go on the head end, the gateway. The deployed devices
have a finite trust list that relies on certain roots. Getting new EEs on
10k devices that don't auto-enroll is physically impossible in a merchant
network with no OTA/OTW path. Plus if the issue was TLS mutual auth from
the client, presumably root store trust reconfiguration at the 9 servers
could be done while holding lunch in the other hand.

Kind regards,
Steve


On Wed, Feb 24, 2016 at 2:27 PM Jeremy Rowley <jeremy...@digicert.com>
wrote:

> Technically, Worldpay had a lot longer than four days to figure this
> out....
> It's not like SHA1 issues jumped out from a behind a bush to scare
> everyone.
>
>
> I believe the concern is that Worldpay is asking for an exception by
> saying,
> "We've tried 'things' and they didn't work - can we please have a SHA1
> cert?" We don't know what these 'things' they've tried are or whether there
> is an alternative. Lots of customers have asked for SHA1 certs on the
> premises that they need them because of old devices. Is this one special?
> Perhaps, but the alternatives should first be considered.
>
> When creating OneCRL, Mozilla expressed concerns about the potential size
> of
> the CRL if end entity certs were included. Now, they are being asked to
> include 10,000 end-entity certs in OneCRL (which are not even revoked).
> This
> is contrary to their previous policy decision to keep OneCRL small. 10k
> certs isn't big. 10k certs for ONE customer is significant.
>
> Jeremy
>
> -----Original Message-----
> From: dev-security-policy
> [mailto:dev-security-policy-bounces+jeremy.rowley
> =digice...@lists.mozilla
> .org] On Behalf Of Steve
> Sent: Wednesday, February 24, 2016 7:43 AM
> To: Gervase Markham; Eric Mill;
> mozilla-dev-s...@lists.mozilla.org
> Cc: Kathleen Wilson; Richard Barnes
> Subject: Re: Proposed limited exception to SHA-1 issuance
>

Rob Stradling

unread,
Feb 24, 2016, 3:32:28 PM2/24/16
to Gervase Markham, mozilla-dev-s...@lists.mozilla.org, Kathleen Wilson, Richard Barnes
On 24/02/16 14:40, Gervase Markham wrote:
> Hi Rob,
>
> These are extremely good questions. I have some of the answers.
>
> On 24/02/16 10:16, Rob Stradling wrote:
>> Gerv, I would really like to see more technical details about the PKI
>> software in WorldPay's terminals before offering an opinion on whether
>> or not an limited exception should be granted. Perhaps we can find an
>> alternative technical solution for WorldPay that would have less (or
>> even no) impact on the Web PKI. With that in mind, some questions...
>
> One trouble is that they have a heterogeneous mix of 200,000+ terminals,
> and so asking about "the software in them" is very difficult. Many are
> old, some have lost technical documentation. (Yes, this is not a good
> situation. I know.) This is why anything other than a reissuance from
> the same root of a near-identical cert is quite likely to break something.
>
> These terminals are being cycled out of use, but the industry deadline
> for the latest EMV standards is mid-2017, which doesn't match up with
> the Web PKI SHA-1 deadline.
>
>> Which roots do these WorldPay terminals trust?
>
> See above - different ones trust different roots. 90% trust a root which
> has been removed from browser root stores, but 10% do not - so if they
> use that solution, about 10% of terminals will break.

It's very unfortunate that the technical documentation for some of these
terminal models has been lost. However, given that WorldPay have been
able to do that 90%/10% analysis, presumably they do have a good idea of
which root(s) are trusted by which terminal models.

Could we please have a list of which roots are trusted by which terminal
models?

Do the 10% not trust the yanked VeriSign root because they're newer than
the 90%? If so, presumably these are models for which the technical
documentation has not been lost...so...are the 10% sufficiently new that
they do support SHA-2?

Are any WorldPay representatives following this thread and able/willing
to answer questions?

Dean Coclin

unread,
Feb 24, 2016, 3:52:47 PM2/24/16
to mozilla-dev-s...@lists.mozilla.org
This is Dean from Symantec (same Dean as the CA/B Forum Chair but I'm leaving that hat off right now). I'd like to answer some questions about this situation on which I agree is less than ideal.
First off, as Gerv mentioned, many device manufacturers erroneously embedded public roots in their devices way before CA/B Forum Baseline Requirements were ever conceived. One can argue they didn't know any better. CAs didn't (and still don't) always know who has downloaded their public roots or copied them from browsers to use in their own devices. These devices go as far back as the 90s and are still in use in merchants as POS and ATMs. Many devices cannot have their root stores or their crypto code remotely upgraded. The payment processors don't always provide the devices to the merchants. Much like the way you can buy your own telephone and plug it into the public network, merchants can source devices from many places. Hence, the payment processors don't control the endpoints. The number of device manufacturers coupled with the myriad of various firmware revisions makes it difficult to give precise numbers. 
This is apparently a widespread problem in this ecosystem and unfortunately Worldpay is first up, given the imminent expiration of their certificates. However, the problem is much larger than Worldpay. The FS-ISAC, which is an association of these processors, is scoping the entire problem space by surveying their members and I'm hoping to get the results to the browsers shortly. But let's focus on WP for now since time is short.
Some of the suggestions on this thread are very good. But given the tight time constraint and the fact that they would have to be tested, some can't be considered for this instance. For example:

1. Creating a new intermediate. This will have to be tested to see if the terminals will accept this.
2. What roots are trusted in the devices? Hard to say given the myriad of devices, firmware and changes over the years
3. Including >80 bit of entropy. Great idea and I believe that is already the case for these existing certificates, and will be the case for any new ones we sign.
4. Technically constraining the sub CA. Another great idea but would require testing first
5. Use an older hashing algorithm. Prohibited by PCI.
6. Use a smaller (1024) key size root. Interesting idea that might work but again would have to be tested
7. Use an older root that has been removed from the browsers. This has worked for at least 85% of the terminals. It's the last 10-15% that are affected.
In the end, everyone is right. This should have been brought to light some time ago. Symantec did notify these customers but unfortunately not all certificates were renewed in time. Perhaps they didn't understand the impact. While the processor in question didn't renew their SHA-1 certs before the deadline, that's neither here nor there. But as others have said, this is a serious issue with a short timeline to address.
Thank you
Dean Coclin
Symantec
 
 
On 02/24/16, Gervase Markham<ge...@mozilla.org> wrote:
 
On 24/02/16 19:27, Jeremy Rowley wrote:
> I believe the concern is that Worldpay is asking for an exception by saying,
> "We've tried 'things' and they didn't work - can we please have a SHA1
> cert?" We don't know what these 'things' they've tried are or whether there
> is an alternative. Lots of customers have asked for SHA1 certs on the
> premises that they need them because of old devices. Is this one special?
> Perhaps, but the alternatives should first be considered.

I believe that large chunks of Worldpay got this right; one part of the
business "didn't get the memo" and missed the deadline for cert renewal
at the end of the year. Once they found out, a short time ago, they
tried the "non-BR root" method but that only covered 90% of devices due
to divergent root stores. (200,000 devices use these servers, so 20,000
would still be affected if they went that way - that's where the numbers
we have been using come from.)

> When creating OneCRL, Mozilla expressed concerns about the potential size of
> the CRL if end entity certs were included. Now, they are being asked to
> include 10,000 end-entity certs in OneCRL (which are not even revoked).

Sorry if this wasn't clear originally; they don't need one cert per
terminal, they need one cert per receiving server. The number of certs
concerned is nine (9).

Gerv

Peter Bowen

unread,
Feb 24, 2016, 5:37:11 PM2/24/16
to Dean Coclin, mozilla-dev-s...@lists.mozilla.org
Dean as Symantec,

What CA(s) would Symantec use as the issuer for the certificates?

Thanks,
Peter

Dean Coclin

unread,
Feb 24, 2016, 5:54:35 PM2/24/16
to pzb...@gmail.com, mozilla-dev-s...@lists.mozilla.org
Peter,
The same one they've been using and know works: VeriSign Class 3 International Server CA - G3.
 Dean

rba...@mozilla.com

unread,
Feb 24, 2016, 7:11:44 PM2/24/16
to mozilla-dev-s...@lists.mozilla.org
Hey all,

Thanks to everyone for the robust discussion here. Gerv, Kathleen and I have discussed and decided that Mozilla will allow a qualification due to issuance of SHA-1 certificates, subject to the following conditions:

1. SHA-1 certificates MUST NOT be issued for any name other than the seven names requested by Symantec (tptrans.lynksystems.com, tptrans-l.lynksystems.com, tpdev.lynksystems.com, tframe1.rbslynk.com, tframe2.rbslynk.com, tf.lynk-systems.com, and qac.tf.rbslynk.com)

2. On issuance of any such certificate(s), the issuer MUST take the following actions:
2.a. Submit the certificates to one or more Certificate Transparency logs. (There is no requirement for the certificates to contain a Signed Certificate Timestamp.)
2.b. Send email to the cabfpub and dev.security.policy mailing lists announcing this event, including references to the CT entries.

3. The lifetime of the issued SHA-1 certificates MUST be no more than 90 days, and MUST NOT extend beyond 2016-12-31. Reissuance is permitted, but must be requested at least two weeks in advance on the dev.security.policy mailing list. Mozilla reserves the right to decide in the future that the conditions for further issuance of such certificates may vary, including deeming them unacceptable under any circumstances.

4. The serial numbers of issued SHA-1 certificates MUST contain at least 80 bits of entropy.

5. The auditor's qualification MUST actively attest that the extent of SHA-1 issuance is no greater than that disclosed in CT. (Otherwise the qualification will be deemed unacceptable.)

6. This allowance applies only to the Worldpay names above; allowance for any other cases MUST be requested on this mailing list at least two weeks in advance of any further allowance being made.

While we are disappointed that a critical part of the Internet infrastructure is holding back an increase in security, we believe that this allowance strikes an acceptable compromise between minimizing disruption and risk and encouraging migration away from SHA-1 as fast as possible.

Thanks,
--Richard

Peter Gutmann

unread,
Feb 24, 2016, 10:15:59 PM2/24/16
to Dean Coclin, pzb...@gmail.com, mozilla-dev-s...@lists.mozilla.org
Dean Coclin <dean.j...@verizon.net> writes:

>The same one they've been using and know works: VeriSign Class 3
>International Server CA - G3.

So the devices will trust any cert from this CA?

This is a serious question, a contractor once got into USG infrastructure
with a $20 or so cert because they'd done the same thing, contracted their
private PKI out to a public CA, so their infrastructure trusted any cert
chaining up to the CA's roots.

Anyone know if one of these:

http://www.ebay.com/itm/WorldPay-Verifone-Credit-Card-Transaction-Terminal-With-Printer-New-In-Box-/301863681435
http://www.ebay.com/itm/WorldPay-Zinc-Chip-and-Pin-Mobile-Card-reader-/262304198832

would use these certs?

Peter.


Peter Gutmann

unread,
Feb 24, 2016, 10:56:11 PM2/24/16
to rba...@mozilla.com, mozilla-dev-s...@lists.mozilla.org
rba...@mozilla.com <rba...@mozilla.com> writes:

>While we are disappointed that a critical part of the Internet
>infrastructure is holding back an increase in security, we believe that
>this allowance strikes an acceptable compromise between minimizing
>disruption and risk and encouraging migration away from SHA-1 as fast as
>possible.

I'd still really like to know the details of what happened here. As I've
pointed out to others off-list, it's not to assign blame but to learn from
it so that others won't make the same mistake in similar situations in the
future.

Peter.

Richard Barnes

unread,
Feb 25, 2016, 12:59:15 AM2/25/16