Public trust of VISA's CA

[Gervase Markham]

In https://bugzilla.mozilla.org/show_bug.cgi?id=1391087 , as part of
their comments on a report of BR-non-compliant certificate issuance, a
representative of VISA said the following:

"I would like to share with you some details regarding our PKI System
and our position within the CA/Browser Forum. Visa is one of the oldest
operating Certificate Authorities and is currently a non-voting member
within the CA/Browser Forum. Visa has been operating a closed PKI
system prior to the inception of the Baseline Requirements, which we had
a number of legacy processes for the issuance and fulfillment of our
certificates to our clients. Certificates that are issued by Visa
public CA’s are issued only to our clients for interconnectivity
purposes. Unlike other CA’s and particularly those that have undergone
your Blink Process, our core business is not PKI. The certificates that
were impacted with the noted issues were not issued erroneously to a bad
actor(s) nor do we issue certificates to the open public. Due to our
unique PKI system, we are not at liberty to divulge with the public our
list of impacted clients and their certificates without our Legals' consent.

Regarding BR compliance, we completed our initial BR audit in September
of 2016. Since that time, we have been addressing the observations
noted by our external auditors. This also would encompass any
certificate issues that have been publically reported. Understanding
that such changes in adopting a new process will have business impact,
it is difficult to provide an accurate timeline of complete compliance
as we are required to assess the impact to our client and payment
systems to avoid any operational impact. We are committed to aligning
with BR and Mozilla requirements as we have continuously move forward in
making the necessary changes."

>From the above, we see that Visa only issues certificates to their own
customers/clients, and not to the public. They believe that this permits
them to keep confidential details of the certificates which they wish to
have public trust.

The Mozilla Root Store Policy, section 2.1, states:

"2.1 CA Operations. CAs whose certificates are included in Mozilla's
root program MUST:
1) provide some service relevant to typical users of our software
products; ..."

My memory suggests to me that this clause is normally understood to
preclude the inclusion of companies who wish to only issue certificates
to themselves and their customers.

We also see that they are unable to provide a timeline for full BR
compliance. This is despite various assurances of current compliance to
Mozilla policies (and thereby the BRs) in various CA communications,
such as April 2017 and March 2016.

In the light of this, I believe it is reasonable to discuss the question
of whether Visa's PKI (and, specifically, the VISA eCommerce Root,
https://crt.sh/?id=896972 , which is the one includes in our store)
meets the criteria for inclusion in Mozilla's Root Store Policy, and
whether it is appropriate for them to continue to hold public trust.
Your comments are welcome.

Gerv

[Alex Gaynor]

https://crt.sh/mozilla-certvalidations?group=version&id=896972 is a very
informative graph for me -- this is the number of validations performed by
Firefox for certs under this CA. It looks like at the absolute peak, there
were 1000 validations in a day. That's very little value for our users, in
return for an awful lot of risk.

Alex

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

[Peter Bowen]

On Tue, Sep 19, 2017 at 8:12 AM, Gervase Markham via

dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
> In https://bugzilla.mozilla.org/show_bug.cgi?id=1391087 , as part of
> their comments on a report of BR-non-compliant certificate issuance, a
> representative of VISA said the following:
>

> "Visa has been operating a closed PKI
> system prior to the inception of the Baseline Requirements, which we had
> a number of legacy processes for the issuance and fulfillment of our
> certificates to our clients. Certificates that are issued by Visa
> public CA’s are issued only to our clients for interconnectivity
> purposes."
>

> From the above, we see that Visa only issues certificates to their own
> customers/clients, and not to the public. They believe that this permits
> them to keep confidential details of the certificates which they wish to
> have public trust.
>
> The Mozilla Root Store Policy, section 2.1, states:
>
> "2.1 CA Operations. CAs whose certificates are included in Mozilla's
> root program MUST:
> 1) provide some service relevant to typical users of our software
> products; ..."
>
> My memory suggests to me that this clause is normally understood to
> preclude the inclusion of companies who wish to only issue certificates
> to themselves and their customers.

Gerv,

I think your statement is a little broad. Every CA only issues
certificates to themselves and their own customers (or as the BRs call
them "Subscribers"). What is key here is that Mozilla requires the
certificates have relevance to "typical users of [Firefox]" while
while Visa explicitly says they only use the certificates for
"interconnectivity" purposes. I take it to mean connections between
their clients and Visa, not between clients and the broad Internet
user base.

> We also see that they are unable to provide a timeline for full BR
> compliance. This is despite various assurances of current compliance to
> Mozilla policies (and thereby the BRs) in various CA communications,
> such as April 2017 and March 2016.
>
> In the light of this, I believe it is reasonable to discuss the question
> of whether Visa's PKI (and, specifically, the VISA eCommerce Root,
> https://crt.sh/?id=896972 , which is the one includes in our store)
> meets the criteria for inclusion in Mozilla's Root Store Policy, and
> whether it is appropriate for them to continue to hold public trust.
> Your comments are welcome.

That crt.sh link shows that it only has one unexpired subordinate CA,
the "Visa eCommerce Issuing CA". CT logs only show 27 unexpired
certificates issued by this subordinate CA:
https://crt.sh/?Identity=%25&iCAID=1414&exclude=expired, of which two
are revoked.

The included CAs list indicates it never followed the current Mozilla
inclusion process, but is one of four "legacy" CAs. The other three
are operated by CAs that have followed the inclusion process for other
roots, so this means Visa is the only CA operator in the Mozilla
program who is a purely "legacy" CA.

I take legacy to mean that they were never assessed against the
Mozilla inclusion standards, so it does seem reasonable to review
whether they continue to meet the inclusion policy.

Thanks,
Peter

[Matthew Hardeman]

On Tuesday, September 19, 2017 at 10:13:26 AM UTC-5, Gervase Markham wrote:

> >From the above, we see that Visa only issues certificates to their own
> customers/clients, and not to the public. They believe that this permits
> them to keep confidential details of the certificates which they wish to
> have public trust.

The overall question of whether they should be issuing special use certificates from a publicly trusted CA is worthwhile, but I wonder whether the point about disclosure / confidential details of certificate issuance isn't practically mooted by the anticipated requirement that from April of next year, leaf certificates from included public CAs will require SCT proofs in order to be trusted by the (currently) largest market share browser? (And presumably the other browsers will likely follow suit similarly?)

The other matters in discussion regarding this root hierarchy almost certainly do merit some attention.

It sounds like VISA is generally using this between software and hardware elements of an intranet / extranet nature between the VISA organization and partner banking institutions and their service providers. What interest of theirs is served by being included in public trust stores?

[Martin Rublik]

On Tue, Sep 19, 2017 at 5:22 PM, Alex Gaynor via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> https://crt.sh/mozilla-certvalidations?group=version&id=896972 is a very
> informative graph for me -- this is the number of validations performed by
> Firefox for certs under this CA. It looks like at the absolute peak, there
> were 1000 validations in a day. That's very little value for our users, in
> return for an awful lot of risk.
>
> Alex

Hi,

I agree that 1000 validations in a day is not much, or better to say really
low number. Anyway I was wondering what should be a minimum value or
whether this number is a good metric at all. I went through the Mozilla
validations telemetrics and there are more CAs with similliar number of
validations.


Martin

[Peter Bowen]

Note that Firefox 55 had a regression on how it does chain building
(https://bugzilla.mozilla.org/show_bug.cgi?id=1400913) that causes it
prefer the longest chain rather than the shortest chain. This means,
for Root CAs that are cross-signed, Firefox 55 will frequently
attribute to the wrong bucket. The total on the buckets does not
change, but the validations per day did shift. For example, Firefox
55 shows "AddTrust External CA Root" is a super popular root while
prior versions had "COMODO RSA Certification Authority" as a top root.
"Go Daddy Class 2 CA" and "Go Daddy Root Certificate Authority - G2"
also flipped in Firefox 55.

This does not impact the Visa bucket, as far as I know, as the Visa
root is not cross-signed by any other root.

Thanks,
Peter

[Jakob Bohm]

In interpreting that statistic, it is worth noting that Banks etc. tend
to have strong internal security configuration policies, which probably
include the disabling of all kinds of application "telemetry".

But it is still worth considering if this CA root should only be a
non-public CA root, included only by local configuration (typically
using the Firefox/Thunderbird enterprise deployment tools in the case of
Firefox/Thunderbird).

The age of their inclusion suggests a long transition period if removal
is solely for formal reasons rather than actual insecurity. And of
cause such removal should be in a form that doesn't block manual
reinclusion via configuration.


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

[Gervase Markham]

Additionally, 13 days ago it was reported to VISA that their OCSP
responder was misconfigured to return "good" responses for non-existent
certificates:
https://bugzilla.mozilla.org/show_bug.cgi?id=1398261
As far as I can see, this is the case for their end-entity certificates,
not just some roots or intermediates.

Two weeks later, they have not yet provided any response.

Gerv

[Paul Kehrer]

I can confirm that as of this moment the VISA OCSP responders are still
responding GOOD for non-existent certificates. VISA was originally
contacted by me on August 29 so it has now been over 21 days since initial
report.

-Paul

On September 21, 2017 at 9:32:12 PM, Gervase Markham via

[Tim Smith]

On Tuesday, September 19, 2017 at 8:13:26 AM UTC-7, Gervase Markham wrote:
> In the light of this, I believe it is reasonable to discuss the question
> of whether Visa's PKI (and, specifically, the VISA eCommerce Root,
> https://crt.sh/?id=896972 , which is the one includes in our store)
> meets the criteria for inclusion in Mozilla's Root Store Policy, and
> whether it is appropriate for them to continue to hold public trust.
> Your comments are welcome.

Has Visa responded to this privately, and is there an upcoming decision point? Visa has failed for some time to respond to open questions in https://bugzilla.mozilla.org/show_bug.cgi?id=1391087; I am not sure Visa is willing to be accountable to Mozilla's process or the public interest.

[Jonathan Rudenberg]

> On Sep 19, 2017, at 11:12, Gervase Markham via dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
>
> In the light of this, I believe it is reasonable to discuss the question
> of whether Visa's PKI (and, specifically, the VISA eCommerce Root,
> https://crt.sh/?id=896972 , which is the one includes in our store)
> meets the criteria for inclusion in Mozilla's Root Store Policy, and
> whether it is appropriate for them to continue to hold public trust.
> Your comments are welcome.

I don’t think this issue ever got a conclusion. It is clear to me that Visa should be removed from the Mozilla root program immediately.

Visa has been extremely unresponsive in general. The most recent case was their non-compliant OCSP responder: https://bugzilla.mozilla.org/show_bug.cgi?id=1398261

It took them five months to fix the problem, and there is still no incident report.

In their response to January CA Communication Action 2 (about insecure domain validation methods), they did not provide a comment response, but selected the option indicating they were using these vulnerable methods and required a date for an update in the comment field:

> We have active (not expired or revoked) certificates issued using these methods. We will review our implementation for vulnerabilities and report our findings on the mozilla.dev.security.policy list by the date specified in the comments section below.

There are currently only 90 unexpired certificates issued by this CA known to CT: https://crt.sh/?Identity=%25&iCAID=1414&exclude=expired (last time we looked, there were only 27 and two were revoked)

Additionally, the telemetry shows an extremely small number of validations.

It’s not clear to me from their responses whether they are even currently BR-compliant, and as of September 13, 2017 it seems like they weren’t:

> Regarding BR compliance, we completed our initial BR audit in September of 2016. Since that time, we have been addressing the observations noted by our external auditors. This also would encompass any certificate issues that have been publically reported. Understanding that such changes in adopting a new process will have business impact, it is difficult to provide an accurate timeline of complete compliance as we are required to assess the impact to our client and payment systems to avoid any operational impact. We are committed to aligning with BR and Mozilla requirements as we have continuously move forward in making the necessary changes .

Given all this, I don’t think there is a lot of risk to Mozilla’s users with no benefit if Visa continues to be included in the root program.

Jonathan

[Wayne Thayer]

On Tue, Feb 13, 2018 at 10:49 AM, Jonathan Rudenberg <jona...@titanous.com>
wrote:

>
> > On Sep 19, 2017, at 11:12, Gervase Markham via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
> >
> > In the light of this, I believe it is reasonable to discuss the question
> > of whether Visa's PKI (and, specifically, the VISA eCommerce Root,
> > https://crt.sh/?id=896972 , which is the one includes in our store)
> > meets the criteria for inclusion in Mozilla's Root Store Policy, and
> > whether it is appropriate for them to continue to hold public trust.
> > Your comments are welcome.
>
> I don’t think this issue ever got a conclusion. It is clear to me that
> Visa should be removed from the Mozilla root program immediately.
>

> We did reach a conclusion on the original question that Gerv raised in
this thread: does Visa meet the following requirement from section 2.1 of
the Mozilla root store policy:

CAs whose certificates are included in Mozilla's root program MUST provide
some service relevant to typical users of our software products.

In the thread on this list titled "Updating Root Inclusion Criteria" it was
decided that we will not attempt to restrict organizations from
participating in the Mozilla CA program based on a judgement of their value
to our users.

Visa has been extremely unresponsive in general. The most recent case was
> their non-compliant OCSP responder: https://bugzilla.mozilla.org/s
> how_bug.cgi?id=1398261
>
> It took them five months to fix the problem, and there is still no
> incident report.
>

> Correct. Would a representative from Visa care to comment on this?

In their response to January CA Communication Action 2 (about insecure
> domain validation methods), they did not provide a comment response, but
> selected the option indicating they were using these vulnerable methods and
> required a date for an update in the comment field:
>
> > We have active (not expired or revoked) certificates issued using these
> methods. We will review our implementation for vulnerabilities and report
> our findings on the mozilla.dev.security.policy list by the date specified
> in the comments section below.
>

> Good point. I would appreciate a response from a Visa representative with
the date by which these findings will be reported.

> There are currently only 90 unexpired certificates issued by this CA known
> to CT: https://crt.sh/?Identity=%25&iCAID=1414&exclude=expired (last time
> we looked, there were only 27 and two were revoked)
>
> Additionally, the telemetry shows an extremely small number of validations.
>
> It’s not clear to me from their responses whether they are even currently
> BR-compliant, and as of September 13, 2017 it seems like they weren’t:
>
> > Regarding BR compliance, we completed our initial BR audit in September
> of 2016. Since that time, we have been addressing the observations noted
> by our external auditors. This also would encompass any certificate issues
> that have been publically reported. Understanding that such changes in
> adopting a new process will have business impact, it is difficult to
> provide an accurate timeline of complete compliance as we are required to
> assess the impact to our client and payment systems to avoid any
> operational impact. We are committed to aligning with BR and Mozilla
> requirements as we have continuously move forward in making the necessary
> changes .
>

> The most recent BR audit report for the Visa eCommerce Root contains 3
qualifications: http://enroll.visaca.com/WTBR%20eComm.pdf

[Jonathan Rudenberg]

> On Feb 13, 2018, at 19:16, Wayne Thayer via dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
>
> On Tue, Feb 13, 2018 at 10:49 AM, Jonathan Rudenberg <jona...@titanous.com>
> wrote:
>
>>
>>> On Sep 19, 2017, at 11:12, Gervase Markham via dev-security-policy <
>> dev-secur...@lists.mozilla.org> wrote:
>>>
>>> In the light of this, I believe it is reasonable to discuss the question
>>> of whether Visa's PKI (and, specifically, the VISA eCommerce Root,
>>> https://crt.sh/?id=896972 , which is the one includes in our store)
>>> meets the criteria for inclusion in Mozilla's Root Store Policy, and
>>> whether it is appropriate for them to continue to hold public trust.
>>> Your comments are welcome.
>>
>> I don’t think this issue ever got a conclusion. It is clear to me that
>> Visa should be removed from the Mozilla root program immediately.
>>
> We did reach a conclusion on the original question that Gerv raised in
> this thread: does Visa meet the following requirement from section 2.1 of
> the Mozilla root store policy:
>
> CAs whose certificates are included in Mozilla's root program MUST provide
> some service relevant to typical users of our software products.

This is an answer to the first question, the second part is “whether it is appropriate for them to continue to hold public trust.”

> In the thread on this list titled "Updating Root Inclusion Criteria" it was
> decided that we will not attempt to restrict organizations from
> participating in the Mozilla CA program based on a judgement of their value
> to our users.

Right, but a judgement based on risk to users seems prudent.

> The most recent BR audit report for the Visa eCommerce Root contains 3
> qualifications: http://enroll.visaca.com/WTBR%20eComm.pdf

And one of these qualifications is for a critical component of the job:

> We were unable to obtain evidence of the domain validation documentation for a certificate issued.

Jonathan

[Paul Kehrer]

On February 14, 2018 at 4:17:16 AM, Wayne Thayer via dev-security-policy (

dev-secur...@lists.mozilla.org) wrote:

> The most recent BR audit report for the Visa eCommerce Root contains 3
qualifications: http://enroll.visaca.com/WTBR%20eComm.pdf

Does Mozilla have any guidelines or official position on what constitutes
sufficient audit issues to result in sanctions? Frankly I'm stunned that
any CA in the Mozilla root program can apparently ignore the baseline
requirements for approximately 4 years after their effective date, get an
initial BR audit with multiple qualifications, and see no penalty from this
behavior. And this is disregarding several other BR violations found in the
wild by independent researchers. I realize I'm banging the same drum as in
my other thread, but without consistent enforcement of escalating penalties
I don't believe we're teaching CAs anything other than that Mozilla will
ultimately forgive almost any transgression. Unless you catch them on a bad
day, in which case you might get distrusted entirely.

-Paul (reaperhulk)

[Wayne Thayer]

On Tue, Feb 13, 2018 at 11:26 PM, Paul Kehrer via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On February 14, 2018 at 4:17:16 AM, Wayne Thayer via dev-security-policy (
> dev-secur...@lists.mozilla.org) wrote:
>
> > The most recent BR audit report for the Visa eCommerce Root contains 3
> qualifications: http://enroll.visaca.com/WTBR%20eComm.pdf
>
> Does Mozilla have any guidelines or official position on what constitutes
> sufficient audit issues to result in sanctions?

As Gerv described in the other thread [1], Mozilla's current approach is to
document each issue and view them in aggregate, rather than defining a set
of penalties that apply in given situations. Mozilla has certainly required
actions from CAs as a condition to remaining in the program, but those
"sanctions" have been defined in the context of specific situations. While
I also find the idea of defining more generic penalties appealing on the
surface, I'm not convinced that it would lead to better outcomes for our
users.

Frankly I'm stunned that
> any CA in the Mozilla root program can apparently ignore the baseline
> requirements for approximately 4 years after their effective date, get an
> initial BR audit with multiple qualifications, and see no penalty from this
> behavior.

Their initial BR PITRA was in 2016. It lists 7 qualifications [2]

And this is disregarding several other BR violations found in the
> wild by independent researchers. I realize I'm banging the same drum as in
> my other thread, but without consistent enforcement of escalating penalties
> I don't believe we're teaching CAs anything other than that Mozilla will
> ultimately forgive almost any transgression. Unless you catch them on a bad
> day, in which case you might get distrusted entirely.
>

> In this particular case, my conclusion is that the existing Mozilla
process is working. We have documented a number of issues that when
considered in aggregate warrant an investigation.

- Wayne

[1]
https://groups.google.com/d/msg/mozilla.dev.security.policy/d-m48lVtYoQ/HvlXcfwWAQAJ
[2] https://bug1301210.bmoattachments.org/attachment.cgi?id=8795503

-Paul (reaperhulk)
>
>

[westm...@gmail.com]

It seems to me that some CA's hold unanswered Mozilla's questions because they know that it will not cause any serious consequences. I mean removing a root certificates from Mozilla Root Store. However, this point of view here seems to have already been voiced.

[Tim Smith]

On Wednesday, February 14, 2018 at 8:43:19 AM UTC-8, Wayne Thayer wrote:
> In this particular case, my conclusion is that the existing Mozilla
> process is working. We have documented a number of issues that when
> considered in aggregate warrant an investigation.

Hi Wayne,

Forgive me if I'm overinterpreting your comment, but: does this mean that an investigation is ongoing, or that Mozilla anticipates opening an investigation? If there is or will be an investigation, do you have a general sense of when to expect a result, and what that might look like?

Thanks,
Tim

[Wayne Thayer]

On Wed, Feb 14, 2018 at 10:47 AM, Tim Smith via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On Wednesday, February 14, 2018 at 8:43:19 AM UTC-8, Wayne Thayer wrote:

> > In this particular case, my conclusion is that the existing Mozilla
> > process is working. We have documented a number of issues that when
> > considered in aggregate warrant an investigation.
>

> Hi Wayne,
>
> Forgive me if I'm overinterpreting your comment, but: does this mean that
> an investigation is ongoing, or that Mozilla anticipates opening an
> investigation? If there is or will be an investigation, do you have a
> general sense of when to expect a result, and what that might look like?
>

> My comment means that I have acknowledged the issue and am looking into
it, but haven't yet committed Mozilla to a specific course of action or
timing.


> Thanks,
> Tim
>
>