Summary of Camerfirma's Compliance Issues

[Ben Wilson]

All,

We have prepared an issues list as a summary of Camerfirma's compliance
issues over the past several years. The purpose of the list is to collect
and document all issues and responses in one place so that an overall
picture can be seen by the community. The document is on the Mozilla wiki:
https://wiki.mozilla.org/CA:Camerfirma_Issues.
<https://wiki.mozilla.org/CA:Camerfirma_Issues>

I will now forward the link above to Camerfirma to provide them with a
proper opportunity to respond to the issues raised and to ask that they
respond directly in m.d.s.p. with any comments, corrections, inputs, or
updates that they may have.

If anyone in this group believes they have an issue appropriate to add to
the list, please send an email to certif...@mozilla.org.

Sincerely yours,
Ben Wilson
Mozilla Root Program

[Ryan Sleevi]

Hi Ben,

This is clearly a portrait of a CA that, like those that came before
[1][2][3][4], paint a pattern of a CA that consistently and regularly fails
to meet program requirements, in a way that clearly demonstrates these are
systemic and architectural issues.

As with Symantec, we see a systematic failure to appropriately supervise
RAs and Sub-CAs.
As with Procert, we see systemic technical failures continuing to occur. We
also see problematic practices here of revocations happening without a
systemic examination about why, which leads them to overlook incident
reports.
As with Visa, we see significant issues with their audits that are
fundamentally irreconcilable. As called out in [5] (Issue JJ), short of
distrusting their certificates, there isn't a path forward here.

I'm concerned that there's been no response from Camerfirma, even
acknowledging this publicly, as is the norm. I wanted to give a week, even
to allow for a simple acknowledgement, since Mozilla Policy requires that
CAs MUST follow and be aware of discussions here, above and beyond your
direct communication with them pointing this out.

Given that there haven't been corrections proposed yet, is it appropriate
to begin discussing what next steps should be to protect users?

[1] https://wiki.mozilla.org/CA:PROCERT_Issues
[2] https://wiki.mozilla.org/CA:Visa_Issues
[3] https://wiki.mozilla.org/CA:Symantec_Issues
[4] https://wiki.mozilla.org/CA:WoSign_Issues
[5] https://bugzilla.mozilla.org/show_bug.cgi?id=1583470#c3

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

[Ramiro Muñoz]

Hi Ryan

Camerfirma is working writing a complete answer to Ben comunicación. Ben has been informed from the first day. Hope we can send it in the next days.

Regards
Ramiro



Obtener Outlook para Android<https://aka.ms/ghei36>

________________________________
De: dev-security-policy <dev-security-...@lists.mozilla.org> en nombre de Ryan Sleevi via dev-security-policy <dev-secur...@lists.mozilla.org>
Enviado: jueves, 10 de diciembre de 2020 21:44
Para: Ben Wilson
Cc: mozilla-dev-security-policy
Asunto: Re: Summary of Camerfirma's Compliance Issues

[Ramiro Muñoz]

Hi Ben, Ryan and All.

Sorry for the delay in answering this communication in this channel. Even though we have been in contact with Ben from the very moment we were notified, a prompt answer to the community is convenient as Ryan said.

Camerfirma is working to deliver a detailed report to transmit our improvements in these years. Obviously, we were not succeeded to do it in the different bugs we have reported so far. We plan to send it to mdsp next Tuesday.

We have fixed most of the issues reported as we will try to explain, but we are aware that other persist despite the remediation measures adopted.

Summarizing:

. We have increased the PKI team to managing the bugs and administrative task to avoid delays in notifications, responses and CCADB management. We honestly think that we have made significant improvements in this area. Nevertheless, we plan to increase the resources this year to be more active in the community and understand deeply the BR and Mozilla requirement that have been root cause of some bugs.

. We have developed automations in the certificate issuing process, pre-issuing, and post-issuing. We plan to keep on doing this, like integrate ACME or develop a suspicions activity detection in our platform like certificate revocations with a short period of live time.

. We are rethinking the SubCA policy since some of the problems come from there. Camerfirma has increase the control imposing our 3 external SubCA to use the camerfirma central lint control in the pre-isuance process. However, next year we plan to convert external SubCA to Wite label CA, that's means to run them inside Camerfirma infrastructure. Regarding RA we already closed all the external RA for SSL certificates keeping only one inside our company group.

. We are working with our customers to face a certificate substitution process honouring the timeline requested by the Mozilla policy and the BR. We also work with the camerfirma internal high management to assume that some damage could be produced to our customers services because an unilateral revocation by the CA. This is the most difficult issue to manage and the only way to resolve it is avoiding mistakes. We think that the community should rethink the misissued revocation timeline requirement, at least to increase the number of user cases.

Finally, emphasize that in anyway Camerfirma try or have consciously mislead the community and our aim is to act transparently in order to be a valuable member of this community.

KR
Ramiro

-----Mensaje original-----
De: dev-security-policy <dev-security-...@lists.mozilla.org> En nombre de Ryan Sleevi via dev-security-policy
Enviado el: jueves, 10 de diciembre de 2020 21:44
Para: Ben Wilson <bwi...@mozilla.com>
CC: mozilla-dev-security-policy <mozilla-dev-s...@lists.mozilla.org>

[Ryan Sleevi]

Thanks Ramiro for the update.

I do want to make sure we're on the same page. Responding point-by-point to
the issues would probably be the least productive path forward. If there
are specific disagreements with the facts as presented, which were taken
from the Bugzilla reports, it would be good to highlight that. However, I
believe the intent is that the Bugzilla report is the source of truth, so
if there are details that were *not* on the incident reports, I would say
that's more concerning than it is reassuring.

I'm a bit concerned to see your latest reply still highlight the "increased
the PKI team", since that's been a sort of repeated refrain (e.g. Issue T,
Issue BB, Issue PP). I don't disagree that it's important to ensure that
there are adequate resources devoted to compliance _and_ development, but I
think it's also important to make sure that the solution is not to throw
more people at the problem.

While the integrity of the CA is perhaps not as obviously cut and dry, as
was the case with WoSign/StartCom, I do believe it's reasonable to ask
whether or not the CA has the skills, expertise, and understanding of the
systemic issues to effectively manage things going forward, especially when
we have seen the regular repetition of issues. This suggests more systemic
fixes are needed, but also suggests that there are more systemic flaws in
how things are operated, designed, and administered that do call into
question the trustworthiness of the current infrastructure.

If Camerfirma were approaching with a new CA/new certificate hierarchy, it
would be reasonable to ask why, given all of these incidents, Camerfirma
should be included, since it puts a lot of risk onto the community.
Regaining trust and reputation, once lost, is incredibly difficult, and
requires going above and beyond. I hope that, in your formal response, you
provide a holistic picture of how Camerfirma has changed its operations,
implementation, infrastructure, management process, policies, and quite
frankly, the use cases/subscribers that it targets with these certificates,
so that the community can make an informed decision about risk.

Similarly, in thinking about what this would be for a "new" PKI, and how
difficult it would be, given the current evidence, to see that as good for
users, it's also worth asking what should happen with the current PKI.
Should we continue to assume that the implementation of the EV guidelines
is correct, even though we have ample evidence of basic RFC 5280 and BR
violations? Should we consider sunsetting (e.g. with a notBefore
restriction) trust? Would it be reasonable to remove trust altogether?

For example, Camerfirma currently has four roots ([1], [2], [3], [4]). [1]
appears to have issued less than 3,500 still valid certificates, [2] / [3]
are only trusted for e-mail, and [4] seems to be a super-CA of sorts (with
sub-CAs operated by Intesa Sanpaolo, Infocert, Multicert, and Govern
d'Andorra). For the sub-CAs that aren't constrained/revoked, it seems
Intesa Sanpaolo has only issued ~2200 certificates, Infocert ~650, and
Multicert ~3000.

Is that accurate? I totally could have made a mistake here, since this was
just manually inspecting every sub-CA from Camerfirma and I totally could
have clicked wrong, but that suggests that there is... very little
practical risk here to removal, compared to the rather significant risk of
having a CA that has established a pattern of compliance and supervision
issues.

[1] https://crt.sh/?caid=869
[2] https://crt.sh/?caid=140
[3] https://crt.sh/?caid=8453
[4] https://crt.sh/?caid=1114

[Ramiro Muñoz]

Hi Ryan

Thanks. We will take into account your observations.
As I said we planed to send the formal answer today but the team has asked me for more time in order to give a more accurate answer. We plan to postpone to this Friday.

KR
Ramiro

De: Ryan Sleevi <ry...@sleevi.com>
Enviado el: lunes, 14 de diciembre de 2020 22:41
Para: Ramiro Muñoz <ram...@camerfirma.com>
CC: ry...@sleevi.com; Ben Wilson <bwi...@mozilla.com>; mozilla-dev-security-policy <mozilla-dev-s...@lists.mozilla.org>
Asunto: Re: Summary of Camerfirma's Compliance Issues

[Burton]

It doesn't look great to the community when a CA that is under
investigation for serious compliance issues asks for more time to provide
detailed answers.

Also you said 'accurate answers' ? Were the answers you were going to post
today inaccurate in some way?

Burton

[Ramiro Muñoz]

I really sorry for the Delay Burton.

Obviously we do not intend to send inaccurate answers. Maybe I was not using the right wording may be I should use “precise” instead ? sorry for my English language limitation. Your complain about my email prove that some misunderstanding problem could have happen also in our bugs reports. We want to be sure that the answers are fully understand to give to the community the complete information to evaluate them. As you can imagine in the situation we are, it’s critical for us. That’s the reason.

Thanks
Ramiro

De: Burton <j...@0.me.uk>
Enviado el: martes, 15 de diciembre de 2020 19:39
Para: Ramiro Muñoz <ram...@camerfirma.com>
CC: ry...@sleevi.com; mozilla-dev-security-policy <mozilla-dev-s...@lists.mozilla.org>; Ben Wilson <bwi...@mozilla.com>
Asunto: Re: Summary of Camerfirma's Compliance Issues

It doesn't look great to the community when a CA that is under investigation for serious compliance issues asks for more time to provide detailed answers.

Also you said 'accurate answers' ? Were the answers you were going to post today inaccurate in some way?

Burton

On Tue, Dec 15, 2020 at 6:13 PM Ramiro Muñoz via dev-security-policy <dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>> wrote:
Hi Ryan

Thanks. We will take into account your observations.
As I said we planed to send the formal answer today but the team has asked me for more time in order to give a more accurate answer. We plan to postpone to this Friday.

KR
Ramiro

De: Ryan Sleevi <ry...@sleevi.com<mailto:ry...@sleevi.com>>
Enviado el: lunes, 14 de diciembre de 2020 22:41
Para: Ramiro Muñoz <ram...@camerfirma.com<mailto:ram...@camerfirma.com>>
CC: ry...@sleevi.com<mailto:ry...@sleevi.com>; Ben Wilson <bwi...@mozilla.com<mailto:bwi...@mozilla.com>>; mozilla-dev-security-policy <mozilla-dev-s...@lists.mozilla.org<mailto:mozilla-dev-s...@lists.mozilla.org>>
Asunto: Re: Summary of Camerfirma's Compliance Issues

Thanks Ramiro for the update.

I do want to make sure we're on the same page. Responding point-by-point to the issues would probably be the least productive path forward. If there are specific disagreements with the facts as presented, which were taken from the Bugzilla reports, it would be good to highlight that. However, I believe the intent is that the Bugzilla report is the source of truth, so if there are details that were *not* on the incident reports, I would say that's more concerning than it is reassuring.

I'm a bit concerned to see your latest reply still highlight the "increased the PKI team", since that's been a sort of repeated refrain (e.g. Issue T, Issue BB, Issue PP). I don't disagree that it's important to ensure that there are adequate resources devoted to compliance _and_ development, but I think it's also important to make sure that the solution is not to throw more people at the problem.

While the integrity of the CA is perhaps not as obviously cut and dry, as was the case with WoSign/StartCom, I do believe it's reasonable to ask whether or not the CA has the skills, expertise, and understanding of the systemic issues to effectively manage things going forward, especially when we have seen the regular repetition of issues. This suggests more systemic fixes are needed, but also suggests that there are more systemic flaws in how things are operated, designed, and administered that do call into question the trustworthiness of the current infrastructure.

If Camerfirma were approaching with a new CA/new certificate hierarchy, it would be reasonable to ask why, given all of these incidents, Camerfirma should be included, since it puts a lot of risk onto the community. Regaining trust and reputation, once lost, is incredibly difficult, and requires going above and beyond. I hope that, in your formal response, you provide a holistic picture of how Camerfirma has changed its operations, implementation, infrastructure, management process, policies, and quite frankly, the use cases/subscribers that it targets with these certificates, so that the community can make an informed decision about risk.

Similarly, in thinking about what this would be for a "new" PKI, and how difficult it would be, given the current evidence, to see that as good for users, it's also worth asking what should happen with the current PKI. Should we continue to assume that the implementation of the EV guidelines is correct, even though we have ample evidence of basic RFC 5280 and BR violations? Should we consider sunsetting (e.g. with a notBefore restriction) trust? Would it be reasonable to remove trust altogether?

For example, Camerfirma currently has four roots ([1], [2], [3], [4]). [1] appears to have issued less than 3,500 still valid certificates, [2] / [3] are only trusted for e-mail, and [4] seems to be a super-CA of sorts (with sub-CAs operated by Intesa Sanpaolo, Infocert, Multicert, and Govern d'Andorra). For the sub-CAs that aren't constrained/revoked, it seems Intesa Sanpaolo has only issued ~2200 certificates, Infocert ~650, and Multicert ~3000.

Is that accurate? I totally could have made a mistake here, since this was just manually inspecting every sub-CA from Camerfirma and I totally could have clicked wrong, but that suggests that there is... very little practical risk here to removal, compared to the rather significant risk of having a CA that has established a pattern of compliance and supervision issues.

[1] https://crt.sh/?caid=869
[2] https://crt.sh/?caid=140
[3] https://crt.sh/?caid=8453
[4] https://crt.sh/?caid=1114
_______________________________________________
dev-security-policy mailing list

dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>
https://lists.mozilla.org/listinfo/dev-security-policy

[Ramiro Muñoz]

Hi Ben, Ryan, Burton and all:

Camerfirma will present its claims based on a description of the problems found by associating the references to the specific bugs.
After making a complete analysis of the bugs as presented by Ben, always considering that bugs are the main source of truth, we see that the explanations offered by Camerfirma could generally be better developed. We hope to make up for these deficiencies with this report.
We have included a list of issues that we consider to be fully addressed in the Appendix I: State of the fixed issues.

We will classify the issues in different categories:
• SUBCA SUPERVISION
• REVOCATION DELAYS
• TECHNICAL ISSUES AND AUTOMATISMS
1.- SUBCA SUPERVISION

MULTICERT, INFOCERT, INTESA SANPAOLO.
Issue H: Non-compliant OCSP Responders by Third-Party Subordinates (Dec. 2017)
Issue R: Failure to disclose unconstrained sub-CA (DigitalSign) (2018)
Issue T: Failure to disclose unconstrained sub-CA (MULTICERT) (2018 - 2020)
Issue X: MULTICERT Misissuance (2018 - 2019)
Issue Z: Intesa Sanpaolo Misissuance (2017 - 2020)
Issue BB: Failure to revoke underscores (2019)
Issue DD: Infocert Misissuance (2017 - 2020)
Issue PP: Failure to disclose unconstrained Sub-CA (Government of Andorra) (2013 - 2019)
Issue RR: Failure to disclose unconstrained Sub-CA (Intesa Sanpaolo and Infocert) (2018 - 2020)
Issue TT: Certificate with Incorrect OrganizationName (Nov. 2020)

In addition to the following requirement stated in the Mozilla policy and just in place
• Requirement of pointing in time audit (PIT) and report (at the beginning of each new CA)
• Requirement of the annual audits and report (from the creation of each Intermediate CA)
We are currently carrying out additional controls on the activities of the organizations that manage intermediate CAs with different implementation time frames:
• Adoption of a centralised LINTS (the same one used by Camerfirma) by all intermediate CAs before issuing certificates. (March 2019 Multicert and April 2019 Infocert e Intesa SanPaolo)
• Contractual reinforcement of Camerfirma's rights with regards to the activities carried out by Intermediate CAs and their obligation to a periodic communication. (October 2019).
• Contractual rights and tools for Camerfirma to insource, when deemed appropriate, intermediate CAs operational activities in order to be able to apply all needed (and already implemented for Camerfirma CAs) controls and to force certificate revocation in a timely manner.
Planned changes:
• Stop the issuance of certificates. (January 2021)
• Implementation of Intermediate CA PIT. (January 2021)
• Audit by Camerfirma of 100% of the active certificates issued (Task carry out during January 2021)
• Change in the audit process. We are currently requesting the audit report to be issued by a recognised auditor. The new process will require the audit to be carried out by an auditor selected by Camerfirma. (All new audits from January 2021)
• Technical environment set-up and procedural definition to be able to insource the management of operational activities of intermediate CAs by the first half of 2021
3.-REVOCATION DELAYS

Issue D: Duplicate subjectAlternativeNames and incorrect Subject fields (April 2017)
Issue J: Invalid DNS names within certificates (August 2017)
Issue L: Invalid subjectAlternativeName within certificates (July 2017)
Issue N: Improper issuance and failure to revoke intranet certificates (2015 - 2017)
Issue X: MULTICERT Misissuance (2018 - 2019)
Issue Z: Intesa Sanpaolo Misissuance (2017 - 2020)
Issue BB: Failure to revoke underscores (2019)
Issue PP: Failure to disclose unconstrained Sub-CA (Government of Andorra) (2013 - 2019)
Issue DD: Infocert Misissuance (2017 - 2020)
Issue LL: Invalid authorityKeyIdentifier (2003 - 2020)
Issue NN: Incorrect OCSP Delegated Responder Certificate (2013 - 2020)
We face the problem when the customers ask more time to complete certificates substitution in their own applications.
Controls already in place:
• All our External Intermediate CAs and clients have accepted new general terms and condition allowing Camerfirma to revoke all the problematic certificates without their permission if necessary (October 2019). There are not so many problems in reissuing some hundreds of certificates in a couple of days, the problem is awaiting critical customers activities (for example Intesa Sanpaolo one of the largest banks in Europe) before being able to revoke the misissued certificates.
• We have developed and started using a massive revocation and substitution tool to be more effective in that process. (June 2020).

After implementing all those measures, we have noticed that they were not enough to comply with the required deadlines, and we are planning to incorporate the following additional measures:

• Implement ACME to control the revocations and substitution automatically (planned for June 2021 for Camerfirma infrastructure and to be designed for the Intermediate CAs)
• Limit the number of DNS names that can appear in a certificate to make the substitution easier (planned for March 2021 for Camerfirma infrastructure and to be designed for the Intermediate CAs)
• Have the contractual right and the operational procedures in place to insource the management of all the operational activities of the intermediate CAs (June 2021)

Only In some specific cases where a fast revocation could caused much more damages (to the impacted client) than benefits (to the entire community) in those cases we asked the community for some extra time.

4.-TECHNICAL ISSUES AND AUTOMATISMS

Issue D: Duplicate subjectAlternativeNames and incorrect Subject fields (April 2017)
Issue H: Non-compliant OCSP Responders by Third-Party Subordinates (Dec. 2017)
Issue J: Invalid DNS names within certificates (August 2017)
Issue L: Invalid subjectAlternativeName within certificates (July 2017)
Issue P: Invalid characters within the OU field (2018)
Issue X: MULTICERT Misissuance (2018 - 2019)
Issue Z: Intesa Sanpaolo Misissuance (2017 - 2020)
Issue BB: Failure to revoke underscores (2019)
Issue DD: Infocert Misissuance (2017 - 2020)
Issue NN: Incorrect OCSP Delegated Responder Certificate (2013 - 2020)
Issue LL: Invalid authorityKeyIdentifier (2003 - 2020)
Issue UU: Certificate for unregistered domain (Oct. 2020)

Regarding the automations, these are currently implemented:
• Control of the DNS and Email domain (August 2020)
• CT Control (April 2017)
• Control cablint, x509lint and zlint (pre-issuance - post-issuance) (January 2019)
• Syntax control of the domain (August 2020)
• Control of black and white lists of domains (August 2020)
• Automatic verification of CAA (June 2020)
• Additional automatic control to verify the correction of AKI extension before certificate issuance (April 2020)

New controls:
• Control of suspicious activity patterns. (March 2021)
• Remove fields as OU in the profile and avoiding other manual filling fields not validated in the certificate content. We have an exception for Spanish Public Administration Requirements where the field OU is mandatory for Spanish CAs (discussion https://github.com/cabforum/servercert/pull/225 ) (from January 2021).

Besides, in order to be as transparent as possible, each time we discover a new certificate misissued, we will review our internal procedure to make sure that those situations will always be promptly disclosed in https://misissued.com (for all new certificates detected from now on).
Other controls for the incorporation of correct information in the fields that automatic controls cannot detect, for example:

- Issue HH: EV Certificates with wrong businessCategory (2018 - 2019)
BUG 1600114 Information that shall be included in the BusinessCategory field of the EV certificates. If some EV certificates were issued with wrong values (there are four values that can be used)
- Issue D: Duplicate subjectAlternativeNames and incorrect Subject fields (April 2017)
BUG 1667430 Interpretation of what values can place in the stateOrProvinceName field
Audit process is the most effective answers to solve such residual problems. We will audit 100% of the certificates issued to detect wrong values in all the certificate’s fields (100% of certificates issued till January 2021).
Additionally, we must consider the importance of dedicated resources, because it has been an important aspect that has influenced past registered issues.
We think that thanks to the increased team, as described before, we have achieved a better issue and compliance management. For the future, our objective is to have a more proactive attitude to avoid incidents instead of a reactive action to quickly fix them.
Adding more employees, by itself, cannot be considered a solution. But the origin of some of the issues is a poor follow-up. This is the case with bugs, audits, activities of Intermediate CAs or matters concerning the community. Therefore, we believe that a proper sizing of the team can be considered an important structural change. Right sizing the department will improve operational and administrative management processes. Coupled with the automation of the processes to minimize manual operations, this will give us the quality we aim for.

Planned Changes:
• The exclusive dedication of the quality area in Camerfirma to the management of SSL certificates reinforced with a new recruitment to check 100% SSL certificates issued. (January 2021)
• Increase resources exclusively dedicated during the next year to the creation and maintenance of automatisms in the process of generating SSL certificates. January (2021)

Appendix I: State of the issues

Closed and remediated issues:
- Issue B: Delegation of validation to StartCom following Mozilla's distrust of StartCom (March 2017)
- Issue F: Ignoring CAA based on another CA's Certificate Transparency disclosure (Nov. 2017)
- Issue P: Invalid characters within the OU field (2018).
- Issue V: Audit Qualifications (2017 - 2018).
- Issue H: Non-compliant OCSP Responders by Third-Party Subordinates (Dec. 2017).
- Issue LL: Invalid authorityKeyIdentifier (2003 - 2020).

Closed but not considered remediated issues:
- Issue JJ: Unresolvable Gap in Audits (Camerfirma) (2018 - 2019).
- Issue FF: Intentional unrevocation of externally-operated sub-CA (2019).
Regards
Ramiro.

[Ramiro Muñoz]

Besides, in order to be as transparent as possible, each time we discover a new certificate misissued, we will review our internal procedure to make sure that those situations will always be promptly disclosed in https://misissued.com<https://misissued.com/> (for all new certificates detected from now on).

Other controls for the incorporation of correct information in the fields that automatic controls cannot detect, for example:

* Issue HH: EV Certificates with wrong businessCategory (2018 - 2019)

BUG 1600114 Information that shall be included in the BusinessCategory field of the EV certificates. If some EV certificates were issued with wrong values (there are four values that can be used)

* Issue D: Duplicate subjectAlternativeNames and incorrect Subject fields (April 2017)

BUG 1667430 Interpretation of what values can place in the stateOrProvinceName field
Audit process is the most effective answers to solve such residual problems. We will audit 100% of the certificates issued to detect wrong values in all the certificate’s fields (100% of certificates issued till January 2021).
Additionally, we must consider the importance of dedicated resources, because it has been an important aspect that has influenced past registered issues.
We think that thanks to the increased team, as described before, we have achieved a better issue and compliance management. For the future, our objective is to have a more proactive attitude to avoid incidents instead of a reactive action to quickly fix them.
Adding more employees, by itself, cannot be considered a solution. But the origin of some of the issues is a poor follow-up. This is the case with bugs, audits, activities of Intermediate CAs or matters concerning the community. Therefore, we believe that a proper sizing of the team can be considered an important structural change. Right sizing the department will improve operational and administrative management processes. Coupled with the automation of the processes to minimize manual operations, this will give us the quality we aim for.

Planned Changes:

• The exclusive dedication of the quality area in Camerfirma to the management of SSL certificates reinforced with a new recruitment to check 100% SSL certificates issued. (January 2021)

• Increase resources exclusively dedicated during the next year to the creation and maintenance of automatisms in the process of generating SSL certificates. January (2021)


Appendix I: State of the issues

Closed and remediated issues:

* Issue B: Delegation of validation to StartCom following Mozilla's distrust of StartCom (March 2017)
* Issue F: Ignoring CAA based on another CA's Certificate Transparency disclosure (Nov. 2017)
* Issue P: Invalid characters within the OU field (2018).
* Issue V: Audit Qualifications (2017 - 2018).
* Issue H: Non-compliant OCSP Responders by Third-Party Subordinates (Dec. 2017).
* Issue LL: Invalid authorityKeyIdentifier (2003 - 2020).

Closed but not considered remediated issues:

* Issue JJ: Unresolvable Gap in Audits (Camerfirma) (2018 - 2019).
* Issue FF: Intentional unrevocation of externally-operated sub-CA (2019).
Regards
Ramiro.

[Wayne Thayer]

On Sat, Dec 19, 2020 at 1:03 AM Ramiro Muñoz via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Hi Ben, Ryan, Burton and all:
>
> Camerfirma will present its claims based on a description of the problems
> found by associating the references to the specific bugs.
> After making a complete analysis of the bugs as presented by Ben, always
> considering that bugs are the main source of truth, we see that the
> explanations offered by Camerfirma could generally be better developed. We
> hope to make up for these deficiencies with this report.
>
>

It's worth pointing out that in April 2018, the Camerfirma '2016 roots'
inclusion request [1] was denied [2] after a host of issues were
documented. At that time it was made clear that ongoing trust in the older
roots was in jeopardy [3]. While some progress was made, the number,
severity, and duration of new and ongoing bugs since then remains quite
high. In this context, I don't find these new disclosures and commitments
from Camerfirma to form a convincing case for their trustworthiness.

- Wayne

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=986854
[2]
https://groups.google.com/g/mozilla.dev.security.policy/c/skev4gp_bY4/m/snIuP2JLAgAJ
[3]
https://groups.google.com/g/mozilla.dev.security.policy/c/skev4gp_bY4/m/ZbqPhO5FBQAJ

[Ramiro Muñoz]

Hi Wayne

I understand your concern but, Camerfirma has indeed achieved huge improvements in terms of Mozilla’s policy compliance during recent years. Camerfirma nowadays has a much more mature management system. It’s true, some bugs have occurred but, looking at the bugs dashboard, our situation cannot be considered very different from other CAs. We firmly believe that the improvements already implemented together with the proposed measures will strengthen the governance of our SSL certificate activities in a very impactful and lasting way. In that regard, it’s important to highlight that we have the full support of our top management - both at the company level as well as at InfoCert Group level - in making everything will be required in order to come out successfully from this unpleasant situation.

[Ryan Sleevi]

On Mon, Dec 28, 2020 at 6:35 AM Ramiro Muñoz via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> El miércoles, 23 de diciembre de 2020 a las 0:01:23 UTC+1, Wayne Thayer
> escribió:

> Hi Wayne
>
> I understand your concern but, Camerfirma has indeed achieved huge
> improvements in terms of Mozilla’s policy compliance during recent years.
> Camerfirma nowadays has a much more mature management system. It’s true,
> some bugs have occurred but, looking at the bugs dashboard, our situation
> cannot be considered very different from other CAs.

So there's three specific claims here, as to why serious consideration of
distrust isn't warranted:

1. Camerfirma has made huge improvements
2. Camerfirma nowadays has a much more mature management system.
3. Camerfirma is not very different from other CAs.

These statements are ones that are sort of "true by degree". That is, if I
was to dispute 1, Camerfirma would/could rightfully point out that they
were *much* worse before, and so yes, it's true that they've improved.
Similarly, to point out at how laughably bad the old system was does show
that there is a degree of truth in 2. And, as I laid out in my own post,
Camerfirma *is* not very different from other CAs - CAs that have been
distrusted, for not very different reasons than Camerfirma. I'm sure
Camerfirma meant to mean "not much different than other *currently trusted*
CAs", but that's equally a degree of truth - many individual incidents
affected other CAs, even though the sheer volume *and nature* of Camerfirma
bugs is troubling.

This is an issue of judgement here, about whether or not the degree of
truth to these statements adequately reflects the very risk that continued
trust in Camerfirma poses. The sheer volume of bugs do help paint a
trendline, which is what Camerfirma is arguing here, but just there's a big
difference between y = x + x, y = x * x, and y = x ^ x, there's a big
difference in the nature of the incidents, the quality of response, and the
(lack of) a meaningful rate of improvement that don't really inspire
confidence. Similarly, the risk in removing trust is exceedingly low, which
helps show the "risk to current users" (from trusting) versus the "risk of
breaking things" (by distrusting) is not a major consideration.

I would be curious if folks believe there is evidence that is being
overlooked from the Wiki, or believe that there is a different perspective
that can be reached from that data, and if they'd like to show how and why
they reached that conclusion. I've shared my perspective, but value
learning more if others do see things differently.

[Ramiro Muñoz]

In response to Ryan’s latest post, we want to provide the community with Camerfirma’s due responses and we hope this clears up any doubts that might have arisen.

Ryan argument number 1: “These statements are ones that are sort of "true by degree". That is, if I was to dispute 1, Camerfirma would/could rightfully point out that they were *much* worse before, and so yes, it's true that they've improved.”
1. Camerfirma has made huge improvements.

Camerfirma has improved its operations to avoid errors. We have procedures in place to incentivate improvement in every level in our operations. We shall continue to work in this way in the future.
We have worked to automate internal processes, also improving the management and level of attention on each incident. We have involved more experts in the process. Our goal has always been to meet the highest demands, including monitoring of CA activities that Mozilla has been implementing over the past years.
In addition, Camerfirma publishes SSL certificates in the CT, EV since (May 2016) and OV since (April 2017) ( even if it was mandatory only from April 30, 2018). We have always been in favour of increasing transparency and providing useful information to the community.
We have implemented several improvements during 2019 and 2020 (as we have already mentioned in previous reports):

• Decreased response time and increased attention to incidents. As a result, we have eliminated communications failures and we have avoided response delays (2020).
• Use of a centralized lint. Our three unconstrained subCA (Multicert, Infocert and Intesa Sao Paolo) with codification errors in issued certificates were added to our centralized lint. The first one since March 2019 and the other two since April 2019.
• Contractual cover of unilateral revocations with the subCAs has been clarified and streamlined. (October 2019).
• Mass revocation processes have been implemented (June 2020).
• Verification of CAA has been revised and reinforced with automatic procedures (June 2020).
• Control zlint has been implemented, both at pre-issuance and post-issuance (January 2019).

• Syntax control of the domain (August 2020)

• Additional automatic control to verify the correctness of AKI extension before certificate issuance has been implemented (April 2020).

In addition, Camerfirma periodically evaluates the efficacy of these measures and every other improvement implemented.

Ryan argument number 2: “Similarly, to point out at how laughably bad the old system was does show that there is a degree of truth in 2”

2. Camerfirma nowadays has a much more mature management system.

Subjective opinions aside, the community bug reports from the last 4 years show the improvement and efficacy of controls in Camerfirma's systems and procedures:

CAMERFIRMA InfoCert MULTICERT DIGITALSIGN CGCOM TOTAL
BUGS 2020 1 3 2 0 1 7
BUGS 2019 5 1 1 0 0 7
BUGS 2018 4 1 2 1 0 8
BUGS 2017 5 1 0 0 0 6
TOTAL 15 6 5 1 1 28

Regarding the nature of the errors, the bug associated with Camerfirma’ s systems in the last year (2020) was:
• #1623384 AKI issue encountered due to the incomplete templates in the certificate model. The modification of the profile correction procedure and establishment of an additional automatic control to verify the AKI before the issue of certificates was implemented in a timely manner.

If we consider also bugs concerning the external SubCAs, the total number of bugs registered in 2020 is in line with the previous years. In order to extend to subCAs the same rate of improvement registered by Camerfirma we’ve proposed to obtain the contractual right and the operational procedures in place to insource the management of all the operational activities of subCAs.

Ryan argument number 3: “…and, as I laid out in my own post, Camerfirma *is* not very different from other CAs - CAs that have been distrusted, for not very different reasons than Camerfirma. I'm sure Camerfirma meant to mean "not much different than other *currently trusted* CAs", but that's equally a degree of truth - many individual incidents affected other CAs, even though the sheer volume *and nature* of Camerfirma bugs is troubling.
3. Camerfirma is not very different from other (trusted) CAs.

Obviously, when we state that the reported errors do not differ from other CAs, we mean that we see in those trusted CAs a situation similar to the one we have stated. Stating otherwise is a strawman fallacy.
In no way Camerfirma could be compared with more critical cases such as WoSign or Diginotar, as it might be implied in Ryan messages. These messages risk to give a misleading view to the entire community. The WoSign ot Diginotar incidents had direct effects on the security of the system with the possibility of issuing fraudolent certificates with devastating consequences. Those sort of incidents cannot – and shall not - be related at all with Camerfirma’s situation.
It is obvious that some unfair comments give an extremely negative image of our company, implying lack of collaboration or willful deception to community requests that do not represent the real situation.

Ryan argument number 4: “This is an issue of judgement here, about whether or not the degree of truth to these statements adequately reflects the very risk that continued trust in Camerfirma poses. The sheer volume of bugs do help paint a trendline, which is what Camerfirma is arguing here, but just there's a big difference between y = x + x, y = x * x, and y = x ^ x, there's a big difference in the nature of the incidents, the quality of response, and the (lack of) a meaningful rate of improvement that don't really inspire confidence”.

Camerfirma does not agree that the trend and nature of the bugs were different, nor that our bugs are multiplying or enhancing rather than adding up. We have always been open and transparent, but we shall try to be even more so.

Ryan argument number 5: “…similarly, the risk in removing trust is exceedingly low, which helps show the "risk to current users" (from trusting) versus the "risk of breaking things" (by distrusting) is not a major consideration.”

We do not understand the risk assessment carried out here; it seems to be based on the aphorism "not too big to fall" which is not a fair or realistic parameter in terms of the possibility of evaluating incidents according to the number of certificates issued by a CA. Stating that the number of certificates issued would be a parameter when deciding if a CA should be distrusted, could lead to favouring monopolistic business practices.

Risk is a probabilistic parameter to be used with the impacts produced.

In Camerfirma’s case, a certificate with a wrong character in the Organization name, or a problem with an audit date has been issued, but never was a serious impact produced that put the community at risk. We do not mean to ignore these incidents but, simply to take into consideration the real impact/danger they’ve produced on the community. As stated before, in the WoSign or Diginotar cases, we have had misissued certificates capable of replacing the identity of public website with serious and potentially devastating impacts, which by no means could compare with Camerfirma case.
Accepting any phrase such as “ the risk in removing trust is exceedingly low” without true statistical analysis could become a precedent towards the elimination of more CAs in the ecosystem even when striving to uphold the best business practices.

Ryan argument number 6. “I would be curious if folks believe there is evidence that is being overlooked from the Wiki, or believe that there is a different perspective that can be reached from that data, and if they'd like to show how and why they reached that conclusion. I have shared my perspective, but value learning more if others do see things differently.”

Camerfirma appeals to the community to fairly evaluate the number and the nature of incidents reported as well as the improvements made by Camerfirma to maintain the trust in our certificates. In addition to the above mentioned actions already completed, Camerfirma presents a series of 10 measures -already mentioned in our previous communications- that will further reinforce transparency and quality in the life cycle management processes of certificates, while maintaining strict compliance with BR and Mozilla policies. We highlight in brackets the dates already planned for putting them into production.

1. Control of outlier activity patterns (March 2021).
2. Replace information not expressly validated in the certificate attributes with standardized values to avoid human errors. (March 2021)
3. Use of https://misissued.com (from January 2021)
4. Specific enhancements to the Business Category and StateofProvinceName fields that assist the RA requester and operator. (January 2021)
5. Audit of the totality (100%) of the certificates issued during January-february 2021.
6. Dedication of exclusive resources for the issuance of SSL certificates for compliance activities. (March 2021)
7. Proactive activities in the CA/B forum WG. (February 2021)
8. Exclusive dedication of developers on the improvement of automatic controls in the process of managing the life cycle of SSL certificates. Implementation of ACME for automatic certificate replacement (June 2021).
9. Limit the number of DNS names that can appear in a certificate to make the substitution easier (planned for March 2021 for Camerfirma infrastructure and to be designed for the Intermediate CAs).
10. Insource the management of all the operational activities of the intermediate CAs (June 2021).

These controls will reinforce transparency in the life cycle management processes of the certificates issued, while maintaining strict compliance with BR and Mozilla policies.
We hope to have addressed in depth the doubts that Ryan stated in his latest contribution, and are available to further clarification in case anyone would like to comment.

[Ryan Sleevi]

On Tue, Jan 5, 2021 at 9:01 AM Ramiro Muñoz via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> In response to Ryan’s latest post, we want to provide the community with
> Camerfirma’s due responses and we hope this clears up any doubts that might
> have arisen.
>

> Ryan argument number 1: “These statements are ones that are sort of "true

> by degree". That is, if I was to dispute 1, Camerfirma would/could
> rightfully point out that they were *much* worse before, and so yes, it's

I understand why Camerfirma feels it important to exhaustively list these
things, but I think the Wiki page, and its details, provides a much more
honest look at these. The adoption of Certificate Transparency before it
was mandatory, for example, does not highlight Camerfirma's leadership in
the area: it reveals how many issues Camerfirma's own management was
letting through its quality control processes, even though tools readily
existed to prevent this. The centralized lints for sub-CAs, for example,
were not imposed proactively to prevent incidents, but reactively in
response to a failure to supervise sub-CAs. Each of these improvements you
list, while there are some improvements, have been reactive in nature,
after Camerfirma has been shown to repeatedly fail to meet the basic
expectations of CAs, fail to detect this, and have the community highlight
it.

Perhaps no greater example of this can be found than "Syntax control of the
domain", implemented in August 2020, despite issues having been raised in
2017 highlighting the importance of this requirement. [1]

What Camerfirma has done here has list the controls they've implemented in
response to their specific incidents, which, while important, overlooks
that many of these were basic expectations that would have prevented
incidents. This is akin to a student asking for full credit for a paper
they turned in a semester late, on the principle that they at least turned
it in, even after their (failing) grade had already been received.

[1]
https://groups.google.com/g/mozilla.dev.security.policy/c/D0poUHqiYMw/m/Pf5p0kB7CAAJ


>
> Ryan argument number 2: “Similarly, to point out at how laughably bad the
> old system was does show that there is a degree of truth in 2”

> 2. Camerfirma nowadays has a much more mature management system.
>

Alternatively, this highlights that, throughout the years, Camerfirma has
failed to appropriately supervise sub-CAs.


>
> Ryan argument number 3: “…and, as I laid out in my own post, Camerfirma

> *is* not very different from other CAs - CAs that have been distrusted, for
> not very different reasons than Camerfirma. I'm sure Camerfirma meant to
> mean "not much different than other *currently trusted* CAs", but that's
> equally a degree of truth - many individual incidents affected other CAs,
> even though the sheer volume *and nature* of Camerfirma bugs is troubling.

> 3. Camerfirma is not very different from other (trusted) CAs.
>
> Obviously, when we state that the reported errors do not differ from other
> CAs, we mean that we see in those trusted CAs a situation similar to the
> one we have stated. Stating otherwise is a strawman fallacy.
> In no way Camerfirma could be compared with more critical cases such as
> WoSign or Diginotar, as it might be implied in Ryan messages. These
> messages risk to give a misleading view to the entire community. The WoSign
> ot Diginotar incidents had direct effects on the security of the system
> with the possibility of issuing fraudolent certificates with devastating
> consequences. Those sort of incidents cannot – and shall not - be related
> at all with Camerfirma’s situation.
> It is obvious that some unfair comments give an extremely negative image
> of our company, implying lack of collaboration or willful deception to
> community requests that do not represent the real situation.
>

Absolutely, Camerfirma is, subjectively, as bad or worse than WoSign and
DigiNotar right here. Camerfirma's focus on "Well, nothing _really_ bad
happened here" underscores exactly the risk of continued trust in
Camerfirma. What we have here is an undeniable pattern of issues, from a
failure to understand requirements, a failure to supervise subordinates,
and a failure to implement appropriate controls. At the core, all of these
issues similar to those faced by WoSign and DigiNotar. DigiNotar's biggest
mistake was not getting compromised, for example, but the failure to report
that compromise in a timely fashion that could have allowed immediate
action to take place. From Camerfirma's incidents, we see that same hubris
and misunderstanding of what it means to be a trusted CA. We see a failure
to timely detect and report issues, from both Camerfirma and its
subordinates, and a failure to appropriately design systems robust to
ensure prompt action can be taken. Indeed, Camerfirma's argument here is
that, despite repeated incidents year over year, they're "finally" in a
place to now observe the Baseline Requirements as they existed in 2015.
This is not something we should praise Camerfirma for.

Ryan argument number 5: “…similarly, the risk in removing trust is

> exceedingly low, which helps show the "risk to current users" (from
> trusting) versus the "risk of breaking things" (by distrusting) is not a

> major consideration.”
>
> We do not understand the risk assessment carried out here; it seems to be
> based on the aphorism "not too big to fall" which is not a fair or
> realistic parameter in terms of the possibility of evaluating incidents
> according to the number of certificates issued by a CA. Stating that the
> number of certificates issued would be a parameter when deciding if a CA
> should be distrusted, could lead to favouring monopolistic business
> practices.


> Risk is a probabilistic parameter to be used with the impacts produced.

The risk Mozilla, and any other root program, must contend with, is the
risk to their users. Camerfirma's practices reveal poor management, poor
incident response, poor understanding of expectations, and poor awareness
of industry trends and practices, all of which are required. The risk
Camerfirma highlights is the risk to their business, which is unsurprising,
but that risk is clearly outweighed by the risk to the community of users.
In this set, all users must be considered, as every single user is affected
by the failure of Camerfirma. This is not about "too big to fail", of which
there is demonstrably no such thing, but about understanding the
compatability risk to the users who rely on Camerfirma certificates
regularly, who would be affected by a removal of trust, compared to the
risk to all the users who do not rely on such certificates, but would and
are affected by Camerfirma's failures.

That Camerfirma does not understand or express appreciation for this risk
is, to the extent, of great cause for concern.

[Ramiro Muñoz]

Dear Ryan,

We are looking at the same data but we’re reading two completely different stories.

We are reading a story of a small CA that had its own graduation journey, struggled but eventually managed to emerge stronger from such journey.

You are reading a story of deceitful and unreliable CA that represents the worst danger to the entire community (your even wrote: “Camerfirma is as bad or worse than WoSign and DigiNotar”!), even if you yourself recognised that was your subjective opinion on the matter.

It is a huge distance in interpreting the same data and this subjectiveness do not guarantee a fair governance of our community.

Finally We are not saying we have not had issues during the time we have been operational. It may be more conducive for future discussions to not overlook the fact that any risk analysis involves not only risk identification but also the identification of the qualitative and quantitative impact of the risk event.


All the best,

[Ryan Sleevi]

> Dear Ryan,
>
> We are looking at the same data but we’re reading two completely different
> stories.
>
> We are reading a story of a small CA that had its own graduation journey,
> struggled but eventually managed to emerge stronger from such journey.
>
> You are reading a story of deceitful and unreliable CA that represents the
> worst danger to the entire community (your even wrote: “Camerfirma is as
> bad or worse than WoSign and DigiNotar”!), even if you yourself recognised
> that was your subjective opinion on the matter.

I am concerned about the attempts to so significantly dismiss the concerns
as merely subjective.

I’m saddened that Camerfirma does not recognize the seriousness of these
issues, despite this thread, as evidenced by this latest response.
Camerfirma continues to suggest “risk” as if this is some absolute that
should the the guiding pole.

The analogy, in the hopes that it helps Camerfirma understand, is a bit
like saying to a bank “I know we borrowed $100, and defaulted on that loan
and never paid it back, but we were a small CA, we’ve grown, and now we
would like to borrow $1 million. We cannot demonstrate our financials, nor
can we offer collateral, but we believe we are low risk, because it was
only $100”.

More concretely, Camerfirma is viewing this through the lens of what did go
wrong, and continuing to be blind to how that signals, from a risk
perspective, of what can go wrong. They are asking to be judged based on
the direct harm to users by their (many, more than any CA I can think of)
failures, while similarly asking the community to disregard the
significance of that pattern of failures, and what it says about the
overall operations of the CA.

In short, Camerfirma is asking to be trusted implicitly and explicitly for
the future, and asking that their $100 default not hold back their $1m
loan. In banking, as in trust, this is simply unreasonable.

Some have suggested that “trust” is the ability to use pst actions to
predict future outcomes. If you say you do X, and as long as I’ve known
you, you’ve done X, then when I say I “trust” you to do X, it’s an
indicator I believe your future actions will be consistent with those past
actions.

Camerfirma has, undisputed, shown a multi-year pattern that continues,
which demonstrates both a failure to correctly implement requirements, but
also a failure to reasonably and appropriately respond to and prevent
future incidents. The incident responses, which Camerfirma would like to
assert are signs of maturity, instead show a CA that has continued to
operate below the baseline expectations for years.

Camerfirma would like the community to believe that they now meet the bare
minimum, as if that alone should be considered, and all of these past
actions disregarded because of this.

Yet the risk is real: that Camerfirma has not met the bare minimum at
present, and that Camerfirma is not prepared to continue to meet that
minimum as the requirements are improved over time. We have exhaustive
evidence of this being the case in the past, and the only assurances we
have that things are different now is Camerfirma’s management believing
that, this time, they have finally got it right. However, the responses on
even the most recent incidents continue to show that Camerfirma is
continuing to pursue the same strategy for remediation it has for years: a
strategy that has demonstrably failed to keep up with industry
requirements, and failed to address the systemic issues.

These are objective statements, demonstrated by the evidence presented, but
Camerfirma would like to present them as subjective, because they take
consideration of the full picture, and not merely the rosy, but misleading,
image that Camerfirma would like to present.

That these are persistent, sustained issues, without systemic change, is
something demonstrably worse than DigiNotar. Further, when considering the
capability for harm, and the sustained pattern of failure, it would be
foolish to somehow dismiss the risk, pretending as if Chekhov’s gun of
failure is not destined to go off in the next act.

At the core, Camerfirma is treating this as if any response from the
community should be directly proportional to the *individual* failures, as
many as they are, and is asking the community to ignore both the systemic
patterns and what it says about the future. This is abundantly clear when
they speak of risk: they apparently are unable to comprehend or acknowledge
what the patterns predict, and the risk of that, and thus ask such patterns
be disregarded entirely as somehow, incorrectly, being too subjective.

If these failures were to be plotted on a time series, there is no question
that the slope of this graph is worrying, and the number of incidents - and
the type and pattern of incidents - is worrying. Camerfirma would ask we
ignore all such statistics and data, under the assertion that the slope of
sheer number of incidents is trending downward. Yet to do so would be to
disregard the data we have, and disregard the trendlines that show the type
of incidents have not meaningfully changed, that even with a downward trend
it is unacceptably above the baseline and will be for some time, and would
like us to forget everything we know because, Finally, Once and For All,
they’ve hired enough people to do the job that they’ve been required to do
from the beginning.

[Ramiro Muñoz]

Dear Ryan:

We’re not asking to ignore statistics, as matter of fact we’ve shown our objective bug evolution trend in our previous message.

We don’t ask the community to disregard the data, on the contrary we ask the community to analyze the data thoroughly including the impacts produced.

We acknowledge we have room for improvement but, we’re not applying for a loan we won’t be able to pay back. We are one of the largest and oldest CA in Spain, we issue (overall, not only SSL) more than 200.000 SMIME certificates per year and in our long history we’ve never caused any damage to anyone. Some certificates may have been syntactically incorrect due to misinterpretation, but we have never compromised any vetting, identification or information validation.

We’ve been an active and collaborative member of this community since 2008. We take our responsibility as a CA very seriously and the safeguard of all our stakeholders (including this community and all our clients) is our outmost priority.

[Matt Palmer]

On Sun, Jan 17, 2021 at 12:51:29AM -0800, Ramiro Muñoz via dev-security-policy wrote:
> We don’t ask the community to disregard the data, on the contrary we ask
> the community to analyze the data thoroughly including the impacts
> produced.

OK, I'll bite. As a member of the community, I've analyzed the data
thoroughly, and I'm not impressed. Camerfirma does not appear to grasp the
fact that "nothing bad has happened yet" is a *bad take*. "Nothing bad has
happened yet" is how every CA starts its life. It is not something to be
proud of, it's the absolute bare minimum. The volume of incidents that
Camerfirma has had is troubling, but it's the repetition of the nature of
the incidents, and the lacklustre way in which they have been responded to,
that causes me to think that Camerfirma has no place in the Mozilla trust
store.

- Matt

[Ramiro Muñoz]

Dear Matt,

Thanks for your input, we really appreciate your time in contributing to this discussion.

We are trying to make this discussion as objective as possible, and talking about objectivity I’d like to ask you where does the ‘bare minimum’ threshold stands according to Mozilla Root Store Policy. And why you are positioning Camerfirma below such a ‘bare minimum’ bar considering that Camerfirma, according to the public data, is not the member with the highest number of incidents nor the member with the most severe ones.

Finally, I’d like to ask you, based on which article of Mozilla Root Store Policy, you are sentencing a removal from the Mozilla store.

Again we appreciate your time and input in contributing to this discussion.

-Ramiro

[Ramiro Muñoz]

El martes, 19 de enero de 2021 a las 0:49:42 UTC+1, Matt Palmer escribió:

Dear Matt,

Thanks for your input, we really appreciate your time in contributing to this discussion.

We are trying to make this discussion as objective as possible, and talking about objectivity I’d like to ask you where does the ‘bare minimum’ threshold stands according to Mozilla Root Store Policy. And why you are positioning Camerfirma below such a ‘bare minimum’ bar considering that Camerfirma, according to the public data, is not the member with the highest number of incidents nor the member with the most severe ones.

Finally, I’d like to ask you, based on which article of Mozilla Root Store Policy, you are sentencing a removal from the Mozilla store.

Again we appreciate your time and input in contributing to this discussion.

Ramiro

[paul.leo....@gmail.com]

On Tuesday, January 19, 2021 at 11:01:15 AM UTC+1, Ramiro Muñoz wrote:

> Finally, I’d like to ask you, based on which article of Mozilla Root Store Policy, you are sentencing a removal from the Mozilla store.

Oh, I know this one: It is in the Mozilla Root Store Policy, 7.3: "Mozilla MAY, at its sole discretion, decide to disable (partially or fully) or remove a certificate at any time and for any reason." (You might really want to start to read the Mozilla Root Store Policy and BR before posting here or in incident reports.)

But please note that Matt is not sentencing anyone but merely providing arguments for the module peers/owner, who, by Mozilla's decision-making process, will call the shots in the end (by their sole discretion possibly based or not based on arguments in this thread).

Also, your whataboutisms might not serve you well. If you think that other CAs have handled incidents inadequately, your questions in the respective incident report bugs would surely have been much appreciated.

On the subject, since the start of this thread, things have actually got worse. Camerfirma evidently got under pressure, which, for a functioning CA, would result in better incident handling and an opportunity to show their solid foundation as a CA. Instead, Camerfirma, besides engaging in absurd argumentation in this thread, has started to request bugs clearly not fully remediated be closed (<https://bugzilla.mozilla.org/show_bug.cgi?id=1668331#c17>, <https://bugzilla.mozilla.org/show_bug.cgi?id=1667430#c35>). Recently, we have also learned that Camerfirma does not even have an understanding (or process) about the BR's revocation timelines (<https://bugzilla.mozilla.org/show_bug.cgi?id=1686966>).

There cannot be such a thing as a "last chance" ("Let's see how things work out"/"Camerfirma gets removed with the next incident") as this would put even more pressure on Camerfirma. It would also come with a massive incentive for Camerfirma to not report any more incidents. For Mozilla and their users, this would come with the risk of unreported incidents but also the need for an emergency release of Firefox and other relying software in case Camerfirma has to be removed in an unorderly way. Thus, orderly (pre-announced) distrust in one of the next Firefox release is the only way forward.

[Kurt Roeckx]

On 2021-01-19 11:02, Ramiro Muñoz wrote:
> El martes, 19 de enero de 2021 a las 0:49:42 UTC+1, Matt Palmer escribió:
>> On Sun, Jan 17, 2021 at 12:51:29AM -0800, Ramiro Muñoz via dev-security-policy wrote:
>>> We don’t ask the community to disregard the data, on the contrary we ask
>>> the community to analyze the data thoroughly including the impacts
>>> produced.
>> OK, I'll bite. As a member of the community, I've analyzed the data
>> thoroughly, and I'm not impressed. Camerfirma does not appear to grasp the
>> fact that "nothing bad has happened yet" is a *bad take*. "Nothing bad has
>> happened yet" is how every CA starts its life. It is not something to be
>> proud of, it's the absolute bare minimum. The volume of incidents that
>> Camerfirma has had is troubling, but it's the repetition of the nature of
>> the incidents, and the lacklustre way in which they have been responded to,
>> that causes me to think that Camerfirma has no place in the Mozilla trust
>> store.
>>
>> - Matt
>
> Dear Matt,
>
> Thanks for your input, we really appreciate your time in contributing to this discussion.
>
> We are trying to make this discussion as objective as possible, and talking about objectivity I’d like to ask you where does the ‘bare minimum’ threshold stands according to Mozilla Root Store Policy. And why you are positioning Camerfirma below such a ‘bare minimum’ bar considering that Camerfirma, according to the public data, is not the member with the highest number of incidents nor the member with the most severe ones.

I think you misunderstand Matt's mail.

If "something bad has happened" was the case, this would be a much
different discussion. As far as we know, you're still meeting the bare
minimum. But the bare minimum is not good enough.


Kurt

[Ramiro Muñoz]

Paul,
Thanks for your contribution.

Yes, we know art. 7.3 of the Mozilla Root Store Policy and we are aware that Mozilla may remove a certificate at any time and for any reason.
Of course in the event such decision be taken, we would respect it but, for sake of a proper governance of our community the reasons for such a traumatic decision should be clearly communicate to all community members. And the reason should be as objective as possible, bearing in mind – again - that Camerfirma is not the member with the highest number of incidents nor the member with the most severe ones.

>Camerfirma, besides engaging in absurd argumentation in this thread, has started to request bugs clearly not fully remediated be closed (<https://bugzilla.mozilla.org/show_bug.cgi?id=1668331#c17>, <https://bugzilla.mozilla.org/show_bug.cgi?id=1667430#c35>).

Paul it was a mistake when we try to close all those open bugs when the answers from our side ware already sent. This wasn't the case in this bug, in fact we keep on adding new information to it.

> Recently, we have also learned that Camerfirma does not even have an understanding (or process) about the BR's revocation timelines (<https://bugzilla.mozilla.org/show_bug.cgi?id=1686966>).

We have already published an answers in this bug and expaling the timeling considered.

[Andrew Ayer]

On Sun, 17 Jan 2021 00:51:29 -0800 (PST)
Ramiro Mu__oz via dev-security-policy

<dev-secur...@lists.mozilla.org> wrote:

> Some certificates may have been syntactically
> incorrect due to misinterpretation, but we have never compromised any
> vetting, identification or information validation.

This is false, as shown by incidents like
https://bugzilla.mozilla.org/show_bug.cgi?id=1672423
(issuing for a non-existent domain) and
https://bugzilla.mozilla.org/show_bug.cgi?id=1420871 (not checking CAA),
not to mention the validation failures by sub-CAs for which Camerfirma
is ultimately responsible. And even misissuances that are just
"syntactically incorrect" are concerning because they show a disregard
for the policies that exist to prevent harm to innocent parties.

It's troubling that even at this stage, Camerfirma still doesn't seem
to grasp the seriousness of their compliance problems. Today,
they are arguing that there was no security threat from a certificate
issued for a domain without authorization because the subdomain
in the certificate "does not exist": https://bugzilla.mozilla.org/show_bug.cgi?id=1672409#c8

Camerfirma was warned in 2018 that trust in their CA was in jeopardy,
yet compliance problems continued. There is no reason to believe
Camerfirma will improve, and there are many indications that they won't.
Mozilla's users deserve CAs that take security more seriously than this.
It's time to take action to protect Mozilla's users by distrusting
Camerfirma.

Regards,
Andrew

[Jonathan Rudenberg]

On Tue, Jan 19, 2021, at 12:01, Andrew Ayer via dev-security-policy wrote:
> Camerfirma was warned in 2018 that trust in their CA was in jeopardy,
> yet compliance problems continued. There is no reason to believe
> Camerfirma will improve, and there are many indications that they won't.
> Mozilla's users deserve CAs that take security more seriously than this.
> It's time to take action to protect Mozilla's users by distrusting
> Camerfirma.

I strongly agree. The consistent pattern of documented failures and insufficient remediation is deeply problematic, and reflects a level of danger to Mozilla users that can only be mitigated by distrusting the CA.

Jonathan

[Paul Kehrer]

I also agree with this sentiment. Camerafirma's extensively documented
issues (https://wiki.mozilla.org/CA:Camerfirma_Issues) and the
responses in this thread reveal a CA which cannot responsibly handle
the burden of being a publicly trusted authority.

-Paul

[Matt Palmer]

On Tue, Jan 19, 2021 at 07:28:17AM -0800, Ramiro Muñoz via dev-security-policy wrote:
> Camerfirma is not the member with the highest number of

> incidents nor the member with the most severe ones.

No, but Camerfirma's got a pretty shocking history of poor incident
response, over an extended period, with no substantive evidence of
improvement. *That* makes me not want to trust Camerfirma, because I have
no confidence that problems are being handled in a manner befitting a
globally-trusted CA. Further, Camerfirma's continued insistence that "it's
all better now", in the face of all the contrary evidence, does not inspire
confidence that there will be future improvement.

- Matt

[Filippo Valsorda]

2021-01-19 18:01 GMT+01:00 Andrew Ayer via dev-security-policy <dev-secur...@lists.mozilla.org>:
> It's troubling that even at this stage, Camerfirma still doesn't seem
> to grasp the seriousness of their compliance problems. Today,
> they are arguing that there was no security threat from a certificate
> issued for a domain without authorization because the subdomain
> in the certificate "does not exist": https://bugzilla.mozilla.org/show_bug.cgi?id=1672409#c8

In my personal capacity, I want to stress how worrying this response by Camerafirma is. Arguing that a certificate doesn't present any risk if it's issued for a name that doesn't exist in DNS betrays a deep misunderstanding of the web platform, which the WebPKI serves. (For example, an attacker in a privileged network position can fake a DNS response for that domain, and use it to set Secure cookies on the whole site.)

[Claves Nostrum]

One issue that really stands out for me is "Issue NN: Incorrect OCSP Delegated Responder Certificate (2013 - 2020)".

Despite detailed public discussion on the risk and remedial actions (including what would properly demonstrate destruction of the affected CA keys through e.g. ISAE3000 independent key destruction witnessing) CamerFirma failed to deliver any report providing proper assurance to the community that the affected CA keys have been properly and completely destroyed.

If I understand the details disclosed in crt.sh correctly, it seems that in the case of CamerFirma one of the CAs having the OCSP signing EKU set (https://crt.sh/?id=12729554) appears to have been operated by a third party "CONSEJO GENERAL DE COLEGIOS OFICIALES DE MEDICOS".

Not having assurance on the destruction of CA keys held within the CA's own environment is one terrible thing. In the case of the above CA certificate we are confronted with an even worse situation were a third party sub-CA potentially still holds keys (because no proper assurance on destruction) that can be abused to craft certificates and manipulate chain validity status of certificates in scope of the Mozilla root program. Camerfirma does not have any technical capability to prevent or stop this scenario would if ever materialize, as the certificate revocation by the parent CA / Camerfirma can be overwritten by keys of a CA that has the OCSP signing EKU set, and we simply don't know if they were destroyed properly.

Can Camerfirma provide additional details on why they did not work with an external auditor to produce key destruction witnessing reports (which was so apparent they should from the public discussions and what other affected CA were doing) and confirm in which environment the above mentioned CA key (https://crt.sh/?id=12729554) did/does effectively reside?

Matt Palmer:

[Ramiro Muñoz]

Andrew

Thanks for your contributions. For the sake of clarity, I would like to split your comments:

A. For what concerns 2018 and 2019 issues (https://bugzilla.mozilla.org/show_bug.cgi?id=1420871 and https://bugzilla.mozilla.org/show_bug.cgi?id=1672423), we did not consider those certificates as a security risk because they were never delivered. Furthermore, those errors have been already fixed and nowadays detected automatically.

B. For what concerns 2020 issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1672409#c8), there is a more complex explanation in this bug about potential security threats: reducing it to the statement “the subdomain in the certificate does not exist" is in some way misleading. In fact, the certificate should have been installed by the SubCA itself for hosting the expired web page (this certificate is part of the three “technical” certificates as required in clause 2.2 of CAB Forum BR) but – due to a mistake – it was not installed. This is the reason why the SubCA is confident that the certificate was not used, and it did not produce any security issue.

When we say that security issues are more important than compliance ones, we do not (at all) disregard compliance issues: we just start from the technical side to comply with both. We are trying to analyse our bugs and security issues, making more improvements both on regulatory and technical side: this approach has led us to reinforce also our compliance procedures.

All those failures you mentioned were fixed, and – in addition – we deployed automatic controls to avoid them in a future.

After the said 2018 warning to comply with the request of improvement, we have done many actions:

• Control cablint, x509lint and zlint (pre-issuance - post-issuance) (January 2019).
• Additional automatic control to verify the correction of AKI extension before certificate issuance (April 2020). Automatic verification of CAA (June 2020).
• Control of the DNS and Email domain (August 2020).
• Syntax control of the domain (August 2020).
• Control of black and whitelists of domains (August 2020).

Regards
Ramiro

[Ramiro Muñoz]

Hello Paul, Jonathan

I am sorry that you have this feeling. I would like to know what comments in this discussion reveal that we cannot take responsibility for the burden of running a CA. I would be happy to give you more detailed information to clarify your doubts. We see that in some answers we have not been understood or we have no be enough clear.

KR
Ramiro

[Ramiro Muñoz]

Hi Filippo, thanks for your contribution.

I think there has been a misunderstanding about Camerfirma answer since we do not argue that issuing a certificate for a name that doesn't exist in DNS doesn't present any risk. We meant that in this specific incident there haven’t been any security issues because this specific certificate – and the corresponding private key – was used inside a closed and protected environment. In fact, it was managed internally by the SubCA itself because it was one the three technical certificates (a valid one, an expired one and a revoked one) that every CA shall create and install according to clause 2.2 of CAB Forum BR: it was never sent outside this environment and released into the wild, where – indeed – it could have created some risks.

Nevertheless, the bug is still open, and we are giving additional information to evaluate it. https://bugzilla.mozilla.org/show_bug.cgi?id=1672409#c8.

[Ramiro Muñoz]

Hi Matt.

As we recognized previously, we have room for improvement. As we remain open to suggestions, we are still working a lot on incident response, action details and language skills.

Ramiro

[Watson Ladd]

Why exactly should Mozilla trust your record of failure to improve on these after the first, second, third, fourth, fifth, sixth, seventh, eighth, etc. time? Far from getting better Camerfirma continues to have issues with implementing obvious steps to manage risks in the issuance process. It is not the job of members of this forum to tell CAs how to shape up, it's the job of CAs to not misissue certificates. It's actually more important than issuing certificates. Why should I believe that Camerfirma can do the job?

We're on to double Latin letters for a CA that has a very small issuance volume over very few years. About once every 1.5 months on average Camerfirma has a deficiency. It's a shocking error rate, made even more shocking by the failure to learn lessons and its worsening over time. Many of the issues stem from a reliance on manual processing and inability or unwillingness to automate routine validity checks, even after the folly of this approach has been made clear. Another class of issues involve failure to properly manage delegation of authority, be it trusting WoSign, uncontrolled CAs not disclosed, etc. A third overarching theme is continued denial and making of excuses.

This is not the right attitude a publicly trusted CA should have. Every one of these incidents was an unacceptable violation of trust, that by the terms of inclusion needed to be reported with a detailed answer as to actions taken in response. You've consistently failed to articulate in these incidents a root cause analysis, what steps will be taken to prevent a recurrence, and failed to learn from the failures of other CAs or even your own issues. In several cases commitments Camerfirma made to the community were not followed through on: see issue LL, where commitments from issues J and Z were seemingly undone, and the supposed remediation was incomplete. After issue R, issues T, PP, and RR should have been impossible.

It is not your English that is lacking but your candor and sense of duty to the responsibilities with which you have been entrusted. I do not understand how any statements on your part can restore confidence given this record. I do not understand how Mozilla can trust any remediation plan Camerfirma articulates when the past response to incidents has been lackluster, incomplete, and, incredibly, continued to get worse. The proposed remediation actions strike me as woefully insufficient and should have been taken as part of any incident response already. I see no remedy but distrust.

Sincerely,
Watson Ladd

>
> Ramiro

[Ramiro Muñoz]

El jueves, 3 de diciembre de 2020 a las 19:01:55 UTC+1, Ben Wilson escribió:
> All,
>
> We have prepared an issues list as a summary of Camerfirma's compliance
> issues over the past several years. The purpose of the list is to collect
> and document all issues and responses in one place so that an overall
> picture can be seen by the community. The document is on the Mozilla wiki:
> https://wiki.mozilla.org/CA:Camerfirma_Issues.
> <https://wiki.mozilla.org/CA:Camerfirma_Issues>
>
> I will now forward the link above to Camerfirma to provide them with a
> proper opportunity to respond to the issues raised and to ask that they
> respond directly in m.d.s.p. with any comments, corrections, inputs, or
> updates that they may have.
>
> If anyone in this group believes they have an issue appropriate to add to
> the list, please send an email to certif...@mozilla.org.
>
> Sincerely yours,
> Ben Wilson
> Mozilla Root Program

Thanks everyone for your valuable contribution to the discussion. We’ve prepared a throughful Remediation Plan that addresses all areas of improvement emerged both in this public discussion as well as direct contacts with some of the members https://drive.google.com/file/d/1DV7cUSWqdOEh3WwKsM5k1U5G4rT9IXog/view?usp=sharing. The plan is very ambitious but, we’ve our BoD commitment to align Camerfirma to the highest level of standards of the Mozzilla community. Please feel free send us any request for clarification or any suggestion to improve the attached document.

Thanks for your contribution.

[Watson Ladd]

On Sunday, January 24, 2021 at 11:58:29 AM UTC-8, Ramiro Muñoz wrote:
<snip>

>
> Thanks everyone for your valuable contribution to the discussion. We’ve prepared a throughful Remediation Plan that addresses all areas of improvement emerged both in this public discussion as well as direct contacts with some of the members https://drive.google.com/file/d/1DV7cUSWqdOEh3WwKsM5k1U5G4rT9IXog/view?usp=sharing. The plan is very ambitious but, we’ve our BoD commitment to align Camerfirma to the highest level of standards of the Mozzilla community. Please feel free send us any request for clarification or any suggestion to improve the attached document.

The remediation plan seems to raise, not eliminate issues:

- Action point 1 raises the possibility that anomalous actions are possible. Why aren't the issuance processes automated and logged already?

- Action point 2 will not work. Humans are bad at monitoring for rare conditions. Some of the issues were not misspellings or confusion over the name of a company, but syntactic defects that machines could detect. It should at minimum be paired with automated validation.

- Action point 5 should already be achieved as a result of commitments made in https://bugzilla.mozilla.org/show_bug.cgi?id=1390977#c30

- Action point 9 should be trivial. But it isn't. Why not?

Beyond that all of these are actions that should have been undertaken in remediation of the past issues when they happened. I see very little that would remediate the risks of missussance, such as e.g exiting the sub-CA business, and migrating to a proven CA infrastructure rather than the homegrown one that seems to give operators plenty of scope to make mistakes. There's no issuance freeze to permit these necessary controls to be in place before resuming.

Sincerely,
Watson Ladd

>
> Thanks for your contribution.

[Ryan Sleevi]

(Writing in a Google capacity)

I personally want to say thanks to everyone who has contributed to this
discussion, who have reviewed or reported past incidents, and who have
continued to provide valuable feedback on current incidents. When
considering CAs and incidents, we really want to ensure we’re considering
all relevant information, as well as making sure we’ve got a correct
understanding of the details.

After full consideration of the information available, and in order to
protect and safeguard Chrome users, certificates issued by AC Camerfirma SA
will no longer be accepted in Chrome, beginning with Chrome 90.

This will be implemented via our existing mechanisms to respond to CA
incidents, via an integrated blocklist. Beginning with Chrome 90, users
that attempt to navigate to a website that uses a certificate that chains
to one of the roots detailed below will find that it is not considered
secure, with a message indicating that it has been revoked. Users and
enterprise administrators will not be able to bypass or override this
warning.

This change will be integrated into the Chromium open-source project as
part of a default build. Questions about the expected behavior in specific
Chromium-based browsers should be directed to their maintainers.

To ensure sufficient time for testing and for the replacement of affected
certificates by website operators, this change will be incorporated as part
of the regular Chrome release process. Information about timetables and
milestones is available at https://chromiumdash.appspot.com/schedule.

Beginning approximately the week of Thursday, March 11, 2021, website
operators will be able to preview these changes in Chrome 90 Beta. Website
operators will also be able to preview the change sooner, using our Dev and
Canary channels, while the majority of users will not encounter issues
until the release of Chrome 90 to the Stable channel, currently targeting
the week of Tuesday, April 13, 2021.

When responding to CA incidents in the past, there have been a variety of
approaches taken by different browser vendors, determined both by the facts
of the incident and the technical options available to the browsers. Our
particular decision to actively block all certificates, old and new alike,
is based on consideration of the details available, the present technical
implementation, and a desire to have a consistent, predictable,
cross-platform experience for Chrome users and site operators.

For the list of affected root certificates, please see below. Note that we
have included a holistic set of root certificates in order to ensure
consistency across the various platforms Chrome supports, even when they
may not be intended for TLS usage. However, please note that the
restrictions are placed on the associated subjectPublicKeyInfo fields of
these certificates.

Affected Certificates (SHA-256 fingerprint)

- 04F1BEC36951BC1454A904CE32890C5DA3CDE1356B7900F6E62DFA2041EBAD51
- 063E4AFAC491DFD332F3089B8542E94617D893D7FE944E10A7937EE29D9693C0
- 0C258A12A5674AEF25F28BA7DCFAECEEA348E541E6F5CC4EE63B71B361606AC3
- C1D80CE474A51128B77E794A98AA2D62A0225DA3F419E5C7ED73DFBF660E7109
- 136335439334A7698016A0D324DE72284E079D7B5220BB8FBD747816EEBEBACA
- EF3CB417FC8EBF6F97876C9E4ECE39DE1EA5FE649141D1028B7D11C0B2298CED

On Thu, Dec 3, 2020 at 1:01 PM Ben Wilson via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> All,
>
> We have prepared an issues list as a summary of Camerfirma's compliance
> issues over the past several years. The purpose of the list is to collect
> and document all issues and responses in one place so that an overall
> picture can be seen by the community. The document is on the Mozilla wiki:
> https://wiki.mozilla.org/CA:Camerfirma_Issues.
> <https://wiki.mozilla.org/CA:Camerfirma_Issues>
>
> I will now forward the link above to Camerfirma to provide them with a
> proper opportunity to respond to the issues raised and to ask that they
> respond directly in m.d.s.p. with any comments, corrections, inputs, or
> updates that they may have.
>
> If anyone in this group believes they have an issue appropriate to add to
> the list, please send an email to certif...@mozilla.org.
>
> Sincerely yours,
> Ben Wilson
> Mozilla Root Program

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

[Matthias van de Meent]

On Sun, 24 Jan 2021 at 20:58, Ramiro Muñoz via dev-security-policy
<dev-secur...@lists.mozilla.org> wrote:
>
> Thanks everyone for your valuable contribution to the discussion. We’ve prepared a throughful Remediation Plan that addresses all areas of improvement emerged both in this public discussion as well as direct contacts with some of the members https://drive.google.com/file/d/1DV7cUSWqdOEh3WwKsM5k1U5G4rT9IXog/view?usp=sharing. The plan is very ambitious but, we’ve our BoD commitment to align Camerfirma to the highest level of standards of the Mozzilla community. Please feel free send us any request for clarification or any suggestion to improve the attached document.

In one of your previous replies on this thread you stated the following:

> . We are rethinking the SubCA policy since some of the problems come from there. Camerfirma has increase the control imposing our 3 external SubCA to use the camerfirma central lint control in the pre-isuance process. However, next year we plan to convert external SubCA to Wite label CA, that's means to run them inside Camerfirma infrastructure.

But in this document it looks like this decision has been reverted:

> a. Action Point 10.

> Insource the management of all the operational activities of the intermediate CAs (June
2021).

> How:
> - Obligation of the SubCA to use the application of controls defined by Camerfirma (obligation to send us evidence that they have implemented them).
> - SubCA obligation to submit to audits set by Camerfirma and with an auditor selected by Camerfirma.
> - Insource management of all the operational activities of the intermediate CAs in a discretionary manner.
>
> Resources: Internal resources (Legal).

Specifically, the need for SubCAs to submit to audits implies that
this SubCA (company) is still in control of the ICA's signing.
Additionally, the lack of operational resources required for this
change further reinforces this implication.

As we've seen a lot of problems also stemming from the implementation
of external SubCAs, this seems like a serious deterioration in
trustability, as that requires Mozilla to trust Camerfirma to hold the
SubCA to the requirements of the relevant root stores, while it has
repeatedly shown not to do that.

Could you explain why you decided to revert that decision?

Regards,

Matthias van de Meent

[Ramiro Muñoz]

Dear Matthias;

We’ve had the chance to discuss the topic with some of our subCAs and arrived to the proposed solution.
What we’re proposing is not an immediate insourcing of all SubCAs operational activities but, a right to do so if and when we deem it necessary.
This means the following:

1. SubCAs will be obliged to implement the application controls defined by Camerfirma
2. SubCAs will be obliged to submit to audits set by Camerfirma and with an auditor selected by Camerfirma
3. SubCAs will accept contractually that Camerfirma can at any time decide to gain full control of their operational activities

In such a way we’ll be able, at any timer, to insource the management of operational activities of our ‘problematic’ intermediate CA.
While virtuous intermediate CA will be allowed to retain the control of their operational activities as long as they remain virtuous.

In addition, In our remediation plan we are going to incorporate additional resources and controls that will allow us to carry out this monitoring with all guarantees.
However, we are open to discuss further tuning of our remediation plan to gain the community confident and assure the security and compliance with the BRs and Mozilla's policies.

[Eric Mill]

Just to build on what Ryan said, and to clarify any confusion around the scope of Chrome’s action here - Chrome is no longer accepting Camerfirma certificates that are specifically used for *TLS server authentication* for websites.

Our planned action is related to the certificates Chrome uses and verifies, which are only those used for TLS server authentication. This does include any type of certificate used in Chrome for TLS server authentication, including Qualified Website Authentication Certificates (QWACs) and certificates used to comply with the Revised Payment Services Directive (PSD2). However, it does not cover other use cases, such as TLS client certificates or the use of Qualified Certificates for digital signatures.

In order to ensure Chrome’s response is comprehensive, the list of affected roots includes all of those operated by Camerfirma that have the technical capability to issue TLS server authentication certificates, even if those roots are not currently being used to issue TLS server authentication certificates. But please note that the changes we announced for Chrome will not impact the validity of these roots for other types of authentication, only current and future use of those roots for TLS server authentication in Chrome.

[Matthias van de Meent]

On Tue, 26 Jan 2021 at 16:28, Ramiro Muñoz via dev-security-policy
<dev-secur...@lists.mozilla.org> wrote:
>
> El lunes, 25 de enero de 2021 a las 13:31:18 UTC+1, Matthias van de Meent escribió:

> Dear Matthias;
>
> We’ve had the chance to discuss the topic with some of our subCAs and arrived to the proposed solution.
> What we’re proposing is not an immediate insourcing of all SubCAs operational activities but, a right to do so if and when we deem it necessary.
> This means the following:
>
> 1. SubCAs will be obliged to implement the application controls defined by Camerfirma
> 2. SubCAs will be obliged to submit to audits set by Camerfirma and with an auditor selected by Camerfirma
> 3. SubCAs will accept contractually that Camerfirma can at any time decide to gain full control of their operational activities
>
> In such a way we’ll be able, at any timer, to insource the management of operational activities of our ‘problematic’ intermediate CA.
> While virtuous intermediate CA will be allowed to retain the control of their operational activities as long as they remain virtuous.

Could you share what your definition of 'virtuous' means? This, too,
is critical in determining if the trust in your actions (including
your actions to keep trusting these SubCAs with key, operational and
managemental control of parts of your CA infrastructure) is misplaced
or not. And since your SubCAs have a less than virtuous track record,
this wording seems out-of-place.

> In addition, In our remediation plan we are going to incorporate additional resources and controls that will allow us to carry out this monitoring with all guarantees.
> However, we are open to discuss further tuning of our remediation plan to gain the community confident and assure the security and compliance with the BRs and Mozilla's policies.

[Ryan Sleevi]

(Writing in a Google capacity)

In [1], we removed support in Camerfirma certificates, as previously
announced [2]. This included removing support for any subordinate CAs. As
announced, this was planned to roll out as part of the Chrome 90 release
schedule, scheduled to hit stable on 2021-04-06.

As with any CA removal, we’ve continued to examine ecosystem progress. When
appropriate, we've also reached out to specific organizations to understand
any challenges that might significantly impact our users. While we actively
discourage CAs and site operators from directly contacting us to request
exceptions, we do reach out to organizations that may significantly affect
a non-trivial number of users. This situation is particularly unique due to
the global pandemic, as several Portugese and Spanish government websites
related to COVID-19 are affected.

We've received confirmation from these organizations that they are on track
to migrate. These organizations have requested 1-2 additional weeks to
replace their certificates, beyond the three months since the original
announcement. Normally, we would not honor such requests, given the
industry standard expectations around certificate replacement being doable
in five days or less. However, the global pandemic has brought unique and
unprecedented challenges. Given the importance of these websites in helping
resolve this global health crisis, we’re inclined to provide that
additional migration support under these circumstances.

We plan to temporarily suspend the Camerfirma removal for at least the
first Chrome 90 release, to provide that additional time to migrate these
pandemic-related websites. Although we considered other technical
approaches, we believe this approach to be the least risky for this
situation. We’ll continue to work directly with these COVID-19 related
websites on their migration. We plan to remove trust no later than Chrome
91, but may remove it sooner, such as part of a Chrome 90 security update,
based on these migration efforts. This will not affect Chrome Beta, Dev,
and Canary channels, as they will continue to block certificates from the
Camerfirma hierarchy.

[1] https://bugs.chromium.org/p/chromium/issues/detail?id=1173547
[2]
https://groups.google.com/g/mozilla.dev.security.policy/c/dSeD3dgnpzk/m/iAUwcFioAQAJ

On Thu, Jan 28, 2021 at 10:39 PM Eric Mill via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Just to build on what Ryan said, and to clarify any confusion around the
> scope of Chrome’s action here - Chrome is no longer accepting Camerfirma
> certificates that are specifically used for *TLS server authentication* for
> websites.
>
> Our planned action is related to the certificates Chrome uses and
> verifies, which are only those used for TLS server authentication. This
> does include any type of certificate used in Chrome for TLS server
> authentication, including Qualified Website Authentication Certificates
> (QWACs) and certificates used to comply with the Revised Payment Services
> Directive (PSD2). However, it does not cover other use cases, such as TLS
> client certificates or the use of Qualified Certificates for digital
> signatures.
>
> In order to ensure Chrome’s response is comprehensive, the list of
> affected roots includes all of those operated by Camerfirma that have the
> technical capability to issue TLS server authentication certificates, even
> if those roots are not currently being used to issue TLS server
> authentication certificates. But please note that the changes we announced
> for Chrome will not impact the validity of these roots for other types of
> authentication, only current and future use of those roots for TLS server
> authentication in Chrome.
>
>
> On Monday, January 25, 2021 at 12:01:42 AM UTC-8, Ryan Sleevi wrote: