Incident Report - Misissuance of one certificate without DNS CAA authorization (Certigna)

[josselin....@gmail.com]

The audit of our previous CAA check practices ensured that the CA/B Forum requirements were met except for a single certificate for which the CA was not authorized to issue according to the DNS CAA record.

This failure is related to our old practices that led to a control of the DNS CAA records with automatic alerts for the Registration Officers, but the blocking of the certificate request was not automatic unlike today. It was found that the request had been approved despite this alert, and in particular because of the provision of additional supporting documents by the applicant such as a request for a certificate signed by the legal representative of the entity accompanied by a photocopy of his identity document, which attest to the consent to issue.

We checked the logs of the controls carried out and re-rolled these controls on all the SSL certificates issued since September 8th and confirm that only this certificate was the object of a failure.

This certificate, which has not yet been deployed and used by the customer, has been identified and revoked by the CA and is now included in the CRL with the following serial number: 476abeb2bc78d588ef4b8f27dbd25f6a (see http://crl.certigna.fr/servicesca.crl).

Note that this incident will not be able to happen again by means of our new practices that automatically block any certificate request for which the DNS CAA record controls induce that the CA is not allowed to issue, without possible bypass by the RA. These practices are described in the latest updated versions of our CP/CPS.

We remain at your disposal if you want further information.

[Matt Palmer]

On Tue, Sep 11, 2018 at 12:34:34AM -0700, josselin.allemandou--- via dev-security-policy wrote:
> This failure is related to our old practices that led to a control of the
> DNS CAA records with automatic alerts for the Registration Officers, but
> the blocking of the certificate request was not automatic unlike today.

I have some questions surrounding this incident, the answers to which will,
I hope, be useful in identifying the larger ecosystem risks which can be
extrapolated from this incident, both for Certigna as well as other CAs who
may have similarly at-risk systems and processes.

1. Can you explain in more detail the user experience of the RA interface
which was used to misissue this certificate? Specifically, did approval of
the issuance in the face of a CAA validation failure require a
positive acknowledgement and override of the CAA validation failure,
separate and distinct from the actions required to issue a certificate which
passed CAA validation?

2. What details can you share around the risk analysis that deemed it
appropriate to deploy a system where human operators were able to
override critical issuance controls without any oversight, approval, or
subsequent review of that override?

3. What other parts of the issuance process have had a similar risk
analysis undertaken with similar initial results?

4. Which other parts of the issuance process have been modified as a result
of the review of those risk analyses, in light of the failure of the risk
analysis around CAA override to correctly control for the misissuance
risk inherent in CAA override functionality being available to RAs
without sufficient compensating controls?

5. What compensating controls *were* in place to prevent misuse of this manual
CAA override?

6. In what specific ways did those compensating controls fail to properly
manage the risk?

7. What training were RAs given in the policy and procedures surrounding CAA
validation and the circumstances in which manual override of a CAA denial
was appropriate and valid?

8. What aspect(s) of the training that RAs were given have been identified
as being deficient, such that the RA determined that the action they
took was appropriate?

9. What other aspects of the issuance process are trained in a similar
manner to the CAA override training that was delivered?

10. In what specific ways has the training for those other aspects of the
issuance process been improved, as a result of identifying the
deficiencies in those training packages?

11. Have all RAs been retrained in line with the revised training packages,
and if not, by what date do you expect such retraining to have been
completed?

12. If all RAs have not yet been retrained, what additional compensating
controls have you put in place in the meantime to handle the deficiency
in RA training?

- Matt

[josselin....@gmail.com]

Hello,

Thank you for your contribution. We hope that the returns below will allow you to better understand our past practices that led to the creation of this ticket.

It is important to remember that our CA is also subject to compliance with national standards (e.g. RGS) which are more stringent for some controls than ETSI standards or BRs. These standards require us, in particular, that each certificate request be signed by the legal representative of the organization, and that the identity document copy of the latter be collected and verified in connection with the application. This control entails the collection of evidence of the consent of a legal representative of the organization for us to issue a certificate, while CAA DNS setup can be done by a technical manager, even without having been validated by the organization itself.

So we were already asking for the consent of the organization for each request, and we have added the CAA DNS controls and alerts both for compliance to the CA/Browsers Forum, but also to trace incompatibilities with the declared wishes of the organization.

1. Can you explain in more detail the user experience of the RA interface which was used to misissue this certificate? Specifically, did approval of the issuance in the face of a CAA validation failure require a positive acknowledgement and override of the CAA validation failure, separate and distinct from the actions required to issue a certificate which passed CAA validation?
2. What details can you share around the risk analysis that deemed it appropriate to deploy a system where human operators were able to override critical issuance controls without any oversight, approval, or subsequent review of that override?

Our Registration Authority has a dedicated application for request management and step-by-step validation of certificate applications. The addition of technical controls is gradually reducing the human intervention of RA operators, which now only concern the validation of supporting documents and controls operated by face-to-face or by telephone. We are currently working to automate some of these controls. Nowadays, the checks of domain control and DNS CAA records are automatic and block requests.

3. What other parts of the issuance process have had a similar risk analysis undertaken with similar initial results?

As mentioned above, the practices implemented in addition to the DNS CAA alerts already provided the consent of the organization, which is why, while waiting for the development of an automatic locking process, we had positioned an alert to RA to identify any incompatibilities of authorization. It should be noted that for the certificate object of this incident, the organization had already given its formal consent and had been notified by email of the need to update its DNS configuration.Risk assessments are conducted annually on these processes and each process evolution is validated during security committees.

4. Which other parts of the issuance process have been modified as a result of the review of those risk analyses, in light of the failure of the risk analysis around CAA override to correctly control for the misissuance risk inherent in CAA override functionality being available to RAs without sufficient compensating controls?

There have been no other changes to our processes, the other controls being mostly automatic.

5. What compensating controls *were* in place to prevent misuse of this manual CAA override?

As mentioned above, the consent of the organization was already obtained in another form, that imposed by our national requirements. Alerts sent by the control of DNS CAA records should highlight inconsistencies in the authorization statement, which automatically notify the applicant to update the DNS configuration of the server, but for this unique case, the validation was done before the applicant update its configuration.

6. In what specific ways did those compensating controls fail to properly
manage the risk?

It is difficult for each requirement to determine all the risks imagined by its editor. If the risk is to issue a certificate without the consent of the organization, in this case the compensatory measures are sufficient from our point of view. Afterwards, the risk assessment remains a very subjective subject and that is why, we make contribute, as for each evolution of our processes, all the teams via a security committee to validate the practices.

7. What training were RAs given in the policy and procedures surrounding CAA validation and the circumstances in which manual override of a CAA denial was appropriate and valid?
8. What aspect(s) of the training that RAs were given have been identified as being deficient, such that the RA determined that the action they took was appropriate?
9. What other aspects of the issuance process are trained in a similar manner to the CAA override training that was delivered?
10. In what specific ways has the training for those other aspects of the issuance process been improved, as a result of identifying the deficiencies in those training packages?
11. Have all RAs been retrained in line with the revised training packages, and if not, by what date do you expect such retraining to have been completed?
12. If all RAs have not yet been retrained, what additional compensating
controls have you put in place in the meantime to handle the deficiency
in RA training?

As you can see, we are working to be totally transparent about our practices. Thus, in the CPs linked to our old practices, we explicitly specify that we allow ourselves to issue a certificate if we have previously obtained the consent of the organization and this even if the DNS CAA record had not yet been updated by the applicant. The instructions given to the RA operators were initially aligned with this commitment, however, given the few alerts encountered, the operators still blocked the issue until the records were up-to-date and consistent. Thus, except for a certificate that was issued in accordance with our CP / CPS, all the others were subject to greater control than we had committed.

Nevertheless, in addition to the training and awareness operations carried out regularly, following this incident, we sensitized and re-trained all the staff of the RA on the practices and controls to be operated, and also on the controls implemented automatically without possible bypass.

The problem does not relate to our training, and what was implemented was finally better than what was stated in our CP / CPS. The problem comes from the fact that we operate several controls having the same purpose, but whose methods differ because emanating from different requirements standards.

[Matt Palmer]

On Tue, Sep 11, 2018 at 07:22:18AM -0700, josselin.allemandou--- via dev-security-policy wrote:
> It is important to remember that our CA is also subject to compliance with
> national standards (e.g. RGS) which are more stringent for some controls
> than ETSI standards or BRs. These standards require us, in particular,
> that each certificate request be signed by the legal representative of the
> organization, and that the identity document copy of the latter be
> collected and verified in connection with the application. This control
> entails the collection of evidence of the consent of a legal
> representative of the organization for us to issue a certificate, while
> CAA DNS setup can be done by a technical manager, even without having been
> validated by the organization itself.

I'm afraid I don't see how this has any bearing on a failure to properly
validate CAA records, as required by the BRs and Mozilla policy. If it
helps simplify your answers, though, I think you could limit yourself to
addressing issuance controls within the scope of the BRs and Mozilla policy,
without reducing the utility of your answers to this group.

> 1. Can you explain in more detail the user experience of the RA interface
> which was used to misissue this certificate? Specifically, did approval
> of the issuance in the face of a CAA validation failure require a positive
> acknowledgement and override of the CAA validation failure, separate and
> distinct from the actions required to issue a certificate which passed CAA
> validation?
>
> 2. What details can you share around the risk analysis that deemed it
> appropriate to deploy a system where human operators were able to override
> critical issuance controls without any oversight, approval, or subsequent
> review of that override?
>
> Our Registration Authority has a dedicated application for request
> management and step-by-step validation of certificate applications. The
> addition of technical controls is gradually reducing the human
> intervention of RA operators, which now only concern the validation of
> supporting documents and controls operated by face-to-face or by
> telephone. We are currently working to automate some of these controls.
> Nowadays, the checks of domain control and DNS CAA records are automatic
> and block requests.

Sorry, but I can't seem to find an answer to either of the above questions
in your response. I can't find any description (or even allusion) to user
experience, nor any details of the risk analysis process around CAA override
functionality.

> 3. What other parts of the issuance process have had a similar risk
> analysis undertaken with similar initial results?
>
> As mentioned above, the practices implemented in addition to the DNS CAA
> alerts already provided the consent of the organization, which is why,
> while waiting for the development of an automatic locking process, we had
> positioned an alert to RA to identify any incompatibilities of
> authorization. It should be noted that for the certificate object of this
> incident, the organization had already given its formal consent and had
> been notified by email of the need to update its DNS configuration.Risk
> assessments are conducted annually on these processes and each process
> evolution is validated during security committees.

Sorry, but I can't seem to find an answer to the above question in your
response. I would expect to see a list of BR-required validations.

> 4. Which other parts of the issuance process have been modified as a
> result of the review of those risk analyses, in light of the failure of
> the risk analysis around CAA override to correctly control for the
> misissuance risk inherent in CAA override functionality being available to
> RAs without sufficient compensating controls?
>
> There have been no other changes to our processes, the other controls
> being mostly automatic.

It's the parts of the validation process that don't fall under that most
eminently non-specific word "mostly" which are of primary concern.

> 5. What compensating controls *were* in place to prevent misuse of this
> manual CAA override?
>
> As mentioned above, the consent of the organization was already obtained
> in another form, that imposed by our national requirements. Alerts sent
> by the control of DNS CAA records should highlight inconsistencies in the
> authorization statement, which automatically notify the applicant to
> update the DNS configuration of the server, but for this unique case, the
> validation was done before the applicant update its configuration.

Proof of consent is not a control against misuse. Failure to properly
validate CAA is in itself misissuance, regardless of whether or not the
correct organization received the certificate. Running a red light is still
an offence, even if you didn't have an accident.

> 6. In what specific ways did those compensating controls fail to properly
> manage the risk?
>
> It is difficult for each requirement to determine all the risks imagined
> by its editor. If the risk is to issue a certificate without the consent
> of the organization, in this case the compensatory measures are sufficient
> from our point of view. Afterwards, the risk assessment remains a very
> subjective subject and that is why, we make contribute, as for each
> evolution of our processes, all the teams via a security committee to
> validate the practices.

Are you saying that you don't account for the risk of possibly issuing a
certificate in contravention of the BRs and Mozilla policy?

> 7. What training were RAs given in the policy and procedures surrounding CAA validation and the circumstances in which manual override of a CAA denial was appropriate and valid?
> 8. What aspect(s) of the training that RAs were given have been identified as being deficient, such that the RA determined that the action they took was appropriate?
> 9. What other aspects of the issuance process are trained in a similar manner to the CAA override training that was delivered?
> 10. In what specific ways has the training for those other aspects of the issuance process been improved, as a result of identifying the deficiencies in those training packages?
> 11. Have all RAs been retrained in line with the revised training packages, and if not, by what date do you expect such retraining to have been completed?
> 12. If all RAs have not yet been retrained, what additional compensating
> controls have you put in place in the meantime to handle the deficiency
> in RA training?
>
> As you can see, we are working to be totally transparent about our
> practices.

I'm afraid I cannot see that.

> Thus, in the CPs linked to our old practices, we explicitly
> specify that we allow ourselves to issue a certificate if we have
> previously obtained the consent of the organization and this even if the
> DNS CAA record had not yet been updated by the applicant.

Do you consider that this CPS was in line with the BRs and Mozilla policy?
If so, can you expand on your reasoning, and if not, why did you consider it
appropriate to operate to a CPS that did not fully account for your
obligations under the BRs and Mozilla policy?

> The
> instructions given to the RA operators were initially aligned with this
> commitment, however, given the few alerts encountered, the operators still
> blocked the issue until the records were up-to-date and consistent. Thus,
> except for a certificate that was issued in accordance with our CP / CPS,
> all the others were subject to greater control than we had committed.
>
> Nevertheless, in addition to the training and awareness operations carried
> out regularly, following this incident, we sensitized and re-trained all
> the staff of the RA on the practices and controls to be operated, and also
> on the controls implemented automatically without possible bypass.
>
> The problem does not relate to our training,

That's certainly the sense I'm getting. It seems that the core problem is
that your organisation appears to take compliance with the BRs as something
of a "nice to have", rather than a "must have". This raises the question of
what other elements of the BRs you aren't adhering to, but just haven't
tripped over yet, which my questions were attempting to identify.

> finally better than what was stated in our CP / CPS. The problem comes
> from the fact that we operate several controls having the same purpose,
> but whose methods differ because emanating from different requirements
> standards.

Based on what you've provided so far, I'm afraid I can't see how you come to
that conclusion. CAA records indicate consent to issue for a specific CA or
list of CAs given by the party in control of the DNS for a domain. Nothing
you've described appears to be equal to or a strict superset of that.

At any rate, though, the BRs require you to validate CAA records, not
"validate CAA records or do something else the CA thinks is just as good",
and it appears that you systematically failed to do so.

- Matt

[Jakob Bohm]

I have numbered the new questions from 13 and up and added 7 more
questions at the end.

On 12/09/2018 05:11, Matt Palmer wrote:
> On Tue, Sep 11, 2018 at 07:22:18AM -0700, josselin.allemandou--- via dev-security-policy wrote:

.... Snipped the 12 questions that assumed this was an RA mistake and not
a management error, some topics do however reoccur below, but in the
context of management decisions ...

>
>> Thus, in the CPs linked to our old practices, we explicitly
>> specify that we allow ourselves to issue a certificate if we have
>> previously obtained the consent of the organization and this even if the
>> DNS CAA record had not yet been updated by the applicant.
>

13. Did or

> Do you consider that this CPS was in line with the BRs and Mozilla policy?

14.

> If so, can you expand on your reasoning, and if not, why did you consider it
> appropriate to operate to a CPS that did not fully account for your
> obligations under the BRs and Mozilla policy?
>
>> The
>> instructions given to the RA operators were initially aligned with this
>> commitment, however, given the few alerts encountered, the operators still
>> blocked the issue until the records were up-to-date and consistent. Thus,
>> except for a certificate that was issued in accordance with our CP / CPS,
>> all the others were subject to greater control than we had committed.
>>
>> Nevertheless, in addition to the training and awareness operations carried
>> out regularly, following this incident, we sensitized and re-trained all
>> the staff of the RA on the practices and controls to be operated, and also
>> on the controls implemented automatically without possible bypass.
>>
>> The problem does not relate to our training,
>

15.

> That's certainly the sense I'm getting. It seems that the core problem is
> that your organisation appears to take compliance with the BRs as something
> of a "nice to have", rather than a "must have". This raises the question of
> what other elements of the BRs you aren't adhering to, but just haven't
> tripped over yet, which my questions were attempting to identify.
>
>> finally better than what was stated in our CP / CPS. The problem comes
>> from the fact that we operate several controls having the same purpose,
>> but whose methods differ because emanating from different requirements
>> standards.
>

16.

> Based on what you've provided so far, I'm afraid I can't see how you come to
> that conclusion. CAA records indicate consent to issue for a specific CA or
> list of CAs given by the party in control of the DNS for a domain. Nothing
> you've described appears to be equal to or a strict superset of that.
>

Additional questions that would be relevant to get answers for:

17. Which criteria were required under your old policy to override a CAA
failure? (Besides rechecking after a DNS change by the applicant)?
I realize these were only used once, but the incorrect criteria you
had instructed your validation officers to use would be interesting
to understand how this mistaken policy was created.

18. How many certificates were delayed until the customer changed their
CAA records?

19. How many certificates were not issued because CAA failed and was not
corrected by the applicant changing their DNS?

20. Have you informed CAB/F about the specific situations in questions
17 to 19 above.

21. Are there any other places where your CP / CPS do or did fall short
of the BR requirements?

22. Are there any places where your CP / CPS do or did fall short of
Mozilla requirements beyond the BRs? (This wasn't one, it was a BR).

23. Which validation controls listed in your CP / CPS are not fully
enforced by automated systems? (Your answers suggest at least one,
so make sure your answer is complete).


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

[RS Tyler Schroder]

Adding Q24/25

24. Your CPS for the Root and Wild CPSs as of 8/31/2018 note in section 4.2.1 that a certificate issue that did not comply with RFC6844 would be automatically blocked. Your note in the first post says that "our new practices that automatically block any certificate request for which the DNS CAA record controls induce that the CA is not allowed to issue, without possible bypass by the RA". Was this blocking process not fully automated as described prior (going off of Q23 from Jakob)?

25. What exactly prompted the manual override for CAA Checking? A request from the origin certificate requestor, the RA on their own...


Relevant section:
> The following cases do not allow the CA to authorize the issuance of the certificate:
> - The CAA DNS field is present, it contains an "issue" or "issuewild" tag and does not list
> Certigna as an authorized Certificate Authority;
> - The CAA DNS field is present, it is designated as "critical" and the tag used is not supported
> by the CA (it is not an "issue" or "issuewild" tag);
> - The zone is validly DNSSEC-signed and our DNS query times out.
> If any of these cases are encountered, the certificate request is automatically blocked and the
> applicant is notified by email of the need to update the associated DNS records.

[Jakob Bohm]

On 12/09/2018 14:51, RS Tyler Schroder wrote:
> On Tuesday, September 11, 2018 at 3:34:45 AM UTC-4, josselin....@gmail.com wrote:
>> The audit of our previous CAA check practices ensured that the CA/B Forum requirements were met except for a single certificate for which the CA was not authorized to issue according to the DNS CAA record.
>>
>> This failure is related to our old practices that led to a control of the DNS CAA records with automatic alerts for the Registration Officers, but the blocking of the certificate request was not automatic unlike today. It was found that the request had been approved despite this alert, and in particular because of the provision of additional supporting documents by the applicant such as a request for a certificate signed by the legal representative of the entity accompanied by a photocopy of his identity document, which attest to the consent to issue.
>>
>> We checked the logs of the controls carried out and re-rolled these controls on all the SSL certificates issued since September 8th and confirm that only this certificate was the object of a failure.
>>
>> This certificate, which has not yet been deployed and used by the customer, has been identified and revoked by the CA and is now included in the CRL with the following serial number: 476abeb2bc78d588ef4b8f27dbd25f6a (see http://crl.certigna.fr/servicesca.crl).
>>
>> Note that this incident will not be able to happen again by means of our new practices that automatically block any certificate request for which the DNS CAA record controls induce that the CA is not allowed to issue, without possible bypass by the RA. These practices are described in the latest updated versions of our CP/CPS.
>>
>> We remain at your disposal if you want further information.
>
> Adding Q24/25
>
> 24. Your CPS for the Root and Wild CPSs as of 8/31/2018 note in section 4.2.1 that a certificate issue that did not comply with RFC6844 would be automatically blocked. Your note in the first post says that "our new practices that automatically block any certificate request for which the DNS CAA record controls induce that the CA is not allowed to issue, without possible bypass by the RA". Was this blocking process not fully automated as described prior (going off of Q23 from Jakob)?

The unqualified mention of "September 8" confused me at first, but it
obviously refers to the "CAA Mandatory BR" taking effect on "September
8, 2017", thus the single misissuance probably happened between
September 8, 2017 and when they changed the policy on August 31, 2018.

However I cannot tell, because I can't find the serial number on crt.sh.

>
> 25. What exactly prompted the manual override for CAA Checking? A request from the origin certificate requestor, the RA on their own...
>

>
> Relevant section:
>> The following cases do not allow the CA to authorize the issuance of the certificate:
>> - The CAA DNS field is present, it contains an "issue" or "issuewild" tag and does not list
>> Certigna as an authorized Certificate Authority;
>> - The CAA DNS field is present, it is designated as "critical" and the tag used is not supported
>> by the CA (it is not an "issue" or "issuewild" tag);
>> - The zone is validly DNSSEC-signed and our DNS query times out.
>> If any of these cases are encountered, the certificate request is automatically blocked and the
>> applicant is notified by email of the need to update the associated DNS records.

[RS Tyler Schroder]

> The unqualified mention of "September 8" confused me at first, but it
> obviously refers to the "CAA Mandatory BR" taking effect on "September
> 8, 2017", thus the single misissuance probably happened between
> September 8, 2017 and when they changed the policy on August 31, 2018.
>
> However I cannot tell, because I can't find the serial number on crt.sh.

This appears to be the cert in question: https://crt.sh/?id=596401403&opt=ocsp

Dates show they were revoked on the same day of the CPS update, so that explanation checks out for now

[Wayne Thayer]

Josselin: thank you for filing this incident report, and for your answers
to the questions being asked in this thread.

Please add the incident report to the related bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1485413

I will also ask you to answer the new questions that have been asked to the
best of your ability. It is my hope that we can all use this information to
learn from and prevent future incidents from occurring.

On Wed, Sep 12, 2018 at 8:46 AM Jakob Bohm via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On 12/09/2018 14:51, RS Tyler Schroder wrote:

This issue was identified in discussion of Certigna's current root renewal
request [1], in which Certigna stated that their system has automatically
checked CAA records since September 8, 2017, but rather than blocking
issuance, the system alerted RA officers who were responsible for denying
the request. Based on the statements made by Certina, I believe that they
fundamentally misunderstood the BR requirement for a CA to reject all
requests that fail CAA processing, regardless of any other compensating
controls the CA has in place.

[1]
https://groups.google.com/d/msg/mozilla.dev.security.policy/z7iDk9CdTFo/hCZGQRm4AQAJ

>

> The unqualified mention of "September 8" confused me at first, but it
> obviously refers to the "CAA Mandatory BR" taking effect on "September
> 8, 2017", thus the single misissuance probably happened between
> September 8, 2017 and when they changed the policy on August 31, 2018.
>
> However I cannot tell, because I can't find the serial number on crt.sh.
>
> >

> > 25. What exactly prompted the manual override for CAA Checking? A
> request from the origin certificate requestor, the RA on their own...
> >
>
> >
> > Relevant section:
> >> The following cases do not allow the CA to authorize the issuance of
> the certificate:
> >> - The CAA DNS field is present, it contains an "issue" or "issuewild"
> tag and does not list
> >> Certigna as an authorized Certificate Authority;
> >> - The CAA DNS field is present, it is designated as "critical" and the
> tag used is not supported
> >> by the CA (it is not an "issue" or "issuewild" tag);
> >> - The zone is validly DNSSEC-signed and our DNS query times out.
> >> If any of these cases are encountered, the certificate request is
> automatically blocked and the
> >> applicant is notified by email of the need to update the associated DNS
> records.
>
>

> Enjoy
>
> Jakob
> --
> Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
> Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
> This public discussion message is non-binding and may contain errors.
> WiseMo - Remote Service Management for PCs, Phones and Embedded

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

[josselin....@gmail.com]

Hello

Thank you all for your feedback for which we have tried to provide additional information below. Know that if necessary, you will also find the description of our practices through the following links:
• our CPS :
* Services CA : http://politique.certigna.fr/en/DPCcertignaservicesca.pdf
* Wild CA : http://politique.certigna.fr/en/DPCcertignawildca.pdf
• Our BR self-assessment file :https://bugzilla.mozilla.org/attachment.cgi?id=9007984

Wayne: Thank you. We linked the ticket to the bug as requested.

Tyler Schroder: We confirm that the certificate in question is the one that Tyler identified. It was revoked and we updated our CP / CPS to clarify the changes made following the incident and to provide additional information on other practices.

> 1. Can you explain in more detail the user experience of the RA interface
> which was used to misissue this certificate? Specifically, did approval
> of the issuance in the face of a CAA validation failure require a positive
> acknowledgement and override of the CAA validation failure, separate and
> distinct from the actions required to issue a certificate which passed CAA
> validation?

The interface used by registration officer is dedicated to the processing of certificate requests and is accessible with strong authentication with a smartcard certificate. Via this interface, registration officer operates and runs all the required checks for each FQDN requested.
All the manual controls to achieve are successively presented to the registration operator so that he operates and acknowledges the validation of each control. The results of automatic checks, such as CAA DNS and DCV, are also presented to the operator in the validation workflow.
Previously, the CAA DNS control was the only one that was an alert and not a blocking element for validating the request. The interface has been fixed and the negative CAA DNS control result now blocks request validation as for other controls.

> 2. What details can you share around the risk analysis that deemed it
> appropriate to deploy a system where human operators were able to override
> critical issuance controls without any oversight, approval, or subsequent
> review of that override?

3. What other parts of the issuance process have had a similar risk
> analysis undertaken with similar initial results?

6. In what specific ways did those compensating controls fail to properly manage the risk?

> Are you saying that you don't account for the risk of possibly issuing a certificate in contravention of the BRs and Mozilla policy?

To clarify our risk assessment process. We perform a risk assessment on the perimeter of PKI at least once a year and before major changes. We employ an exhaustive methodology aligned with the EBIOS method and the guidelines of ISO/IEC 27005. Our risk management processes are audited both in accordance with the requirements of ETSI standards, BRs, but also ISO/IEC 27001, standard on which we are also certified. We therefore consider legal, regulatory, contractual and normative security requirements for the conduct of risk assessments, and therefore the risks of non-compliance with the requirements set out by the BRs.

Our different risk assessments incorporate the analysis of risks related to the registration operator activities. The different iterations led to the establishment of the RA interface with a rigorous script to follow and to respect in order to limit any risk or non-compliance in the processing of requests.
We made an error regarding CAA DNS control and the fact that its result is not blocking for validation. We assessed during the risk assessment that the form of consent of the legal representative, and the control of the CAA DNS with a warning prompted to the registration officer, would be sufficient to limit the risks of obtaining the consent of the organization. This certificate issuance showed that we should have set up an automatic block as soon as the implementation of the control. Now the automatic blocking is active and the wrong results of all controls are blocking.

4. Which other parts of the issuance process have been modified as a result of the review of those risk analyses, in light of the failure of the risk analysis around CAA override to correctly control for the
misissuance risk inherent in CAA override functionality being available to RAs without sufficient compensating controls?

5. What compensating controls *were* in place to prevent misuse of this manual CAA override?

All controls results, manual as well as automatic, are now blocking for validation of a request. Only the CAA DNS control was not blocking. The risk assessment did not lead to action on the other controls. From now on, even if the control methodologies differ from one set of requirements to another, we will automatically ensure that validation is automatically blocked until all controls are positive, even if the risk is considered to be covered. by one of the methods used, and this also to ensure compliance. It should be noted that control of consent is the only point on which control practices were heterogeneous in our normative requirements. For the other controls, the methodologies used are homogeneous.

13. Did or Do you consider that this CPS was in line with the BRs and Mozilla policy?

Yes. We consider that our CPS was and is in line with the BRs and Mozilla policy. As mentioned earlier, we have implemented CAA DNS control as required. We mistakenly felt that the control of the consent of the legal representative, which is blocking it, and the addition of alerts related to the CAA DNS control would meet the requirements.

17. Which criteria were required under your old policy to override a CAA failure? (Besides rechecking after a DNS change by the applicant)?
I realize these were only used once, but the incorrect criteria you had instructed your validation officers to use would be interesting to understand how this mistaken policy was created.

As mentioned above, in addition to the automatic control of the CAA DNS, a blocking control is already carried out on the supply of a certificate request signed by the legal representative of the organization, with the provision of a copy of his identity document.

18. How many certificates were delayed until the customer changed their CAA records?

The only identified blocking cases corresponded to typographical errors in the registration of Certigna autority name (example: certignia.com, certgna.com, ...).
After CAA DNS records were updated, theses certificates has been issued.

19. How many certificates were not issued because CAA failed and was not corrected by the applicant changing their DNS?

As a result of our audit, other than the certificate identified by this incident and the cases listed for question 18, we have not identified any other case where the CAA DNS control should have been blocking. The majority of our certificates are issued under group contracts or public contracts for which the conditions of delivery and the obligations are known by the applicants even before placing their orders online. As a result, the majority of requests are received the first time with a valid DNS CAA record (when completed) or without (DNS CAA record not used by applicants).

20. Have you informed CAB/F about the specific situations in questions 17 to 19 above.

These points were exchanged as part of the referencing of our new root and this led to the creation of this incident ticket. If we had identified other cases following our audit, we would have also declared them in this ticket.

21. Are there any other places where your CP / CPS do or did fall short of the BR requirements?
22. Are there any places where your CP / CPS do or did fall short of Mozilla requirements beyond the BRs? (This wasn't one, it was a BR).

We currently have not identified any other areas that may be subject to non-compliance. We regularly monitoring our practices to ensure compliance and you will see that we are regularly updating our practices and CP / CPS in line with the evolutions of BRs. In addition to these controls, our internal audits and annual audits by a certification body help to verify the compliance of our practices and documentation.

23. Which validation controls listed in your CP / CPS are not fully enforced by automated systems? (Your answers suggest at least one, so make sure your answer is complete).

The following controls always involve manual control by a registration operator. The interface allows registration operator to track and confirm the validation operations they perform and their results. These manual controls relate to:
⦁ The entity identity validation;
⦁ The identity validation of the Certificate Manager and the legal representative;
⦁ The validation of the certificate application form with the signature of the Certificate Manager and the legal representative;
⦁ The realization of face-to-face with the Certificate Manager and telephone calls;
⦁ Validation of the identity of servers when it is a client server;

We are currently working to automate some of the controls mentioned above such as those on identity checks.

24. Your CPS for the Root and Wild CPSs as of 8/31/2018 note in section 4.2.1 that a certificate issue that did not comply with RFC6844 would be automatically blocked. Your note in the first post says that "our new practices that automatically block any certificate request for which the DNS CAA record controls induce that the CA is not allowed to issue, without possible bypass by the RA". Was this blocking process not fully automated as described prior (going off of Q23 from Jakob)?

Indeed, we have updated our CP and CPS to specify that the DNS CAA negative results automatically blocks the issuance of the certificate, which was not the case at the time of the occurrence of this incident.

25. What exactly prompted the manual override for CAA Checking? A request from the origin certificate requestor, the RA on their own...

As mentioned above, we made an error concerning the control of the DNS CAA records and the fact that its result is not blocking for validation. We assessed during the risk assessment that the form of consent of the legal representative, and the control of the DNS CAA with a warning to registration officer, would be sufficient to limit the risks relating to obtaining the consent of the organization. The registration operator in charge of processing this request, finding that the DNS CAA error was not blocking, and having the consent in another form, performed the validation of the request. This certificate issuance showed that we should have set up an automatic block as soon as the implementation of the control. Now the automatic blocking is active and the results of all controls are blocking.

We hope to have answered your questions and we remain at your disposal if you want more details.

Best regards

[Matt Palmer]

On Sun, Sep 16, 2018 at 03:07:44PM -0700, josselin.allemandou--- via dev-security-policy wrote:
> > 1. Can you explain in more detail the user experience of the RA interface
> > which was used to misissue this certificate? Specifically, did approval
> > of the issuance in the face of a CAA validation failure require a positive
> > acknowledgement and override of the CAA validation failure, separate and
> > distinct from the actions required to issue a certificate which passed CAA
> > validation?
>
> The interface used by registration officer is dedicated to the processing
> of certificate requests and is accessible with strong authentication with
> a smartcard certificate. Via this interface, registration officer
> operates and runs all the required checks for each FQDN requested. All
> the manual controls to achieve are successively presented to the
> registration operator so that he operates and acknowledges the validation
> of each control. The results of automatic checks, such as CAA DNS and
> DCV, are also presented to the operator in the validation workflow.
> Previously, the CAA DNS control was the only one that was an alert and not
> a blocking element for validating the request. The interface has been
> fixed and the negative CAA DNS control result now blocks request
> validation as for other controls.

Was the action required to manually override the CAA validation failure
different from what would be required if the CAA validation had succeeded?
Could an operator have just "clicked the same button they always clicked",
and have the CAA validation failure overridden? Or was there a separate
sequence of UI interactions required to override a CAA validation failure?

> We made an error regarding CAA DNS control and the fact that its result is
> not blocking for validation. We assessed during the risk assessment that
> the form of consent of the legal representative, and the control of the
> CAA DNS with a warning prompted to the registration officer, would be
> sufficient to limit the risks of obtaining the consent of the
> organization. This certificate issuance showed that we should have set up
> an automatic block as soon as the implementation of the control. Now the
> automatic blocking is active and the wrong results of all controls are
> blocking.

This sounds like there may have been a misunderstanding in the BR
description of CAA validation.

What specific language in the BRs surrounding CAA validation caused you to
believe that successful CAA validation was optional in the presence of other
forms of consent?

What changes do you propose in the language of the CAA validation
requirements in the BRs to ensure that no other CA could possibly
misunderstand the CAA validation requirements?

Have you proposed those changes to the CA/B Forum, to improve the BRs for
all participants? If not, why not?

That the BRs have been misunderstood in this fashion is somewhat concerning.
it suggests that there may be other misunderstandings latent in your systems
and processes, which haven't surfaced because they haven't (yet) led to an
identified misissuance or other incident.

What steps have you taken, or are you planning on taking, to address
possible misunderstandings of the BRs or Mozilla policy that may cause
Certigna to unintentionally misissue a certificate or suffer from some other
incident?

> 4. Which other parts of the issuance process have been modified as a
> result of the review of those risk analyses, in light of the failure of
> the risk analysis around CAA override to correctly control for the
> misissuance risk inherent in CAA override functionality being available to
> RAs without sufficient compensating controls?
>
> 5. What compensating controls *were* in place to prevent misuse of this
> manual CAA override?
>
> All controls results, manual as well as automatic, are now blocking for
> validation of a request. Only the CAA DNS control was not blocking. The
> risk assessment did not lead to action on the other controls. From now
> on, even if the control methodologies differ from one set of requirements
> to another, we will automatically ensure that validation is automatically
> blocked until all controls are positive, even if the risk is considered to
> be covered. by one of the methods used, and this also to ensure
> compliance. It should be noted that control of consent is the only point
> on which control practices were heterogeneous in our normative
> requirements. For the other controls, the methodologies used are
> homogeneous.

This answer does not address the controls in place at the time of the
misissuance. Any action which an operator can manually take should be able
to be audited for correctness, but I don't see any mention of that being a
possibility in your response. I take that to mean that there are no
functional audit logs, or at least that there are no effective audits
(sampling or otherwise) of those logs, for decisions and actions undertaken
by registration agents.

I'm also concerned by your use of the word "blocking" in this context. You
say that "all controls results [...] are now blocking", which I would
normally take to mean that there is no way that a registration agent would
be able to influence the issuance of a certificate in contravention of your
policies. However, in your answer to Q23 (which I've included below) you
state that there are a number of validation checks which are performed
manually, which (I presume, and please provide a detailed correction if I am
wrong) would allow for a certificate to be misissued as a result of
misunderstanding, mistake, or malfeasance on the part of a registration
agent.

Could you please explain further what exactly you mean by "blocking" in the
context of your above answer?

> 23. Which validation controls listed in your CP / CPS are not fully
> enforced by automated systems? (Your answers suggest at least one, so
> make sure your answer is complete).
>
> The following controls always involve manual control by a registration operator. The interface allows registration operator to track and confirm the validation operations they perform and their results. These manual controls relate to:
> ⦁ The entity identity validation;
> ⦁ The identity validation of the Certificate Manager and the legal representative;
> ⦁ The validation of the certificate application form with the signature of the Certificate Manager and the legal representative;
> ⦁ The realization of face-to-face with the Certificate Manager and telephone calls;
> ⦁ Validation of the identity of servers when it is a client server;

> 13. Did or Do you consider that this CPS was in line with the BRs and
> Mozilla policy?
>
> Yes. We consider that our CPS was and is in line with the BRs and Mozilla
> policy. As mentioned earlier, we have implemented CAA DNS control as
> required. We mistakenly felt that the control of the consent of the legal
> representative, which is blocking it, and the addition of alerts related
> to the CAA DNS control would meet the requirements.

I'm having trouble reconciling the first sentence in this response with the
rest of it. How could you currently consider that your CPS *was* in line
with the BRs and Mozilla policy if you know there was a material mistake in
your understanding of those documents?

> 20. Have you informed CAB/F about the specific situations in questions 17
> to 19 above.
>
> These points were exchanged as part of the referencing of our new root and
> this led to the creation of this incident ticket. If we had identified
> other cases following our audit, we would have also declared them in this
> ticket.

As far as I am aware, the CA/B Forum is not required to follow Mozilla
tickets, and thus I don't think that that constitutes informing the CA/B
Forum.

> 21. Are there any other places where your CP / CPS do or did fall short
> of the BR requirements?
>
> 22. Are there any places where your CP / CPS do or did fall short of
> Mozilla requirements beyond the BRs? (This wasn't one, it was a BR).
>
> We currently have not identified any other areas that may be subject to
> non-compliance. We regularly monitoring our practices to ensure
> compliance and you will see that we are regularly updating our practices
> and CP / CPS in line with the evolutions of BRs. In addition to these
> controls, our internal audits and annual audits by a certification body
> help to verify the compliance of our practices and documentation.

In what ways have you modified your monitoring, internal audits, and annual
audits by a certification body in light of the fact that the previous
editions of those were manifestly insufficient to identify the
non-compliance of your CAA-related procedures?

- Matt

[josselin....@gmail.com]

Hello

Thank you for your exchanges. We hope that the additions below will answer your questions.

Was the action required to manually override the CAA validation failure
different from what would be required if the CAA validation had succeeded?
Could an operator have just "clicked the same button they always clicked",
and have the CAA validation failure overridden? Or was there a separate
sequence of UI interactions required to override a CAA validation failure?

This answer does not address the controls in place at the time of the
misissuance. Any action which an operator can manually take should be able
to be audited for correctness, but I don't see any mention of that being a
possibility in your response. I take that to mean that there are no
functional audit logs, or at least that there are no effective audits
(sampling or otherwise) of those logs, for decisions and actions undertaken
by registration agents.

The check on the authorization to be issued for the organization was the only one that combined both the verification of the consent form and the verification of DNS CAA alerts. The operator could view the alert but the same validation button was used to proceed to the next step, whether the DNS CAA is valid or not.
Each operation performed by an operator is traced so that it can be audited. The periodic audits of registration requests are also intended to ensure the conduct of controls by RA and the conformity of their results.

This sounds like there may have been a misunderstanding in the BR
description of CAA validation.

What specific language in the BRs surrounding CAA validation caused you to
believe that successful CAA validation was optional in the presence of other
forms of consent?

What changes do you propose in the language of the CAA validation
requirements in the BRs to ensure that no other CA could possibly
misunderstand the CAA validation requirements?

We have no improvement to issue regarding BRs. We understood the requirement and implemented the DNS CAA control but we made two errors:
- The first is to have assessed that the existing control having the same purpose, made it possible to cover the risks and requirements related to the control of the DNS CAA. This decision led to wrongly allow RA operators to validate this checkpoint despite DNS CAA alerts.
- The second is to have accumulated these two controls (consent + DNS CAA) in the same validation step because they had the same objective, and this without imposing that the DNS CAA negative result be blocking.

Have you proposed those changes to the CA/B Forum, to improve the BRs for
all participants? If not, why not?

We would recommend to the other CAs to segment each control even if their objective is the same, in order to be sure that the result of each control is taken into account and can not be circumvented under the pretext that a control, having the same purpose, is positive.

Regarding the control on obtaining the consent of the legal representative, it is imposed by a national standard that is gradually aligned with the guidelines of the CA\B Forum, but this standard unfortunately does not evolve quickly due to administrative constraints.

We did not perceive the interest of suggesting the use of this method knowing that it is imposed only on the scale of France and not for other international CA and that the control of the DNS CAA had in our opinion the same purpose .

We want as much as possible that the different national, European and international standards are harmonized in order to avoid the implementation of different controls but for the same purposes.

That the BRs have been misunderstood in this fashion is somewhat concerning.
it suggests that there may be other misunderstandings latent in your systems
and processes, which haven't surfaced because they haven't (yet) led to an
identified misissuance or other incident.

What steps have you taken, or are you planning on taking, to address
possible misunderstandings of the BRs or Mozilla policy that may cause
Certigna to unintentionally misissue a certificate or suffer from some other
incident?

In what ways have you modified your monitoring, internal audits, and annual
audits by a certification body in light of the fact that the previous
editions of those were manifestly insufficient to identify the
non-compliance of your CAA-related procedures ?

To address the two errors mentioned above, we have updated the RA Validation Portal to separate the DNS CAA control and make it blocking automatically. The other validation steps are well separated.

In addition to the audit we conducted following the DNS CAA incident, we are conducting the annual internal audit of our practices in order to verify, as each year, their compliance with the requirements of the BRs and EV Guidelines. It is also expected that new auditors will be involved, both in the internal audit and in the certification audit planned for this year.

We also made aware and reminded the following points to all the actors of the organization and in particular the members of the security committee:
- The incident encountered and its origins;
- The need to ensure compliance with the requirements of the CA/Browser Forum;
- The need to report any non-compliance or incident encountered in order to notify the Forum;
- Update any changes to the self-assessment file to ensure compliance with the requirements;
- The need to comply with each requirement even if it is considered that an already existing practice but different can cover the requirement.

The communication process has been updated to specify in more detail how information is sent to the CA\B FORUM via the different portals available.

The certification body in charge of our audit includes in these audit criteria the ETSI standards but also the BR and EV guidelines as we requested. As said above, new auditors are selected for our next certification audit.

I'm also concerned by your use of the word "blocking" in this context. You
say that "all controls results [...] are now blocking", which I would
normally take to mean that there is no way that a registration agent would
be able to influence the issuance of a certificate in contravention of your
policies. However, in your answer to Q23 (which I've included below) you
state that there are a number of validation checks which are performed
manually, which (I presume, and please provide a detailed correction if I am
wrong) would allow for a certificate to be misissued as a result of
misunderstanding, mistake, or malfeasance on the part of a registration
agent.

Could you please explain further what exactly you mean by "blocking" in the
context of your above answer?

Controls are automatic such as DCV or DNS CAA and others are not and are manually performed by the operators as mentioned above. Whether the control is done automatically or manually, if the result of only one of these checks is negative, the request is "blocked" and can not be validated by a RA operator. No validation and authorization to be issued is possible by a RA operator if the result of one of the manual or automatic control points is negative. The request is in an "Invalid" state while waiting for corrections or proofs to be made to the request.

I'm having trouble reconciling the first sentence in this response with the
rest of it. How could you currently consider that your CPS *was* in line
with the BRs and Mozilla policy if you know there was a material mistake in
your understanding of those documents?

Our CPS is not limited only to BR compliance but also to other security requirements, and we believe that the accumulation of existing controls with those of the BRs would serve to meet the objectives of the requirements in question. We are not only committed to taking control of the DNS CAA, but also to carry out the prior checking on the obtaining of the consent by the legal representative of the organization, these two points aim at feeding the control on the authorization to be issued by the organization. So to answer you, yes the practices documented in our CPS were not exactly those presented by the BRs, but this results from the fact that we have additional practices to those expected by the BR and sincerely we thought that this set allowed to answer to BR expectations.

We are now aware that the implementation of additional measures from other standards is not an exception and should not affect the implementation of the controls required by the BRs and that is why the control of DNS CAA is now independent of any other step of validation and control, and if its result is negative the validation of the request is blocked.

As far as I am aware, the CA/B Forum is not required to follow Mozilla
tickets, and thus I don't think that that constitutes informing the CA/B
Forum.

Indeed, it has not been reported by us, and we apologize for it because we sincerely thought that all our practices could meet the requirements on this point. Following the recovery of this non-compliance, we conducted an audit to identify any problem that led to the identification of this certificate.

We are now aware of the need to improve our processes and have strengthened our communication procedures to ensure a report of non-conformities and incidents identified, as well as possible evolutions and practices that we would like to suggest to the forum.

We remain at your disposal if you want further information.

Best regards

[Matt Palmer]

On Wed, Sep 26, 2018 at 07:36:57AM -0700, josselin.allemandou--- via dev-security-policy wrote:
> Thank you for your exchanges. We hope that the additions below will answer your questions.

It appears that your response has removed most indications of what parts of
your message are my questions, compared to your responses. For clarity,
I've re-added appropriate formatting, but it would be appreciated if you
could use standard e-mail formatting in the future.

> > Was the action required to manually override the CAA validation failure
> > different from what would be required if the CAA validation had succeeded?
> > Could an operator have just "clicked the same button they always clicked",
> > and have the CAA validation failure overridden? Or was there a separate
> > sequence of UI interactions required to override a CAA validation failure?
> >
> > This answer does not address the controls in place at the time of the
> > misissuance. Any action which an operator can manually take should be able
> > to be audited for correctness, but I don't see any mention of that being a
> > possibility in your response. I take that to mean that there are no
> > functional audit logs, or at least that there are no effective audits
> > (sampling or otherwise) of those logs, for decisions and actions undertaken
> > by registration agents.
>
> The check on the authorization to be issued for the organization was the
> only one that combined both the verification of the consent form and the
> verification of DNS CAA alerts. The operator could view the alert but the
> same validation button was used to proceed to the next step, whether the
> DNS CAA is valid or not.

Thank you for this clear statement of your validation interface design.
Unfortunately, this sounds like a design which is extremely risky, from an
unintentional errors perspective. What form(s) of review for UX, including
but not limited to human factors of safety, were applied to this interface
prior to it being deployed?

> Each operation performed by an operator is traced so that it can be
> audited. The periodic audits of registration requests are also intended
> to ensure the conduct of controls by RA and the conformity of their
> results.

Based on your initial report, I got the impression that the misissuance
we're discussing was not picked up by an ordinary operational audit, but was
instead identified by some sort of extraordinary review. Is that an
accurate impression? If so, can you provide more detail around the
criteria you use for selecting operations for auditing, and the frequency of
those audits, with particular reference to how such an unusual event
(overriding a CAA validation failure) wasn't picked up by ordinary auditing?

> > This sounds like there may have been a misunderstanding in the BR
> > description of CAA validation.
> >
> > What specific language in the BRs surrounding CAA validation caused you to
> > believe that successful CAA validation was optional in the presence of other
> > forms of consent?
> >
> > What changes do you propose in the language of the CAA validation
> > requirements in the BRs to ensure that no other CA could possibly
> > misunderstand the CAA validation requirements?
>
> We have no improvement to issue regarding BRs. We understood the
> requirement and implemented the DNS CAA control but we made two errors:
>
> - The first is to have assessed that the existing control having the same
> purpose, made it possible to cover the risks and requirements related to
> the control of the DNS CAA. This decision led to wrongly allow RA
> operators to validate this checkpoint despite DNS CAA alerts.
>
> - The second is to have accumulated these two controls (consent + DNS CAA)
> in the same validation step because they had the same objective, and
> this without imposing that the DNS CAA negative result be blocking.

I'm unable to reconcile your two statements. "We understood the
requirements" cannot be true at the same time as you made errors in
understanding the requirements. It's not logically possible for both of
those to be true at the same time.

That the misunderstand occured is not at issue. The important question that
needs answering is: how did the misunderstanding occur? Perhaps the BRs are
insufficiently clear, either on the particulars of the need for CAA
validation, or in the larger sense that the prescriptive controls laid out
in the BRs cannot be substituted for controls which the CA happens to feel
are equivalent. If that is the case, then the BRs need to be clarified in
some way, so that other CAs cannot suffer from the same misunderstanding in
the future.

Alternately, if the BRs *are*, in fact, sufficiently clear in all respects,
the only other possibility that comes to my mind is that Certigna failed to
correctly interpret the BRs, which is far more concerning -- for Certigna,
at least. It would mean that there could be any number of other, as yet
unidentified, misunderstandings in Certigna's procedures. I would imagine
there would need to be a very comprehensive review of Certigna's processes
and procedures, with a detailed public report of the findings of that
review, for confidence in Certigna to be restored.

> > Have you proposed those changes to the CA/B Forum, to improve the BRs for
> > all participants? If not, why not?
>
> We would recommend to the other CAs to segment each control even if their
> objective is the same, in order to be sure that the result of each control
> is taken into account and can not be circumvented under the pretext that a
> control, having the same purpose, is positive.

At the risk of re-iterating the previous point, I'm still at a loss to
understand what led Certigna management to believe that substitution of
controls that were deemed equivalent was permitted by the BRs or Mozilla
policy?

> Regarding the control on obtaining the consent of the legal
> representative, it is imposed by a national standard that is gradually
> aligned with the guidelines of the CA\B Forum, but this standard
> unfortunately does not evolve quickly due to administrative constraints.
>
> We did not perceive the interest of suggesting the use of this method
> knowing that it is imposed only on the scale of France and not for other
> international CA and that the control of the DNS CAA had in our opinion
> the same purpose .
>
> We want as much as possible that the different national, European and
> international standards are harmonized in order to avoid the
> implementation of different controls but for the same purposes.

Fascinating though this is, including this aside does not really improve the
impression you're giving of Certigna's commitment to adhere to the
requirements of the Mozilla root store policy. That it may be inconvenient,
or expensive, to comply with the requirements of multiple regulatory
regimes, does not excuse you from doing so. Certigna's choices in operating
its business are exactly that -- Certigna's choices -- and expense or
inconvenience do not justify failure to adhere to the requirements.

If there are fundamental incompatibilities between regulatory regimes, the
BRs provide a mechanism for drawing attention to those incompatibilities,
and if Certigna has made use of that mechanism, I would appreciate a
reference to that.

> > That the BRs have been misunderstood in this fashion is somewhat concerning.
> > it suggests that there may be other misunderstandings latent in your systems
> > and processes, which haven't surfaced because they haven't (yet) led to an
> > identified misissuance or other incident.
> >
> > What steps have you taken, or are you planning on taking, to address
> > possible misunderstandings of the BRs or Mozilla policy that may cause
> > Certigna to unintentionally misissue a certificate or suffer from some other
> > incident?
> >
> > In what ways have you modified your monitoring, internal audits, and annual
> > audits by a certification body in light of the fact that the previous
> > editions of those were manifestly insufficient to identify the
> > non-compliance of your CAA-related procedures ?
>
> To address the two errors mentioned above, we have updated the RA
> Validation Portal to separate the DNS CAA control and make it blocking
> automatically. The other validation steps are well separated.

Can you expand on how changing the RA portal around the DNS CAA control
addresses any *other* potential vectors for misissuance due to Certigna's
misunderstanding of the BRs or Mozilla policy? Given that you've made this
response in regards to my questions surrounding these other possible
misunderstandings, I assume there is a link, but I'm afraid I can't see it.

> The certification body in charge of our audit includes in these audit
> criteria the ETSI standards but also the BR and EV guidelines as we
> requested. As said above, new auditors are selected for our next
> certification audit.

This statement suggests that Certigna believes there was some degree of
blame to be attached to Certigna's auditors regarding this misissuance
(otherwise why raise it as part of this discussion?). If Certigna believes
that their auditors are partially at fault, can you expand on why that is?
Were they simply incompetent in their duties? If so, have you reported this
incompetence to either Mozilla or the CA/B Forum, so that appropriate steps
can be taken? If you do not believe the auditor to have been incompetent,
what leads you to believe that selecting alternate auditors will necessarily
lead to a more satisfactory outcome? Are their changes or improvements to
the audit criteria that auditors are required to follow that you can
suggest, to prevent future occurrences of this kind?

Frankly, blaming the auditor smacks of scapegoating to me; it's like that
sequence in Monty Python's Quest for the Holy Grail: "the people responsible
for sacking the people responsible have themselves been sacked". It doesn't
solve any fundamental problem within the organisation to get new auditors.

> > I'm also concerned by your use of the word "blocking" in this context. You
> > say that "all controls results [...] are now blocking", which I would
> > normally take to mean that there is no way that a registration agent would
> > be able to influence the issuance of a certificate in contravention of your
> > policies. However, in your answer to Q23 (which I've included below) you
> > state that there are a number of validation checks which are performed
> > manually, which (I presume, and please provide a detailed correction if I am
> > wrong) would allow for a certificate to be misissued as a result of
> > misunderstanding, mistake, or malfeasance on the part of a registration
> > agent.
> >
> > Could you please explain further what exactly you mean by "blocking" in the
> > context of your above answer?
>
> Controls are automatic such as DCV or DNS CAA and others are not and are
> manually performed by the operators as mentioned above. Whether the
> control is done automatically or manually, if the result of only one of
> these checks is negative, the request is "blocked" and can not be
> validated by a RA operator. No validation and authorization to be issued
> is possible by a RA operator if the result of one of the manual or
> automatic control points is negative. The request is in an "Invalid"
> state while waiting for corrections or proofs to be made to the request.

Thanks for that clarification.

- Matt

[Wayne Thayer]

Hi Matt,

I appreciate your interest in getting to the root causes of this issue, and
the polite and professional manner in which you are asking questions.
However, I am concerned that this line of questioning seem to have reached
the limits of Certigna's analysis capabilities, and is thus unlikely to
uncover much more information that helps others in our community to
improve. I think it is time that we draw conclusions from the information
we have rather than continuing a press for answers that can be
misinterpreted as an attack on the CA. If you would like recommend specific
actions from Certigna or Mozilla, please do so.

I have included more specific comments inline, and I would welcome a
meaningful response from Certigna to any of Matt's questions or my comments.

- Wayne

> If the implication here is that CAs should apply a high level of UX
expertise to their internal validation interfaces, I would wager that very
few CAs meet your standards.

> Each operation performed by an operator is traced so that it can be
> > audited. The periodic audits of registration requests are also intended
> > to ensure the conduct of controls by RA and the conformity of their
> > results.
>
> Based on your initial report, I got the impression that the misissuance
> we're discussing was not picked up by an ordinary operational audit, but
> was
> instead identified by some sort of extraordinary review. Is that an
> accurate impression? If so, can you provide more detail around the
> criteria you use for selecting operations for auditing, and the frequency
> of
> those audits, with particular reference to how such an unusual event
> (overriding a CAA validation failure) wasn't picked up by ordinary
> auditing?
>

> Agreed - the misissuance was caught by an audit that was only performed
after Certigna's CAA validation practices were questioned. CAs are required
to audit 3% of issued certificates (BR section 8.7), and I would be
surprised if there are many that exceed that requirement, so this seems
like an industry-wide concern.

> I think we have established that the problem was with Certigna's chosen
interpretation of the BRs. I am not clear on how you are proposing to have
a "comprehensive review of Certigna's processes and procedures, with a
detailed public report of the findings of that review" performed. This
sounds like an audit to me? Certigna's audit period ends on 20-October, so
their next assessment reports are due relatively soon.

> > Have you proposed those changes to the CA/B Forum, to improve the BRs
> for
> > > all participants? If not, why not?
> >
> > We would recommend to the other CAs to segment each control even if their
> > objective is the same, in order to be sure that the result of each
> control
> > is taken into account and can not be circumvented under the pretext that
> a
> > control, having the same purpose, is positive.
>
> At the risk of re-iterating the previous point, I'm still at a loss to
> understand what led Certigna management to believe that substitution of
> controls that were deemed equivalent was permitted by the BRs or Mozilla
> policy?
>

> A key point that I hope has been made here is that compensating controls
are not a substitute for meeting BR requirements. I don't know what more
there is to say on this.

> I agree. I don't think a conflict exists, but if it does, Certigna needs
to describe it in detail both here and to the CA/Browser Forum. Otherwise,
this line of reasoning is misguided.

> It seems more likely that Certigna simply doesn't have an answer to your
question.

> The certification body in charge of our audit includes in these audit
> > criteria the ETSI standards but also the BR and EV guidelines as we
> > requested. As said above, new auditors are selected for our next
> > certification audit.
>
> This statement suggests that Certigna believes there was some degree of
> blame to be attached to Certigna's auditors regarding this misissuance
> (otherwise why raise it as part of this discussion?). If Certigna believes
> that their auditors are partially at fault, can you expand on why that is?
> Were they simply incompetent in their duties? If so, have you reported
> this
> incompetence to either Mozilla or the CA/B Forum, so that appropriate steps
> can be taken? If you do not believe the auditor to have been incompetent,
> what leads you to believe that selecting alternate auditors will
> necessarily
> lead to a more satisfactory outcome? Are their changes or improvements to
> the audit criteria that auditors are required to follow that you can
> suggest, to prevent future occurrences of this kind?
>

> I think that Certigna is just arguing that audits will uncover any other
problems that they might have.

Frankly, blaming the auditor smacks of scapegoating to me; it's like that
> sequence in Monty Python's Quest for the Holy Grail: "the people
> responsible
> for sacking the people responsible have themselves been sacked". It
> doesn't
> solve any fundamental problem within the organisation to get new auditors.
>

> I think we are both in agreement here that Certigna needs to take
responsibility for strict adherence to Mozilla's policies, and they need to
implement policies and processes that ensure compliance.

[Matt Palmer]

On Wed, Oct 03, 2018 at 09:31:08AM -0700, Wayne Thayer wrote:
> On Mon, Oct 1, 2018 at 4:49 AM Matt Palmer via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
> > Thank you for this clear statement of your validation interface design.
> > Unfortunately, this sounds like a design which is extremely risky, from an
> > unintentional errors perspective. What form(s) of review for UX, including
> > but not limited to human factors of safety, were applied to this interface
> > prior to it being deployed?
> >
> If the implication here is that CAs should apply a high level of UX
> expertise to their internal validation interfaces, I would wager that very
> few CAs meet your standards.

Whilst I would not want to take the other side of that wager, it's still
unfortunate that safety-critical UX is treated with such a cavalier attitude
across the industry.

It's useful, for the ecosystem, that Certigna has been willing to identify
this publicly as a risk in this instance, specifically *because* I agree
with you -- that very few CAs probably do meet my standards in UX design and
review -- and it is an important thing that all CAs *should*, ideally, be
taking into account.

My standards, incidentally, are informed by the approaches taken by other
safety-critical industries, which have seen great benefits from taking UX
seriously, and I'd like to see a similar approach taken by CAs who wish to
participate in the web PKI.

> > > Each operation performed by an operator is traced so that it can be
> > > audited. The periodic audits of registration requests are also intended
> > > to ensure the conduct of controls by RA and the conformity of their
> > > results.
> >
> > Based on your initial report, I got the impression that the misissuance
> > we're discussing was not picked up by an ordinary operational audit, but
> > was instead identified by some sort of extraordinary review. Is that an
> > accurate impression? If so, can you provide more detail around the
> > criteria you use for selecting operations for auditing, and the
> > frequency of those audits, with particular reference to how such an
> > unusual event (overriding a CAA validation failure) wasn't picked up by
> > ordinary auditing?
> >
> Agreed - the misissuance was caught by an audit that was only performed
> after Certigna's CAA validation practices were questioned. CAs are required
> to audit 3% of issued certificates (BR section 8.7), and I would be
> surprised if there are many that exceed that requirement, so this seems
> like an industry-wide concern.

Perhaps the BRs, or Mozilla policy, needs to specify some increased
vigilance over certificates whose issuance required any sort of override or
deviation from the "standard" practice? (Tricky to define "standard"
practice in a useful fashion, I know...)

Not that such a policy would have been likely to catch this particular
problem any quicker, since this specific issue under discussion wasn't due
to a mistake or malfeasance by an RA, but was instead due to a systemic,
organisation-wide misinterpretation of the BRs. In general, though, when
unusual things happen, it's useful to pay closer attention to what's going
on, as the chances of mistakes are much higher, and they happen rarely
enough that the standard 3% sampling is unlikely to catch them by accident.

> > Alternately, if the BRs *are*, in fact, sufficiently clear in all respects,
> > the only other possibility that comes to my mind is that Certigna failed to
> > correctly interpret the BRs, which is far more concerning -- for Certigna,
> > at least. It would mean that there could be any number of other, as yet
> > unidentified, misunderstandings in Certigna's procedures. I would imagine
> > there would need to be a very comprehensive review of Certigna's processes
> > and procedures, with a detailed public report of the findings of that
> > review, for confidence in Certigna to be restored.
>
> I think we have established that the problem was with Certigna's chosen
> interpretation of the BRs. I am not clear on how you are proposing to have
> a "comprehensive review of Certigna's processes and procedures, with a
> detailed public report of the findings of that review" performed. This
> sounds like an audit to me?

Not by my understanding of an audit (particularly a WebTrust audit). A
WebTrust audit will validate that the organisation does what it says it
does, but I can't see how it would identify that management has
misinterpreted the BRs, in any but the most egregious of ways. I doubt ETSI
is any better, given that the impression I get is that those audits are even
less closely aligned with the specific requirements of the BRs.

What I'm envisioning in my suggestion is a review of all Certigna's BR-related
processes and procedures, with a view to identifying any potential issues
based on the two factors already identified:

1. That Certigna misinterpreted the BRs to believe that they could
substitute controls they considered to be equivalent for BR-mandated
controls. Since it is entirely possible that this same misinterpretation
may have coloured other aspects of their operations, I would expect a
review of all processes and procedures to identify whether they, too, could
have suffered from the same misinterpretation in their design or
implementation.

2. That Certigna managed to misinterpret the BRs *in general*. Thus, what
other of Certigna's processes and procedures could possibly have been
influenced by other misinterpretations of the BRs? This may require an
external party, or at least someone who wasn't involved in the initial
analysis of the BRs to determine Certigna's processes and procedures, to
ensure that prior misinterpretations are not repeated.

> > > We would recommend to the other CAs to segment each control even if
> > > their objective is the same, in order to be sure that the result of
> > > each control is taken into account and can not be circumvented under
> > > the pretext that a control, having the same purpose, is positive.
> >
> > At the risk of re-iterating the previous point, I'm still at a loss to
> > understand what led Certigna management to believe that substitution of
> > controls that were deemed equivalent was permitted by the BRs or Mozilla
> > policy?
>
> A key point that I hope has been made here is that compensating controls
> are not a substitute for meeting BR requirements. I don't know what more
> there is to say on this.

Without understanding how an error occurred, it is significantly more
difficult to comprehensively remediate a problem, and also to provide
guidance for others in equivalent, but not identical, situations. "Someone
at Certigna made a mistake" would appear to be the *proximate* cause of the
problem, but I don't really care that someone made a mistake. I'm keen to
know *why* the mistake was made.

As an example of one *possible* underlying cause: was the person who read
the BRs in this case a non-native English speaker, and some problematic
phrasing tripped them up? Idioms which seem perfectly comprehensible to a
native speaker can cause all manner of confusion (my use of Australianisms
regularly trips up my professional colleagues). If that were to be the
cause of this misinterpretation, then it would behoove other CAs to be
especially mindful of that possibility in their own operations, and consider
mitigations (get a native English speaker to review high-risk areas of
interpretation, for example). Even better (from a risk mitigation
perspective) would be to fix the BRs to remove all instances of the
troublesome phrasing entirely, and give notice to drafters of future BR
amendments to avoid specific problematic language constructions.

All of that useful remediation can't take place, though, unless the
underlying cause of the mistake can be identified. To repeat: I don't care
that someone at Certigna made a mistake. That's done, and can't be fixed.
What *can* be fixed is the circumstances that allowed the mistake to happen,
so that it (and other similar mistakes) can't happen in the future -- to
Certigna or anyone else.

> > > The certification body in charge of our audit includes in these audit
> > > criteria the ETSI standards but also the BR and EV guidelines as we
> > > requested. As said above, new auditors are selected for our next
> > > certification audit.
> >
> > This statement suggests that Certigna believes there was some degree of
> > blame to be attached to Certigna's auditors regarding this misissuance
> > (otherwise why raise it as part of this discussion?). If Certigna believes
> > that their auditors are partially at fault, can you expand on why that is?
> > Were they simply incompetent in their duties? If so, have you reported
> > this
> > incompetence to either Mozilla or the CA/B Forum, so that appropriate steps
> > can be taken? If you do not believe the auditor to have been incompetent,
> > what leads you to believe that selecting alternate auditors will
> > necessarily
> > lead to a more satisfactory outcome? Are their changes or improvements to
> > the audit criteria that auditors are required to follow that you can
> > suggest, to prevent future occurrences of this kind?
>
> I think that Certigna is just arguing that audits will uncover any other
> problems that they might have.

That would be a troubling argument in and of itself, if it were made.
There's a long and distinguished history of auditors missing all sorts of
problems at CAs, and I'd be worried about anyone who has been on the
receiving end of an AICPA-overseen security audit who thinks the auditors
are a useful adjunct to their security program. Certainly, any organisation
that were to *rely* on auditors to find their problems for them would be
disconcertingly misguided, almost dangerously so.

- Matt

[Jakob Bohm]

On 04/10/2018 04:27, Matt Palmer wrote:
> On Wed, Oct 03, 2018 at 09:31:08AM -0700, Wayne Thayer wrote:
>> On Mon, Oct 1, 2018 at 4:49 AM Matt Palmer via dev-security-policy <
>> dev-secur...@lists.mozilla.org> wrote:

>>> ...
>> ...
> ...

I seem to recall that the bad practice was explicitly called out in
their (old) CP/CPS, which was applicable at the time. Thus any similar
misunderstanding should be discoverable by Mozilla and/or their auditor
comparing the CP/CPS with the BR, Mozilla, National and other applicable
requirements. However this has been a long discussion and some posts
have been expired by the mozilla NNTP server.

> ...

[Wayne Thayer]

On Wed, Oct 3, 2018 at 7:27 PM Matt Palmer via dev-security-policy <

dev-secur...@lists.mozilla.org> wrote:

> On Wed, Oct 03, 2018 at 09:31:08AM -0700, Wayne Thayer wrote:
> > On Mon, Oct 1, 2018 at 4:49 AM Matt Palmer via dev-security-policy <
> > dev-secur...@lists.mozilla.org> wrote:

> Aside from clearly articulating the value of UX design in reducing risk as
you have done, do you have any suggestions for how we could encourage
improvements in this area? It doesn't seem like something that could very
easily be made into a policy.

> Agree - we could perhaps require management review of any deviation from
the standard process, but something like that would be easy to sidestep.

Not that such a policy would have been likely to catch this particular
> problem any quicker, since this specific issue under discussion wasn't due
> to a mistake or malfeasance by an RA, but was instead due to a systemic,
> organisation-wide misinterpretation of the BRs. In general, though, when
> unusual things happen, it's useful to pay closer attention to what's going
> on, as the chances of mistakes are much higher, and they happen rarely
> enough that the standard 3% sampling is unlikely to catch them by accident.
>

> WebTrust for CAs is essentially "we do what our CPS says we do", but
WebTrust Baseline Requirements [1] verifies that the BRs are being
followed. ETSI covers BR requirements in 319 411-1 [2].

[1] http://www.webtrust.org/principles-and-criteria/docs/item83987.pdf
[2]
https://www.etsi.org/deliver/etsi_en/319400_319499/31941101/01.02.02_60/en_31941101v010202p.pdf

What I'm envisioning in my suggestion is a review of all Certigna's
> BR-related
> processes and procedures, with a view to identifying any potential issues
> based on the two factors already identified:
>
> 1. That Certigna misinterpreted the BRs to believe that they could
> substitute controls they considered to be equivalent for BR-mandated
> controls. Since it is entirely possible that this same
> misinterpretation
> may have coloured other aspects of their operations, I would expect a
> review of all processes and procedures to identify whether they, too,
> could
> have suffered from the same misinterpretation in their design or
> implementation.
>

> We do have the BR Self-Assessments. Certigna recently updated theirs [3],
but it still mistakenly refers to the practice of overriding CAA checks in
section 3.2.2.8.

[3] https://bugzilla.mozilla.org/attachment.cgi?id=9007984

2. That Certigna managed to misinterpret the BRs *in general*. Thus, what
> other of Certigna's processes and procedures could possibly have been
> influenced by other misinterpretations of the BRs? This may require an
> external party, or at least someone who wasn't involved in the initial
> analysis of the BRs to determine Certigna's processes and procedures, to
> ensure that prior misinterpretations are not repeated.
>

> I don't know of a better way to accomplish this than a BR audit.

> I understand your point, but Certigna seems unable or unwilling to perform
this level of analysis. I am inferring that someone decided it would be
okay to substitute a compensating control. Certigna has danced around this
question, but perhaps a direct question that Certigna can answer is the
following:

Josselin: Will you please explain how the decision was made to allow "the
authorization to issue the certificate by a legal representative of the
entity which is owner of the domain names" to override the results of a CAA
query, and why that was believed to be permitted under the BRs and Mozilla
policy?

> I agree.

- Matt
>
>

[Wayne Thayer]

On Thu, Oct 4, 2018 at 9:48 AM Jakob Bohm via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> I seem to recall that the bad practice was explicitly called out in
> their (old) CP/CPS, which was applicable at the time. Thus any similar
> misunderstanding should be discoverable by Mozilla and/or their auditor
> comparing the CP/CPS with the BR, Mozilla, National and other applicable
> requirements. However this has been a long discussion and some posts
> have been expired by the mozilla NNTP server.
>

> Devon discovered this during his review of Certigna's root inclusion
request:
https://groups.google.com/d/msg/mozilla.dev.security.policy/z7iDk9CdTFo/9zQpW1-bCwAJ

[Wayne Thayer]

[Jakob Bohm]

On 04/10/2018 19:40, Wayne Thayer wrote:
> On Thu, Oct 4, 2018 at 9:48 AM Jakob Bohm via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
>

(In reply to Matt Palmer in message-id
mailman.78.1538620059.2...@lists.mozilla.org)

>> I seem to recall that the bad practice was explicitly called out in
>> their (old) CP/CPS, which was applicable at the time. Thus any similar
>> misunderstanding should be discoverable by Mozilla and/or their auditor
>> comparing the CP/CPS with the BR, Mozilla, National and other applicable
>> requirements. However this has been a long discussion and some posts
>> have been expired by the mozilla NNTP server.
>>
>> Devon discovered this during his review of Certigna's root inclusion
> request:
> https://groups.google.com/d/msg/mozilla.dev.security.policy/z7iDk9CdTFo/9zQpW1-bCwAJ
>

I was merely suggesting that since the mistake could be found with that
review of the CPS (to which they are already audited), this would be a
more effective way to conduct the review of their general BR
understanding or lack thereof, as requested by Matt Palmer.

[josselin....@gmail.com]

Hello,

The decision was taken at one of our security committees where all changes and developments that could impact the practices and compliance of our authority are validated. This is why all the actors of these security committees have been made aware of the incident and the fact that we can not make any exception to BR practices on the pretext that we have other additional measures implemented in our organization, in addition to all the other actions listed above.

Wayne: Indeed, thank you for this return, I had corrected only chapter 4.2.1 in alignment with our CP / CPS. The file has been updated at chapter 3.2.2.8 and is available at the following address: https://bugzilla.mozilla.org/attachment.cgi?id=9017115

Best regards