Auditing of CA facilities in lockdown because of an environmental disaster/pandemic

[Arvid Vermote]

COVID-19 is going on and there currently is a quarantine of certain areas in
China and also alert levels are further raising in other (mainly East-Asian)
countries.



How will the root programs approach CA facilities with key material that are
in a lockdown or in a territory that is not accessible by auditors and hence
cannot be inspected within the three months after the end of the CA's
period-under-audit?



Lockdown in the above meaning properly secured according to the requirements
and BCP/DR plans but because of external conditions not accessible and
available for external auditors / inspection.

[Ryan Sleevi]

What would/should be the expected response if a natural disaster/act of God
happened and the security of the key material could not be assured by an
independent third party?

For example, an earthquake, typhoon, or military coup disrupting travel to
location(s) with the key material?

Similarly, what would/should happen if a primary location was compromised,
but that compromise not detected due to a fire in the primary location
disrupting access to the security logs, leading to misissued certificates
being trusted and the CA being unaware of their (mis)issuance?

Are there any suggestions for how would/should these two hypotheticals be
distinguished? Wait until it’s detected? Certificate Transparency is not
sufficient in itself, due to the lifetime of certificates and the ability
to backdate certificates so that they appear issued prior to the effective
date of such CT requirements, so CT is not yet a proper mitigation.

[Kathleen Wilson]

All,

First, I would like to add a personal note that I am truly sorry about
the many people, families, and colleagues that are being impacted by the
Coronavirus. This is a heartbreaking situation.

At Mozilla, our responsibility lies in ensuring people's security and
privacy as they navigate the internet. Protecting our users and the
integrity of the web is the reason Firefox exists. The best approach to
do this is to work with certificate authorities as partners, through
open and frank communication.

We will continue to follow our standard process to adjudicate the issue
regarding failures to provide CA audit statements [1] and we will work
with the impacted CAs throughout this process. Pursuant to this process,
Mozilla will file CA incident bugs [2] in Bugzilla when audit statements
are past due. The CA should respond in such bugs providing their
Incident Report [3] explaining the situation with their audits,
precautions that have been taken and their plan to move forward in
reaching compliance again.

If it would be helpful, we could add a note in the Bugzilla whiteboard
to indicate when the delayed audit statements are caused by CAs and
auditors being unable to access facilities to perform the audits due to
circumstances beyond their control. For example, the whiteboard could be
something like: “[ca-compliance] Lockdown - Next Update <date>”. I will
greatly appreciate thoughtful and constructive feedback on this.

Thanks,
Kathleen

References:
[1] https://www.ccadb.org/cas/updates
[2] https://wiki.mozilla.org/CA/Incident_Dashboard
[3] https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report

[Ryan Sleevi]

On Thu, Feb 20, 2020 at 4:58 PM Kathleen Wilson via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> We will continue to follow our standard process to adjudicate the issue
> regarding failures to provide CA audit statements [1] and we will work
> with the impacted CAs throughout this process. Pursuant to this process,
> Mozilla will file CA incident bugs [2] in Bugzilla when audit statements
> are past due. The CA should respond in such bugs providing their
> Incident Report [3] explaining the situation with their audits,
> precautions that have been taken and their plan to move forward in
> reaching compliance again.

I think more guidance is still needed here, and my questions were trying to
solicit feedback as to how CAs affected would propose to address this.

For example, should the CA produce an audit report with the locations that
were examined? Or should they defer the audit report until all locations
have been examined?

What constitutes sufficient evidence of a location being affected?
Similarly, if a CA’s preferred auditors are from a region affected by
travel restrictions, but other accredited auditors exist that would be
capable, would that be sufficient?

If an audit for a region is delayed, but travel in that region becomes no
longer affected, at what point is a new audit expected? And how is that
decided?

These are examples of the questions trying to accommodate this, and it
would be good to see more holistic responses about how this *could* be
addressed, especially when considering an adversarial model where a CA
might opportunistically exploit this or future situations. Presumably, the
CA has thought through some of this, as part of their general business
continuity / disaster recovery plans, and that’s why I framed the original
questions the way I did, to get more transparency about how the CA(s)
affected have prepared.

>

[Ryan Sleevi]

Hi Arvid,

I wanted to follow-up, and see if you had suggestions or ideas here for
appropriate next steps. Understandably, as more countries are affected,
this will no doubt continue to be an issue. I think you're spot on for
asking early, as you did, and I'm hoping GlobalSign (and others!) might
have ideas on appropriate risk mitigation and potential remediation
strategies.

>

[Arvid Vermote]

When I initially raised the topic I had two things in mind:

- What if a facility can’t be audited?

- If main key management facilities are down can WebPKI CA meet SSLBR 4.9.1.2?



As for the inability to audit, a few things come to mind based on the previous shared thoughts:

- Inform root programs ASAP if a certain location is in a situation where it cannot be inspected within the three months after the period under audit

- File an “exception request” (which requires to be renew regularly to ensure CAs need to continuously justify an incomplete audit)

- Complete the audit for all locations that can be audited

- Deliver updated report that incorporates the facilities as soon as the facility is back available for inspection



Ryan, related to your thought “Similarly, if a CA’s preferred auditors are from a region affected by travel restrictions, but other accredited auditors exist that would be capable, would that be sufficient?”

- Auditors are not that flexible on reporting formats and doing a specific subset of a standards on a specific location.

- What would the root programs accept in terms of such an ad-hoc report as it will not be an ISAE3000/CSAE3000/SSAE18 format?

- Depending on the CA it would also be multiple reports that will be incomplete: WebTrust, SSLBR, EV, (EV) Code Signing

- Which controls / criteria should be reported on? Only the ones related to physical security?



For the key material security and key management continuity aspect, compared to the controls I would think a typical CA would implement the WebTrust standard is quite brief and vague:

- Criterion #3.8 focuses on general CA continuity and availability of technology and information. For key material it focuses on having back-ups.

- There is one specific control (#3.8.6) that talks a bit about securing a facility during a disaster



None of these controls really talk focused about the importance of or set any timings for having a capability available at original or remote site to perform critical key management activities such as timely ICA revocation. Also generally our opinion is that key material protection requirements in WT are substandard to the level of protection that is required for WebPKI CAs.



Based on our internal risk appetite we have implemented additional controls, including but not limited to:

- Having key management facilities / capability on different continents in politically stable countries

- Having additional locations on top of the above facilities under different political rule to which we can move key material quickly and securely in case of any threat or instability

- “Key management facility” means a facility where key material is stored. Credentials to unlock the key management facility and key material are stored in at least two other different locations in close proximity to the key management facility and require the presence of different roles to obtain access.

- Rotational key management activities in the different locations to make sure the facilities are and stay operational and plans work when it comes to a BCP execution

- A colluded group of at least six trusted role people is required in order to obtain access to key management material

[Ryan Sleevi]

Thanks Arvid! I think these are good starting points for discussion!

On Wed, Mar 4, 2020 at 8:48 AM Arvid Vermote <arvid....@globalsign.com>
wrote:

> When I initially raised the topic I had two things in mind:
>
> - What if a facility can’t be audited?
>
> - If main key management facilities are down can WebPKI CA meet
> SSLBR 4.9.1.2?
>
>
>
> As for the inability to audit, a few things come to mind based on the
> previous shared thoughts:
>
> - Inform root programs ASAP if a certain location is in a
> situation where it cannot be inspected within the three months after the
> period under audit
>

I'm not sure I understand where the three months comes from. Is this based
on the fact that the final report doesn't need to be delivered for up to 3
months from the end of the audit periods?

Broadly, I would say "ASAP" without qualifications ;)

> - File an “exception request” (which requires to be renew
> regularly to ensure CAs need to continuously justify an incomplete audit)
>

I'm not sure why CAs are so fond of this phrase, but since the horse, now
thoroughly glue, continues to need to be beaten... =)

Exceptions to the Baseline Requirements cannot and are not granted. They
are, without question, incidents and non-compliance. That said, as we've
seen repeatedly, it's not that a single incident or non-compliance
necessarily leads to distrust. As with all matters of non-compliance, the
transparency and detail provided by the CA and the systemic changes
proposed or introduced are essential in regaining or retaining trust in the
CA.

That is, I think you're right that filing an incident report, with the full
details about the challenges, would represent the bare minimum of response.
I think a step further would be a letter from the auditor, in line with how
delays beyond the 3 month period are handled, is also not unreasonable.

I also agree that periodic explanations are also important and essential,
as it shouldn't be sufficient to file a report saying "nCovid19" and then,
say, not provide any updates until the *next* audit period.

> - Complete the audit for all locations that can be audited
>

Agreed :) I'm glad you avoided suggesting holding back the whole audit ;)

> - Deliver updated report that incorporates the facilities as soon
> as the facility is back available for inspection
>

The reason why I wanted to gather your feedback is that, as I understand
it, this isn't really permitted under the relevant professional standards
to restate the audit by expanding the audit scope after a report has been
delivered.

This is where I think a critical gap in the plan is, and we need to find
some solution here for how to address. Would/should we accept an
(additional) report regarding *just* those locations? Can the auditor
themselves report on the controls with only a partial scope or
understanding? Would they need to re-evaluate all of the materials related
to the controls?

I will say this: As much as possible, a solution needs to avoid an Agreed
Upon Procedures Report. That just seems to try and shift all the liability
to the Browsers for each and every case, and that isn't a reasonable shift
for a free and open program.


> Ryan, related to your thought “*Similarly, if a CA’s preferred auditors

> are from a region affected by travel restrictions, but other accredited

> auditors exist that would be capable, would that be sufficient?”*

>
> - Auditors are not that flexible on reporting formats and doing a
> specific subset of a standards on a specific location.
>
> - What would the root programs accept in terms of such an ad-hoc
> report as it will not be an ISAE3000/CSAE3000/SSAE18 format?
>
> - Depending on the CA it would also be multiple reports that will
> be incomplete: WebTrust, SSLBR, EV, (EV) Code Signing
>
> - Which controls / criteria should be reported on? Only the ones
> related to physical security?
>

I wasn't looking at the subsetting/adhoc nature. I was more looking at the
scenario where (adversarially), a CA uses the fact that their locations are
in a non-travel-restricted area, but intentionally chooses an auditor that
is in a particularly affected area on the basis that they won't be able to
travel to the locations. In those cases, there may be an auditor capable of
assessing the affected locations, and the CA is deliberately choosing not
to contract with them as a way of delaying/deferring the audit.

That may seem less applicable to nCovid19, but I'm broadly trying to find
if there are principles we can/should apply that would be reusable, and/or
areas up-front we should denote are exceptional.

> For the key material security and key management continuity aspect,
> compared to the controls I would think a typical CA would implement the
> WebTrust standard is quite brief and vague:
>
> - Criterion #3.8 focuses on general CA continuity and availability
> of technology and information. For key material it focuses on having
> back-ups.
>
> - There is one specific control (#3.8.6) that talks a bit about
> securing a facility during a disaster
>

Are you looking only at WebTrust for CAs, and not WebTrust for CAs SSL BRs?
Principle 3 is more extensive in the SSL BRs with Net Sec. I'm really not
sure how to interpret your concerns.

I ask, because there's significant details regarding Security Plan and
annual risk assessments in the latter, that may be worth looking at part of
the disclosure requirements. My apetite for private disclosure is
incredibly low, because it lacks transparency, although I can understand
why physical security continuity might be reasonably worth treating as an
exception, and only allowing disclosure to Root Stores to evaluate, on
behalf of their users.

I think there's also an element of detailed control reporting, depending on
when the Web Trust Task Force is able to release this.

[jwar...@gmail.com]

Certainly, situations such as the outbreak of COVID-19 (Coronavirus) provide significant business challenges, not to mention all of the heartache felt by those suffering personally. From a business standpoint, the outbreak of the Coronavirus is a reminder how fragile companies are to events out of our control. It is also appropriate to include the outbreak with natural disasters / acts of God when contemplating the necessary reaction needed. both for the enterprise, its stakeholders, auditors, etc. These types of events are also the vary reason companies adopt Business Continuity Plans / Disaster Recovery Plans. When the rubber hits the road, these plans are put to the test.

Auditors are challenged on how these types of events affect the scope of the engagement, in particular the nature, timing, and extent of testing necessary to provide the assurance needed to express an opinion on the subject matter of the engagement. Depending on the circumstances of the event, auditors could be challenged with how to physically inspect documents or access essential critical security installations when travel restrictions are in place, or even faced with availability of necessary documentation/artifacts relevant to the audit that are damaged or destroyed. These scenarios can present significant challenges for the auditor in trying to cover those necessary elements needed to cover the entire scope of the examination.

Ultimately, when an auditor is not able to obtain assurance on the entire scope of the engagement, and realizing a carved out approach is not permitted in a WebTrust audit, for example, when a certain data center is not able to be visited to observe controls operating and underlying documentation, the auditor will not be able to provide an unmodified/unqualified/clean opinion and the client would not be able to display the WebTrust seal. In these situations, the auditor would include an explanatory paragraph that details what gave rise to the scope limitation and issue one of the following modified opinions:

• Qualified opinion (when conditions are least severe but significant enough to mention), stating an except for paragraph explaining the condition(s) arising from the scope limitation, such as not being able to test the data center.
• Adverse opinion (when conditions are more severe), stating that the conditions are such that due to the severity of the scope limitation, the auditor states controls are not operating effectively and they were not able to satisfy themselves that the necessary controls were able to operate.
• Disclaimer of opinion (when conditions are most severe), stating that the auditor is unable to form any opinion due to the nature of the scope limitation.

If the potential threat of a scope limitation is primarily due do an auditor not being able to travel to perform necessary testing, as with the Coronavirus, there are potential remedies for the auditor to consider, including, but not limited to:

• Using the work of another auditor, whereby the lead auditor verifies the independence, qualifications and technical competency of another firm that can do a portion of the work, and the lead auditor directs the work, plans, supervises and reviews the other auditor’s work, taking ultimate responsibility. In this case, no mention of the other firm is made in the report as the lead auditor is taking responsibility for the other firm’s work.
• Using technology to observe physical controls and underlying documents/artifacts via remote means, such as video. In this case, the auditor must ensure the authenticity, integrity, security and confidentiality of the transmission.

If the auditor is able to design the audit plan in a manner that overcomes the challenges present from what otherwise would be a scope limitation, and can obtain satisfaction through adequate testing procedures, the auditor will be able to express an unqualified/unmodified (clean) opinion resulting in the ability to obtain the WebTrust seal. Otherwise, the auditor will explain what gave rise to the scope limitation and no seal will be able to be issued.
CAs should work with their auditors as early as possible to identify any impact on the scope of their audit and communicate any issues with the browsers. It looks like from this thread any impact on the scope and the timing of the release of the audit should be documented in Bugzilla, which should also include the CAs incident response plan.

So what happens if a modified opinion is provided by an auditor, for example, because a data center in China could not be tested in the normal course of the examination? Then say, six months later, the data center becomes accessible and available for audit. Since the audit for the year was already issued with the qualification, as required, you would have the option of waiting for the next annual audit to include the data center in question and proceed as normal. Once again, a WebTrust audit cannot include a carve out of the data center, nor can a WebTrust audit be performed later on just the data center. Depending on the significance of the operations not able to be included in the scope of the most current audit, and depending on the needs and requirements of the users (browsers), a CA could undergo specified/agreed-up procedures in a separate engagement, or conduct a full scope WebTrust audit when possible. There ae no hard and fast rules for this situation and each should be treated on a case by case basis, with discussions including the CA, the browsers, and the auditor.

I hope this helps. Happy to answer any further questions.

Jeff

[Ryan Sleevi]

Thanks Jeff,

This is incredibly helpful to understand the approach (and limitations)
that are relevant in the context of a WebTrust report. I'm hoping our ETSI
colleagues might provide a similar level of detail, as I suspect this is
hardly "just" a WebTrust problem at this point.

On Fri, Mar 6, 2020 at 11:51 AM jwardcpa--- via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> If the potential threat of a scope limitation is primarily due do an
> auditor not being able to travel to perform necessary testing, as with the
> Coronavirus, there are potential remedies for the auditor to consider,
> including, but not limited to:
>
> • Using the work of another auditor, whereby the lead auditor
> verifies the independence, qualifications and technical competency of
> another firm that can do a portion of the work, and the lead auditor
> directs the work, plans, supervises and reviews the other auditor’s work,
> taking ultimate responsibility. In this case, no mention of the other firm
> is made in the report as the lead auditor is taking responsibility for the
> other firm’s work.
> • Using technology to observe physical controls and underlying
> documents/artifacts via remote means, such as video. In this case, the
> auditor must ensure the authenticity, integrity, security and
> confidentiality of the transmission.
>

This is incredibly helpful to understand the possibilities! Thank you for
including this. While I can understand this might not be a universal
solution, especially as the virus continues to spread, it's incredibly
helpful to know what options might be possible and available.

If the auditor is able to design the audit plan in a manner that overcomes
> the challenges present from what otherwise would be a scope limitation, and
> can obtain satisfaction through adequate testing procedures, the auditor
> will be able to express an unqualified/unmodified (clean) opinion resulting
> in the ability to obtain the WebTrust seal. Otherwise, the auditor will
> explain what gave rise to the scope limitation and no seal will be able to
> be issued.
> CAs should work with their auditors as early as possible to identify any
> impact on the scope of their audit and communicate any issues with the
> browsers. It looks like from this thread any impact on the scope and the
> timing of the release of the audit should be documented in Bugzilla, which
> should also include the CAs incident response plan.
>

As a follow-up: would the use of such methods (a supervised auditor,
technology based controls) be something that would be available as part of
the Detailed Reporting? My understanding is that it would be noted in such
a report, but perhaps I'm assuming too much.

> So what happens if a modified opinion is provided by an auditor, for
> example, because a data center in China could not be tested in the normal
> course of the examination? Then say, six months later, the data center
> becomes accessible and available for audit. Since the audit for the year
> was already issued with the qualification, as required, you would have the
> option of waiting for the next annual audit to include the data center in
> question and proceed as normal. Once again, a WebTrust audit cannot
> include a carve out of the data center, nor can a WebTrust audit be
> performed later on just the data center. Depending on the significance of
> the operations not able to be included in the scope of the most current
> audit, and depending on the needs and requirements of the users (browsers),
> a CA could undergo specified/agreed-up procedures in a separate engagement,
> or conduct a full scope WebTrust audit when possible. There ae no hard and
> fast rules for this situation and each should be treated on a case by case
> basis, with discussions including the CA, the browsers, and the auditor.
>

Thanks again for this. It's incredibly helpful to know the limitations here
as well, such as a limited-physical-scope audit being non-viable.

Are there limitations as to how long an audit in the past can be conducted?
That is, I'm imagining a scenario where a report is delivered, potentially
with the issues you note (qualified, adverse, disclaimer). Assuming there
comes a point in the future where the factors leading to that opinion are
eventually addressed, and access to the location again becomes possible, is
it possible to perform a full audit for the original period of time? That
is, for a CA that might have an audit period of March-to-March (and thus,
likely affected by these considerations), and the auditor is unable to
sufficiently determine to an unqualified opinion during that assessment
period (of March ~ June), is it possible to examine after-the-fact (e.g. if
the virus is contained by July, in July) for the original March-to-March
period? What challenges may exist for such audits that are further than 3
months from the audit period?

Realizing there's no hard and fast rules, I'm aware it's perhaps obvious
the line of thinking I'm going towards, which is to require provisions or
contracts for a full audit once/if the virus is contained and travel
restrictions are lifted, as part of accepting such a qualified or adverse
report. I'm also thinking of situations beyond the immediate incident, such
as natural disasters that may impact a region, and trying to understand if
there is an upper-bound or limit as to how far retroactively things may go.

In the situation of the Mar-Mar period, I can also understand that some CAs
and their auditors may be waiting, during the BR alloted 90 days, to see if
it may be possible to inspect the location during that period (such as
waiting to see if by May they can visit). I would still encourage full
transparency now, in such cases, as they still signal an awareness of
potential delays, and we can work with the CA to understand the challenges
and limitations, and work towards reasonable solutions for the worst case.

[jwar...@gmail.com]

Great follow on questions Ryan. As far as the detailed report, whether the end product is in the current form, or in the detailed version, the lead auditor is taking full responsibility and does not make mention of the other auditor in both the opinion, and the detailed section of the controls tested for a detailed report. That being said, nothing prohibits a CA from creating a Bug to draw attention to the fact and explain the auditors obtained the assistance of another firm to complete the scope of the testing. If WebTrust did allow a carve out approach, there would be more flexibility to allow the reference to another firm, but since it is inclusive and the lead auditor takes full responsibility, that is not an option.

As far as your question on the situation where the issue giving rise to a qualification of the report, such as the data center that could not be visited during the audit period subsequently becomes available, the auditor could consider if it is appropriate and not misleading to perform additional testing in the areas that were not able to be tested originally, and if so, what reporting is permissible. I could see a situation where the 90 day deadline was near and the report was issued with a scope limitation, and then after some time passes, testing could be performed to complete the scope. The report would then be unmodified/unqualified/clean assuming the results of the testing are favorable. It could be possible for a firm to re-issue it's report and "dual-date" the opinion, which basically means the date at the bottom of the report would indicate two dates, one for the original issuance, and the second date to extend to the completion of the testing for those areas to complete the scope. For example, in a 12/31 period end, the report would be signed by the audit firm, and dated in a manner to say something like
XYZ CPAs
March 27, XX, except for the subsequent event as detailed in the report, which is dates July 20t.

This example demonstrates the firm was able to complete the scope of the audit testing on July 20th. It is up to the auditor's judgment as to how far the opinion can be dual dated/extended. Once too much time passes, this option is no longer viable.

I hope this is helpful.

Thanks,

Jeff

[Ryan Sleevi]

On Fri, Mar 6, 2020 at 9:03 PM jwardcpa--- via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Great follow on questions Ryan. As far as the detailed report, whether
> the end product is in the current form, or in the detailed version, the
> lead auditor is taking full responsibility and does not make mention of the
> other auditor in both the opinion, and the detailed section of the controls
> tested for a detailed report. That being said, nothing prohibits a CA from
> creating a Bug to draw attention to the fact and explain the auditors
> obtained the assistance of another firm to complete the scope of the
> testing. If WebTrust did allow a carve out approach, there would be more
> flexibility to allow the reference to another firm, but since it is
> inclusive and the lead auditor takes full responsibility, that is not an
> option.
>

Good to know! I don't think this poses any intrinsic problems, since as you
note, the lead auditor is taking full responsibility, but it's helpful to
know what disclosures, if any, would arise in such a situation.

> This example demonstrates the firm was able to complete the scope of the
> audit testing on July 20th. It is up to the auditor's judgment as to how
> far the opinion can be dual dated/extended. Once too much time passes,
> this option is no longer viable.
>

Right, you addressed the scenario of a single report, subsequently updated.
I was actually contemplating two full reports with two full engagements.
That is, the first engagement and report may be qualified, due to the lack
of the datacenter. My question was whether it's possible to engage the
auditor in a second full engagement, this time considering all the
facilities, for the original time period in question.

Think of this as a variation for what we see some CAs do, which is scope
their annual reports into two or more reports, one of which may be
qualified. That is, they may have a Jan - July report which is qualified,
and a July - Dec report which is unqualified. However, those are
non-overlapping date periods. I was wondering if, again, using our March to
March scenario, that it's conceivable a report is delivered in April that
is qualified, access to the facility is restored in July, and the auditor
(either the original firm or a new firm) conducts a full audit of the
original March-to-March period. In effect, conducting a second audit.

I'm trying to tease out if there are limitations on the original firm
performing that work (e.g. because they'd previously been engaged in an
audit of that period), as well as whether there are limitations as to how
far back one can go. For example, could a CA engage an auditor, today, for
a Jan 1, 2018 to Jan 1, 2019 period? What if the engagement was for a
October 1, 2018 to October 1, 2019 period (e.g. 6 months ago)? I can
understand the difficulty of obtaining an audit today for, say, the period
2014-01-01 to 2015-01-01, but I'm wondering what options might exist for
examination of those remaining facilities after-the-fact.

My worst case scenario is that it is determined to not be possible after
some period of time (e.g. 6 months) to obtain such originally-expected
assurances. In those cases, I think the honest and pragmatic answer may
involve discussions of removal of trust in that root, and so I want to make
sure to explore alternatives and options before having to start such
discussions.

[Jeff Ward]

I hate giving an "it depends" answer, but that's where I'll start. In the case where a report is issued with a qualified opinion due to the inability to visit a CA's data center, as an example, and issued primarily to comply with the 90 day rule, and the data center subsequently becomes available, it is possible for the auditor to perform necessary procedures and collect relevant documentation / artifacts subsequent to the original audit period to allow them to issue an unqualified opinion on the original audit period that was previously qualified. Kind of a mouthful. It depends if this will work on the documentation and artifacts that can be obtained during that original period. Collecting evidence on those controls outside of the period is problematic as it is not part of the audit period. That is not to say that the documentation could be obtained from the original audit period and providing the necessary comfort the auditor needed in their previously issued qualified report. In short, the auditor is looking for evidence after the period that the controls in fact were operating effectively during the period. Some documentation lends itself easily to make this determination, but I can see challenges. Reviewing logs and video surveillance are two means that come to mind that would be part of the auditor's assessment to see if that evidence can satisfy the audit objectives. These monitoring activities typically are time stamped and demonstrate when the controls were operating. Controls that can only be physically observed by the auditor, and a lack of any compensating control or controls to evaluate, will be an issue. In the end, if there is no way to determine whether or not a control or controls operated effectively then the auditor would not be able to issue an unqualified opinion on the previous audit period as the evidence collected is part of the next period. So sorry to conclude, it depends on the facts and circumstances.

[clemen...@tuv-austria.com]

Situation from ACAB'c ETSI auditors point of view:

On one hand it is quite simple: if the auditor cannot perform the audit as foreseen in the certification program no certificate can be issued. In case a surveillance audit cannot be performed, the certification body must withdraw the affected certificate.

On the other hand the situation is complex because the reason for not performing the audit is neither the fault of the CA neither the fault of the auditor but due to a force majeure. In this case, the questions to ask comprise:
1. What is the reason for having the audit?
2. What is the reason to perform the audit now?
3. Can the audit be postponed?
4. Would it help to perform the audit in a reduced scope?

In the present case the reason for having the audit and having it at a certain time comes from rules defined by the consumer of the audit attestations/certificates, i. e. the browsers. In this case browsers should think about answers to questions 3 and 4 and considering the possibility of adapting the rules for the case of force majeure. This happens in public life, too. According to the law, kids must go to school (at least in Germany). But today, some (but not all) schools are closed because of the virus. So the rules were changed.

Hence, it is hard to define a universal rule valid for all CAs (schools). The following focuses on the possibilities in case of travel restrictions. Details might be different for other cases like natural disasters.

One possible scenario could be that the Auditors of the accredited CAB are under restriction:
Such a case would mainly be addressed by business continuity provisions of the conformity assessment body (CAB), considering e.g. the following questions:
- What is the duration until return to normal operation? Can the audit still be performed in time after the restrictions have been lifted?
- How is the auditing personnel distributed? With personnel in different locations, maybe not all auditors are affected at the same time.
- Can the CAB subcontract non-affected external auditors?
- If the CAB does not have sufficient options, can another ACAB’c member CAB jump in?
In any case will the accredited CAB take care of the requirement conformant treatment of the travel restricted situation based upon its quality controlled ruleset following ISO/IEC 17065 in conjunction with ETSI EN 319403.

The other possible scenario would be that the TSP or a TSP location is under restriction:
A general rule is, that the scope of the assessment (including “in terms of facilities”) must be clearly defined (ETSI EN 319 403, 7.4.1.0) and that the complete scope must be audited to perform an ETSI certification.
Another general rule is that, depending on some pre-conditions, a sample-based approach could be possible for TSPs using multiple sites. (ETSI EN 319 403, 7.4.3) In some cases, it might be possible to choose a sample that does not include locations under restriction. However, as the CABF requires full audits, this does not apply in this context.
If facilities can’t be audited by auditors of the CAB in person, possible alternatives are more or less identical to those stated for Webtrust
- “Network-assisted auditing techniques” are possible (ETSI EN 319 403, 7.4.1.2)
- CAB may subcontract auditors that do not fall under the restriction, if they fulfil the auditor requirements. The CAB always remains fully responsible for such outsourced activities. (ISO/IEC 17065, 6.2.2 ).
If such alternatives were accepted by the CAB to provide reasonable assurance with regard to the requirements to be audited, this would result in a normal audit conclusion and would not be visible on an audit attestation letter.

If none of the two alternatives is possible, a general rule is, that everything which can be audited will be audited - there is especially no restriction to do a full Stage 1 document audit.

If facilities cannot be audited by auditors of the CAB in person and the alternatives stated above are not possible or the CAB does not regard them to provide reasonable assurance with regard to the requirements to be audited in order to draw a reasonable audit conclusion this shall be documented in the audit report.
An audit attestation letter shall be issued stating which parts have not been covered by the audit.

How to deal with the situation, after travel restrictions have been lifted?

>From the viewpoint of an ETSI certification, no ETSI certificate has been issued:
- It is possible to re-use the original audit results and perform an additional audit just with regard to the non audited requirements (ISO/IEC 17065, 7.4). Usually, the period during which this is possible is limited by the CAB (ACAB’c: 6 month). Once the original audit becomes too old, a completely new audit would be necessary.

>From the viewpoint of an Audit Attestation Letter (AAL) under ETSI, either an updated version of the original audit attestation letter or a completely new audit attestation letter shall be issued based on the additional audit activities described before. We are afraid in all cases this might result in deviations from the Mozilla requirements on audit letters:
- An updated version / new letter for the original period would most likely be dated more than 3 month after the end of the audit period.
- An updated version / new letter with an extend period would most likely result in a period of more than 12 month.
Please note: An extend period would only be possible, if a new full audit has been performed (covering the time of the original period plus the extension till the last day of the audit), because the original audit of course can’t cover a time period after that audit.

As far as we know, there is no general restriction on how far back in time the audited period may start. But it should be obvious that for some kind of evidences the information value might decrease over time or that it might become more and more complicated to determine a certain state of a certain thing at the certain time.

Matthias & Clemens

[Kathleen Wilson]

All,

I will greatly appreciate your input on the following new "Audit Delay"
section.

https://wiki.mozilla.org/CA/Audit_Statements#Audit_Delay

Thanks,
Kathleen

PS: I moved the content of
https://wiki.mozilla.org/CA/Audit_Letter_Validation
to
https://wiki.mozilla.org/CA/Audit_Statements
(with a redirect from the old page)

[Ryan Sleevi]

Suggestions:
1) Rename "Audit Delay" to [audit-delay] and rename "Audit Delay COVID-19"
to [audit-delay] [covid-19] or [audit-delay-covid-19], depending
Rationale: In general, our filters work on word searches, so the brackets
brackets help distinguish the two. To search for "Audit Delay" without
considering COVID-19, the filter would have to be ("Audit" AND "Delay") AND
NOT "COVID-19". The renames help us search for "[audit-delay]" (which would
exclude Covid-19) or "[audit-delay]" AND NOT "[covid-19]", which is...
slightly easier :) This is mostly minor, but also tracks how we handled
[ca-compliance], [auditor-compliance], [delayed-revocation-leaf] and
[delayed-revocation-ca] :)

2) Rename "Potential remediations" to "Minimum expectations"
Rationale: I worry that "potential remediations" may be seen as an
indefinite escape clause. As noted in the discussion of force majeure that
you've linked, in general, these clauses generally temporarily suspend
certain obligations, but may not indefinitely apply. While this situation
continues to evolve, and we will hopefully see a timely global recovery,
there exists the non-negligible possibility that it may become necessary at
some point in the future to limit or restrict trust in CAs in order to
minimize risk to users. These are absolutely case-by-case scenarios, and so
this isn't meant to scare CAs or Auditors into engaging in unsafe or risky
procedures, but more to highlight that as part of recovery from such
scenarios, it may be necessary to figure out how to transition from
uncertainy to certainty, such as rolling keys over to new
roots/intermediates. Keeping people physically safe is certainly the
pressing priority here, and we should be sensitive to that, but I worry
that "potential remediations" suggests that such measures might be
indefinitely accepted.

3) Clarify ETSI documentation and disclosure requirements
Rationale: My concern with the ETSI approach is that Mozilla may not
receive the same information the auditor/CAB provides to the CA/TSP. For
example, as currently worded, it'd be impossible to discover the
limitations that the auditor might have encountered (such as a
documentation-only assessment), because that'd be normally addressed in the
engagement letter between the CAB/TSP, and beyond them, typically only the
Supervisory Body would be party to such details. While your requirements
for disclosure are unambiguous, my worry is how many TSPs/CABs using an
ETSI scheme have failed to uphold Mozilla's expectations / CCADB
expectations, and thus it wouldn't be clear when a TSP was violating policy
(e.g. by not having disclosed the situation).

Potentially: "Audit letters MUST disclose each location that was included
in the scope of the audit, as well as whether the inspection was physically
carried out in person"

There's probably a MUCH better way to word this, but the concept I'm trying
to capture is some positive affirmation by the auditor about what they did.
If a Letter doesn't include that, it's a red flag (to reject the audit). If
it does, that at least provides clarity and fits back in to the incident
report discussion.

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

[Kathleen Wilson]

On 3/18/20 5:16 PM, Ryan Sleevi wrote:
> Suggestions:
> 1) Rename "Audit Delay" to [audit-delay] and rename "Audit Delay COVID-19"
> to [audit-delay] [covid-19] or [audit-delay-covid-19], depending
> Rationale: In general, our filters work on word searches, so the brackets
> brackets help distinguish the two. To search for "Audit Delay" without
> considering COVID-19, the filter would have to be ("Audit" AND "Delay") AND
> NOT "COVID-19". The renames help us search for "[audit-delay]" (which would
> exclude Covid-19) or "[audit-delay]" AND NOT "[covid-19]", which is...
> slightly easier :) This is mostly minor, but also tracks how we handled
> [ca-compliance], [auditor-compliance], [delayed-revocation-leaf] and
> [delayed-revocation-ca] :)

Done

>
> 2) Rename "Potential remediations" to "Minimum expectations"
> Rationale: I worry that "potential remediations" may be seen as an
> indefinite escape clause. As noted in the discussion of force majeure that
> you've linked, in general, these clauses generally temporarily suspend
> certain obligations, but may not indefinitely apply. While this situation
> continues to evolve, and we will hopefully see a timely global recovery,
> there exists the non-negligible possibility that it may become necessary at
> some point in the future to limit or restrict trust in CAs in order to
> minimize risk to users. These are absolutely case-by-case scenarios, and so
> this isn't meant to scare CAs or Auditors into engaging in unsafe or risky
> procedures, but more to highlight that as part of recovery from such
> scenarios, it may be necessary to figure out how to transition from
> uncertainy to certainty, such as rolling keys over to new
> roots/intermediates. Keeping people physically safe is certainly the
> pressing priority here, and we should be sensitive to that, but I worry
> that "potential remediations" suggests that such measures might be
> indefinitely accepted.

Done

>
> 3) Clarify ETSI documentation and disclosure requirements
> Rationale: My concern with the ETSI approach is that Mozilla may not
> receive the same information the auditor/CAB provides to the CA/TSP. For
> example, as currently worded, it'd be impossible to discover the
> limitations that the auditor might have encountered (such as a
> documentation-only assessment), because that'd be normally addressed in the
> engagement letter between the CAB/TSP, and beyond them, typically only the
> Supervisory Body would be party to such details. While your requirements
> for disclosure are unambiguous, my worry is how many TSPs/CABs using an
> ETSI scheme have failed to uphold Mozilla's expectations / CCADB
> expectations, and thus it wouldn't be clear when a TSP was violating policy
> (e.g. by not having disclosed the situation).
>
> Potentially: "Audit letters MUST disclose each location that was included
> in the scope of the audit, as well as whether the inspection was physically
> carried out in person"
>
> There's probably a MUCH better way to word this, but the concept I'm trying
> to capture is some positive affirmation by the auditor about what they did.
> If a Letter doesn't include that, it's a red flag (to reject the audit). If
> it does, that at least provides clarity and fits back in to the incident
> report discussion.
>

Added the following to the top of the "Minimum Expectations" list:
* Both ETSI and WebTrust Audits must:
** Disclose each location that was included in the scope of the audit,
as well as whether the inspection was physically carried out in person.

This way we can easily add more sub-bullet points as needed.


I also added a summary to the top of the page
https://wiki.mozilla.org/CA/Audit_Statements
CA Audits are one of the primary mechanisms relied upon by Mozilla to
ensure that a CA is operating securely and in compliance with our
policies. CA audits and audit statements must comply with the following
requirements.
* Section 3.1 of Mozilla's Root Store Policy.
** An Audit Delay is when one or more of the following requirements in
section 3.1.3 cannot be met:
***"Full-surveillance period-of-time audits MUST be conducted and
updated audit information provided no less frequently than annually."
***"... MUST be provided to Mozilla via the CCADB within three months of
the point-in-time date or the end date of the period."
* Section 5.1 of the Common CCADB Policy.
* Section 8 of the CA/Browser Forum Baseline Requirements, if the root
certificate has the Websites (TLS/SSL) trust bit enabled.


Thanks,
Kathleen

[Kathleen Wilson]

All,

I will greatly appreciate your ideas about the following.

In the Minimum Expectations section in
https://wiki.mozilla.org/CA/Audit_Statements#Audit_Delay
I added:

""
* Both ETSI and WebTrust Audits must:
** Disclose each location that was included in the scope of the audit,
as well as whether the inspection was physically carried out in person.
""

My question: What should "location" mean in the above requirement?

The problem is that we require public-facing audit statements, so I do
not want sensitive or confidential information in the audit statements,
such as the exact physical addresses of CA Operations and root cert
private key storage.

What information could be added to audit statements to give us a clear
sense about which CA facilities were and were not audited?

For example, if a CA happens to have two facilities in the same city
that should be audited, how can the audit statement clearly indicate if
all of that CA's facilities were audited without providing the exact
physical addresses?

Thanks,
Kathleen

[Jeremy Rowley]

What about issues other than audits? For example, with certain locations closing, key ceremonies may become impossible, leading to downed CRLs/OCSP for intermediates. There's also a potential issue with trusted roles even being able to access the data center if something goes down and Sub CAs can't be revoked. Should that be mentioned, requiring CAs to file an incident report as soon as the event becomes likely?

For the location issue, I think including the locations audited and the locations not audited (to the full criteria) as an emphasis of matter would be helpful. So maybe an emphasis like we audited the offices in x, y, and z. Office z was inaccessible to evaluate criteria 1-n. It give you the list of locations and where there were issues in getting access due t o he emergency. Same city is harder. For example, we have two locations in Utah. You could say Utah office 1 and Utah office 2 to obfuscate the information a little.

Jeremy

-----Original Message-----
From: dev-security-policy <dev-security-...@lists.mozilla.org> On Behalf Of Kathleen Wilson via dev-security-policy
Sent: Friday, March 20, 2020 2:07 PM
To: Mozilla <mozilla-dev-s...@lists.mozilla.org>
Subject: Re: Auditing of CA facilities in lockdown because of an environmental disaster/pandemic

All,

I will greatly appreciate your ideas about the following.

In the Minimum Expectations section in
https://wiki.mozilla.org/CA/Audit_Statements#Audit_Delay
I added:
""

* Both ETSI and WebTrust Audits must:
** Disclose each location that was included in the scope of the audit, as well as whether the inspection was physically carried out in person.
""

My question: What should "location" mean in the above requirement?

The problem is that we require public-facing audit statements, so I do not want sensitive or confidential information in the audit statements, such as the exact physical addresses of CA Operations and root cert private key storage.

What information could be added to audit statements to give us a clear sense about which CA facilities were and were not audited?

For example, if a CA happens to have two facilities in the same city that should be audited, how can the audit statement clearly indicate if all of that CA's facilities were audited without providing the exact physical addresses?

Thanks,
Kathleen

[Ryan Sleevi]

On Fri, Mar 20, 2020 at 4:07 PM Kathleen Wilson via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> My question: What should "location" mean in the above requirement?
>

The WebTrust Practitioner Guidance offers a reasonable definition:
https://www.cpacanada.ca/en/business-and-accounting-resources/audit-and-assurance/overview-of-webtrust-services/practitioner-qualification-and-guidance

CA Processing Locations
All reports issued should list the city, state/province (if applicable),
and country of all physical locations
used in CA operations. This includes data center locations (primary and
alternate sites), registration
authority locations (for registration authority operations performed by the
CA), and all other locations
where general IT and business process controls that are relevant to CA
operations are performed.

> For example, if a CA happens to have two facilities in the same city
> that should be audited, how can the audit statement clearly indicate if
> all of that CA's facilities were audited without providing the exact
> physical addresses?

We're primarily interested in making sure that the auditor examined /both/
facilities for the appropriateness of controls. ETSI's lack of rigorous
methodology leaves a lot to be desired here, but it's not difficult to
disambiguate by indicating something like
"Facility 1 in City, State, Country" vs "Facility 2 in City, State, Country"
or
"Primary Facility in City, State, Country" vs "Disaster Recovery Facility
in City, State, Country"

(adjusted as appropriate)

[Ryan Sleevi]

On Fri, Mar 20, 2020 at 4:15 PM Jeremy Rowley via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> What about issues other than audits? For example, with certain locations
> closing, key ceremonies may become impossible, leading to downed CRLs/OCSP
> for intermediates. There's also a potential issue with trusted roles even
> being able to access the data center if something goes down and Sub CAs
> can't be revoked. Should that be mentioned, requiring CAs to file an
> incident report as soon as the event becomes likely?
>

Yes. I think those are, quite honestly, much more concerning, because
that's not about a CA's relationship with an external party, but about a
CA's own preparedness for disaster. In any event, as with /any/ incident,
the sooner it's filed, and the more information and context is provided,
the more effective a response can be.

>
> For the location issue, I think including the locations audited and the
> locations not audited (to the full criteria) as an emphasis of matter would
> be helpful. So maybe an emphasis like we audited the offices in x, y, and
> z. Office z was inaccessible to evaluate criteria 1-n. It give you the list
> of locations and where there were issues in getting access due t o he
> emergency.

Yup. That is the model WebTrust is using, and that reasonably meets the
objective here of informing relying parties when the auditor faced
limitations that should be considered when evaluating their report.

[Kathleen Wilson]

On 3/20/20 1:15 PM, Jeremy Rowley wrote:
> What about issues other than audits? For example, with certain locations closing, key ceremonies may become impossible, leading to downed CRLs/OCSP for intermediates. There's also a potential issue with trusted roles even being able to access the data center if something goes down and Sub CAs can't be revoked. Should that be mentioned, requiring CAs to file an incident report as soon as the event becomes likely?
>

Good point.

I added the following to https://wiki.mozilla.org/CA/Incident_Dashboard
** If the issue is due to mandated restrictions regarding COVID-19, use
Whiteboard = [ca-compliance][covid-19]


I updated https://wiki.mozilla.org/CA/Audit_Statements#Audit_Delay
to:
* Whiteboard = [ca-compliance][audit-delay]
* For audit delays due to mandated restrictions regarding COVID-19, use
Whiteboard = [ca-compliance][audit-delay][covid-19]

Do you think we should also add a section to
https://wiki.mozilla.org/CA/Responding_To_An_Incident about COVID-19?


Thanks,
Kathleen

[Jeff Ward]

Shortly before the COVID-19 pandemic, members of the WebTrust Task Force reviewed this guidance and had discussion focused on whether our reports were providing too much information in a publicly available report as to the operations of a CA. Practitioners have been getting questioned in the past by CAs as to why such specific information should be disclosed to the level of city and state for the location of its operations. It is a good point as certainly not all CAs provide this information freely to all of their employees, let alone outsiders. This is especially true with the larger and more complex CAs. For the more complex CAs, I can envision another Attachment in the audit report, similar to the thumbprint attachment, that lists the locations in a manner that Jeremy suggests that protects the physical location to some degree, yet provides the users of the report enough information to know what was able to be covered. That could be part of our guidance, which of course is just that - guidance. Having our guidance adjusted in this manner would certainly help drive consistency that would be helpful to the CABF. I am sure there will be variations in reports, however, as guidance is non-authoritative for AICAP and CPA Canada.

As far as the term "CA facility", I'd like to get thoughts from this group as to what that includes. For instance, while a facility hosting an active HSM with CA private keys is a certainly a "CA facility", would you also include in this definition things like a bank safe deposit box that stores a deactivated and encrypted copy of a private key a CA facility? Would you expect this level of information disclosed in an audit report?

[Jeremy Rowley]

Although I’m sure every CA has business continuity plans, I think that extended blocked access to every data center they have may not be part of that plan. I’m not sure, but I think if the required shelter’s are in place for long periods you may start to see problems. Early disclosure sounds like the best policy, but I thought the early disclosure requirement may be worth calling out in the Mozilla policy. Then again, that really should be standard procedure at that point.

From: Ryan Sleevi <ry...@sleevi.com>
Sent: Friday, March 20, 2020 2:57 PM
To: Jeremy Rowley <jeremy...@digicert.com>
Cc: Kathleen Wilson <kwi...@mozilla.com>; Mozilla <mozilla-dev-s...@lists.mozilla.org>
Subject: Re: Auditing of CA facilities in lockdown because of an environmental disaster/pandemic

On Fri, Mar 20, 2020 at 4:15 PM Jeremy Rowley via dev-security-policy <dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>> wrote:
What about issues other than audits? For example, with certain locations closing, key ceremonies may become impossible, leading to downed CRLs/OCSP for intermediates. There's also a potential issue with trusted roles even being able to access the data center if something goes down and Sub CAs can't be revoked. Should that be mentioned, requiring CAs to file an incident report as soon as the event becomes likely?

[Kathleen Wilson]

It's still very much a work-in-progress, but I updated the first bullet
point in the "Minimum Expectations" section again.

https://wiki.mozilla.org/CA/Audit_Statements#Audit_Delay

""
Both ETSI and WebTrust Audits should:
- Disclose each location (at the state/province level) that was included
in the scope of the audit or should have been included in the scope of
the audit, whether the inspection was physically carried out in person
at each location, and which audit criteria were checked (or not checked)
at each location.
-- If the CA has more than one location in the same state/province, then
use terminology to clarify the number of facilities in that
state/province and whether or not all of them were audited. For example:
"Facility 1 in Province", "Facility 2 in Province, Facility 3 in
Province" or "Primary Facility in Province", "Secondary Facility in
Province", "Tertiary Facility in Province".
""

TO DO: Clarify the types of CA locations that should be disclosed in the
audit statement. e.g. data center locations, registration authority
locations, where IT and business process controls of CA operations are
performed, facility hosting an active HSM with CA private keys, facility
or bank deposit box storing a deactivated and encrypted copy of a
private key, other?

I will continue to appreciate your feedback on this, and the entire
"Audit Delay" section.


I also filed an issue in GitHub regarding adding this to Mozilla's root
store policy.
https://github.com/mozilla/pkipolicy/issues/207

Thanks,
Kathleen

[Kathleen Wilson]

All,

Just FYI that I updated the CA Incident Dashboard wiki page to separate
the audit delay bugs into their own section.

https://wiki.mozilla.org/CA/Incident_Dashboard#Audit_Delays

Thanks,
Kathleen