Re: ETSI audits not listing audit periods

[Ryan Sleevi]

On Mon, Oct 30, 2017 at 3:50 PM, Kathleen Wilson via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:
>
> How do we get all auditors to start meeting our audit statement
> requirements?
>
> Why haven't all included CAs communicated these requirements to their
> auditors?
>
> Why am I seeing so many audit statements (particularly ETSI audit
> statements) that do not meet our requirements?
>
> I will greatly appreciate thoughtful and constructive ideas on this.
>
> Thanks,
> Kathleen
>

Kathleen,

Thanks for raising this matter! I think it's an important highlighting of
the very different approaches to auditing employed by WebTrust and ETSI,
and the underlying reliability and assurance of those audits.

Below is my attempt to summarize my understanding so far:
- ETSI EN 319 411-1 specifies generally applicable policy and security
requirements for trust service providers (TSPs) - including website
certificates.
- The purpose of EN 319 411-1 is to provide a framework for assessment of a
TSP, but does not specify how that assessment is to be caried out (c.f.
Section 1 of EN 319 411-1)
- EN 319 411-1 mentions EN 319 403 for guidance on such assessments
- EN 319 403 provides the framework for conformity assessment bodies (CABs)
to evaluate TSPs. It's based on/extends ISO/IEC 17065 specific to TSPs.
- ISO/IEC 17065 is incorporated due to Regulation (EC) No 765/2008 to
ensure consistency in CABs in evaluating TSPs

As noted within 319 403 (Introduction), several other documents are
incorporated as well - ISO/IEC 17021 (common requirements for conformity
assessment bodies evaluating management systems) and ISO/IEC 27006 (common
requirements for CABs evaluating information security management systems),
for example.

If we put the layer cake together and simplify:
* ISO/IEC 17065 - Common requirements for conformity assessment bodies
looking at products/services (e.g. "What should all auditors do")
* ISO/IEC 17021 - Common requirements for conformity assessment bodies
looking at management systems
* ISO/IEC 27006 - Common requirements for conformity assessment bodies
looking at information security management systems
* EN 319 403 - Common requirements for conformity assessment bodies
evaluating TSPs (e.g. "What makes an auditor qualified to be a CA auditor")
* EN 319 411-1 - Common requirements on the TSP for websites (combined with
EN 319 401, which 319 411-1 incorporates-and-builds-on)

In trying to understand why the reports are what they are, we need to look
in particular at 17021 and 17065, and the framework they use for both audit
engagements and reporting. 17065 describes a certification scheme.
Reproducing a paragraph from the introduction:

"""
Certification of products, processes or services is a means of providing
assurance that they comply with
specified requirements in standards and other normative documents. Some
product, process or service
certification schemes may include initial testing or inspection and
assessment of its suppliers' quality
management systems, followed by surveillance that takes into account the
quality management system and
the testing or inspection of samples from the production and the open
market. Other schemes rely on initial
testing and surveillance testing, while still others comprise type testing
only.
"""

319 411-1 certification describes a system that is based on an initial
testing or inspection, along with periodic surveillance. Importantly,
within 17065 and 17021, the way of ensuring continued compliance is done
based on contracts and reporting - that is, the client is responsible for
reporting changes to their conformity assessment body, and the CAB may
determine to revoke the certification or indicate corrective actions should
be taken (see ISO/IEC 17065:2012(E) 4.1.2.2, ISO/IEC 17021:2006(E) 8.6.3).
ISO/IEC 17065:2012(E), Section 7.7 describes the general reporting: the
name of the CAB, the date the certification is granted, the name and
address of the client, the scope of the certification, the validity
period/expiration date of the certificate, and anything else required by
the certification scheme (319 411-1).

Within the WebTrust scheme, the reports are based on either US (AICPA)
Standards of AT101, Canadian (CPA Canada) Standards of Section 5025, or
IFAC's ISAE3000 standards. What is important and notable about these is
that they are reports over information, with a scope and norm (e.g.
WebTrust for CAs) applied. This is why there's a consistent period of time
- information is collected and the auditor evaluates, on the basis of that
information, the level of assurance that is being met.

I'm still working to have a better understanding here (and this is all in a
personal capacity), but my conclusion is this:
- "ETSI" audits reflect an engagement at a particular point in time, where
a series of system controls are evaluated, and the result is a certificate
indicating the process and products comply with the relevant criteria (319
411-1). This is similar to a "Point in Time Readiness Assessment". After
the certification is granted, the TSP is responsible for notifying the CAB
of changes (in the future), and the CAB evaluates whether those changes
should cause it to revoke certification or institute some corrective action
(which may not be noted in the certification). An ETSI audit is thus meant
to indicate a CA "is" or "will be" conforming.
- "WebTrust" audits reflect an engagement looking over the past, by the CA
providing documentation and evidence that their policies were consistent
with the requirements. That is, a WebTrust audit is meant to assess whether
or not a CA is being accurate when it says it "was" conforming, but makes
no guarantees about whether or not a CA "is" conforming, nor whether or not
the CA disclosed all relevant information.


The challenge is in determining whether this is a correct understanding of
the respective differences, and then determining whether or not the
certification-based approach provides the desired or sufficient level of
assurance - in particular, its 'forward-looking' approach with
self-reporting, rather than the WebTrust-style assurance engagement that is
more 'backward-looking' without future promises.

[Kathleen Wilson]

Thanks for the explanation. It's very helpful.

Up to this point, it makes sense to me.

> Importantly,
> within 17065 and 17021, the way of ensuring continued compliance is done
> based on contracts and reporting - that is, the client is responsible for
> reporting changes to their conformity assessment body, and the CAB may
> determine to revoke the certification or indicate corrective actions should
> be taken (see ISO/IEC 17065:2012(E) 4.1.2.2, ISO/IEC 17021:2006(E) 8.6.3).
> ISO/IEC 17065:2012(E), Section 7.7 describes the general reporting: the
> name of the CAB, the date the certification is granted, the name and
> address of the client, the scope of the certification, the validity
> period/expiration date of the certificate, and anything else required by
> the certification scheme (319 411-1).

If I am understanding correctly... an ETSI auditor performs a point-in-time audit and then relies on the CA to notify the auditor of material changes.

Wish I could say that I believe that would work. Unfortunately, based on my experience with CAs I do not believe the CAs will be pro-active in this way. And I think the only way to really know what a CA is doing is to look at their data -- look at the actual certs they are issuing, and their documentation regarding such certs/issuance.

>
> Within the WebTrust scheme, the reports are based on either US (AICPA)
> Standards of AT101, Canadian (CPA Canada) Standards of Section 5025, or
> IFAC's ISAE3000 standards. What is important and notable about these is
> that they are reports over information, with a scope and norm (e.g.
> WebTrust for CAs) applied. This is why there's a consistent period of time
> - information is collected and the auditor evaluates, on the basis of that
> information, the level of assurance that is being met.

That makes sense to me.

>
> I'm still working to have a better understanding here (and this is all in a
> personal capacity), but my conclusion is this:
> - "ETSI" audits reflect an engagement at a particular point in time, where
> a series of system controls are evaluated, and the result is a certificate
> indicating the process and products comply with the relevant criteria (319
> 411-1). This is similar to a "Point in Time Readiness Assessment". After
> the certification is granted, the TSP is responsible for notifying the CAB
> of changes (in the future), and the CAB evaluates whether those changes
> should cause it to revoke certification or institute some corrective action
> (which may not be noted in the certification). An ETSI audit is thus meant
> to indicate a CA "is" or "will be" conforming.
> - "WebTrust" audits reflect an engagement looking over the past, by the CA
> providing documentation and evidence that their policies were consistent
> with the requirements. That is, a WebTrust audit is meant to assess whether
> or not a CA is being accurate when it says it "was" conforming, but makes
> no guarantees about whether or not a CA "is" conforming, nor whether or not
> the CA disclosed all relevant information.

This is a very important distinction. Thank you for clarifying.

> The challenge is in determining whether this is a correct understanding of
> the respective differences,

How do we verify that this is correct?

> and then determining whether or not the
> certification-based approach provides the desired or sufficient level of
> assurance - in particular, its 'forward-looking' approach with
> self-reporting, rather than the WebTrust-style assurance engagement that is
> more 'backward-looking' without future promises.

Based on what I've seen with CAs over the past several years, I do not believe the 'forward-looking' approach with self-reporting is sufficient.

I think we have to (and we do!) require the backward-looking approach.

CAs can make all the "future promises" they want, but the proof is in the resulting data -- the certs that they issue.

Based on the above information, I do not think the ETSI audits meet the requirements of Mozilla's Root Store Policy or the CA/Browser Forum Baseline Requirements. Maybe that's the real problem here.

Am I missing something?

Kathleen

[Kathleen Wilson]

To give us a concrete example, here's a Bugzilla Bug that I filed this morning:

https://bugzilla.mozilla.org/show_bug.cgi?id=1412950

The CA's 2015-2016 audit was WebTrust.

Their current audit statement is ETSI.

When I filed the bug I thought there was a gap in auditing from March 10 2016 to January 29 2017.

However, based on Ryan's explanation above, my understanding now is that the ETSI audit is a point-in-time audit, so the CA's activities from March 10 2016 until now have not been audited, with the exception of one month (January 30 to March 1 2017).

Correct?

Kathleen

[Ryan Sleevi]

On Mon, Oct 30, 2017 at 5:39 PM, Kathleen Wilson via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> > Importantly,
> > within 17065 and 17021, the way of ensuring continued compliance is done
> > based on contracts and reporting - that is, the client is responsible for
> > reporting changes to their conformity assessment body, and the CAB may
> > determine to revoke the certification or indicate corrective actions
> should
> > be taken (see ISO/IEC 17065:2012(E) 4.1.2.2, ISO/IEC 17021:2006(E)
> 8.6.3).
> > ISO/IEC 17065:2012(E), Section 7.7 describes the general reporting: the
> > name of the CAB, the date the certification is granted, the name and
> > address of the client, the scope of the certification, the validity
> > period/expiration date of the certificate, and anything else required by
> > the certification scheme (319 411-1).
>
>

> If I am understanding correctly... an ETSI auditor performs a
> point-in-time audit and then relies on the CA to notify the auditor of
> material changes.
>

That is my understanding as well.

> Wish I could say that I believe that would work. Unfortunately, based on
> my experience with CAs I do not believe the CAs will be pro-active in this
> way. And I think the only way to really know what a CA is doing is to look
> at their data -- look at the actual certs they are issuing, and their
> documentation regarding such certs/issuance.
>

17065 and 17021 do specify that there is review of historic data - largely,
to determine whether or not the specified control was operating correctly.

However, 17065/17021 do not specify how far back the CAB needs to look -
that's left for the specific criteria being applied (e.g. EN 319 411-1, 319
401, and 319 403)

The best way to explain how the ETSI audits work that I can think is to
have a read on Section 7.4.5 of EN 319 403 v2.2.2 (the latest version at
https://portal.etsi.org/tbsitemap/esi/trustserviceproviders.aspx ). The
audit is done in two stages. In the first stage, the TSP provides
documentation to the CAB about their systems and practices and the CAB
works with the TSP to gather documentation. After this, there's a site
visit - Stage 2 - in which the auditor attempts to confirm the TSP is
adhering to its practices.

As you can read, these are about evaluating processes going forward. There
is retrospective analysis - such as document review and the Stage 2
evidence collection - but not the period-of-time analysis as done within
WebTrust-based audits.

Further, if you look at Section 7.6, it's worth noting that:
"""
a TSP audit may be passed with pending nonconformities provided that these
do not impact the ability of the
TSP to meet the the intended service. This certification decision is
conditional upon to the implementation of
corrective actions within 3 months after conclusion of the audit (depending
on the type and criticality of the
correction(s))

"""


> > The challenge is in determining whether this is a correct understanding
> of
> > the respective differences,
>
>

> How do we verify that this is correct?
>

I would expect that it would be incumbent on the CABs and the CAs providing
EN 319 411-1 certificates to help the community better understand the level
of assurance provided. That is, I think those supporting the continued
recognition of ETSI should attempt to demonstrate where either the
understanding of WebTrust-based audits or EN 319 411-1 certificates is
incorrect or inaccurate. Otherwise, I think your conclusions - about no
longer recognizing such schemes - are reasonable.

> Based on what I've seen with CAs over the past several years, I do not
> believe the 'forward-looking' approach with self-reporting is sufficient.
>

Agreed

> I think we have to (and we do!) require the backward-looking approach.
>

As noted, there is some retrospective analysis - document review and
evidence gathering - but the fundamental process of a 319 411-1 audit seems
to be with such a different objective and way of measuring that WebTrust to
ETSI is like comparing apples to oranges.

> CAs can make all the "future promises" they want, but the proof is in the
> resulting data -- the certs that they issue.
>
> Based on the above information, I do not think the ETSI audits meet the
> requirements of Mozilla's Root Store Policy or the CA/Browser Forum
> Baseline Requirements. Maybe that's the real problem here.
>
> Am I missing something?

I've been increasingly thinking the same thing.

[Ryan Sleevi]

The auditor granted a certificate on 2017-06-21, after having made a
determination to do so at some point earlier, based on their engagement of
Phase 1 and Phase 2, which was conducted between January 30 and March 1,
2017.

There is no requirement that I can find - within 319 411-1, 319 411-2, 319
403, or 319 401, that would require the CAB to evaluate or consider
evidence from March 10 2016. In particular, 319 403 (7.4.5.2) states "The
objective of the audit is to confirm and certify that the TSP and the trust
services it provides complies with the applicable assessment criteria."

Thus, on the basis of the public information provided, I do not believe we
have a sufficient level of assurance that the CA's activities between March
10 2016 until January 29 2017 were consistent. Further, given the
opportunity for corrective actions without qualification, I do not believe
we have a sufficient level of assurance that the CA's activities between
January 30, 2017 and March 1, 2017 were consistent.

[Kathleen Wilson]

On Monday, October 30, 2017 at 2:59:31 PM UTC-7, Ryan Sleevi wrote:
>
> I would expect that it would be incumbent on the CABs and the CAs providing
> EN 319 411-1 certificates to help the community better understand the level
> of assurance provided. That is, I think those supporting the continued
> recognition of ETSI should attempt to demonstrate where either the
> understanding of WebTrust-based audits or EN 319 411-1 certificates is
> incorrect or inaccurate. Otherwise, I think your conclusions - about no
> longer recognizing such schemes - are reasonable.

I hope that CAs who rely on ETSI audits are following this discussion forum, and that they will promptly add their comments/explanation here, and ask their auditors to do the same.

I've filed this issue:
https://github.com/mozilla/pkipolicy/issues/105
In which I said:
~~
I think that all CAs should be held to the same level of assurance/audits.

So, I think we have two choices:

1) Remove ETSI as an acceptable audit scheme.

2) The ETSI folks update their audit schemes (that Mozilla's Root Store Policy currently allows) to meet our requirements about looking backward at certificate issuance data -- period-of-time audits as described above and in our policy and the BRs.
~~


Thanks,
Kathleen

[Buschart, Rufus]

Our ETSI audit report (https://www.siemens.com/corp/pool/pki/siemens_etsi.pdf) states:

> An audit of the certification service, documented in a report, provided evidence that the requirements of the following
> specification have been fulfilled. The audit was conducted on 22th - 24th February 2017 covering the timeframe
> 27th February 2016 to 21st February 2017. It was a full audit covering all aspects of the standard performed.
> A second and third audit was performed on 19th and 20th June 2017 to implement further Issuing CAs and in the time
> between 23rd to 30th August.

We repeat this full audit annually. From what I understand out of this discussion, this will meet your requirements, correct?

If you want us to move from ETSI to Webtrust we, and probably every other CA relying on ETSI, would highly appreciate a reasonable grace period to do so, since we are already in the middle of the preparation of our next audit in February 2018.

With best regards,
Rufus Buschart

Siemens AG
Information Technology
Human Resources
PKI / Trustcenter
GS IT HR 7 4
Hugo-Junkers-Str. 9
90411 Nuernberg, Germany
Tel.: +49 1522 2894134
mailto:rufus.b...@siemens.com

www.siemens.com/ingenuityforlife

-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+rufus.buschart=sieme...@lists.mozilla.org] On Behalf Of Kathleen Wilson via dev-security-policy
Sent: Montag, 30. Oktober 2017 23:31
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: ETSI audits not listing audit periods

On Monday, October 30, 2017 at 2:59:31 PM UTC-7, Ryan Sleevi wrote:
>

> I would expect that it would be incumbent on the CABs and the CAs
> providing EN 319 411-1 certificates to help the community better
> understand the level of assurance provided. That is, I think those
> supporting the continued recognition of ETSI should attempt to
> demonstrate where either the understanding of WebTrust-based audits or
> EN 319 411-1 certificates is incorrect or inaccurate. Otherwise, I
> think your conclusions - about no longer recognizing such schemes - are reasonable.

I hope that CAs who rely on ETSI audits are following this discussion forum, and that they will promptly add their comments/explanation here, and ask their auditors to do the same.

I've filed this issue:
https://github.com/mozilla/pkipolicy/issues/105
In which I said:
~~
I think that all CAs should be held to the same level of assurance/audits.

So, I think we have two choices:

1) Remove ETSI as an acceptable audit scheme.

2) The ETSI folks update their audit schemes (that Mozilla's Root Store Policy currently allows) to meet our requirements about looking backward at certificate issuance data -- period-of-time audits as described above and in our policy and the BRs.
~~


Thanks,
Kathleen

_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

[Kathleen Wilson]

On Monday, October 30, 2017 at 5:02:08 PM UTC-7, Buschart, Rufus wrote:
> Our ETSI audit report (https://www.siemens.com/corp/pool/pki/siemens_etsi.pdf) states:
>
> > An audit of the certification service, documented in a report, provided evidence that the requirements of the following
> > specification have been fulfilled. The audit was conducted on 22th - 24th February 2017 covering the timeframe
> > 27th February 2016 to 21st February 2017. It was a full audit covering all aspects of the standard performed.
> > A second and third audit was performed on 19th and 20th June 2017 to implement further Issuing CAs and in the time
> > between 23rd to 30th August.
>
> We repeat this full audit annually. From what I understand out of this discussion, this will meet your requirements, correct?

Yes, that meets our requirement regarding stating the audit period and if it is a period-of-time/full audit. The problem is that most ETSI audit statements that we get do not say this. And it has been an uphill battle for me to get ETSI audit statements to say this.

Please note that there is still information missing from the audit statement, such as SHA-256 fingerprints. See:
https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#public-audit-information


But your audit statement is much better than most ETSI audit statements I get.

>
> If you want us to move from ETSI to Webtrust we, and probably every other CA relying on ETSI, would highly appreciate a reasonable grace period to do so, since we are already in the middle of the preparation of our next audit in February 2018.

I understand.

Thanks,
Kathleen

[Moudrick M. Dadashov]

FYI:

see section 7.4.4 of ETSI EN 319 403, Electronic Signatures and
Infrastructures (ESI); Trust Service Provider Conformity Assessment -
Requirements for conformity assessment bodies assessing Trust Service
Providers,
http://www.etsi.org/deliver/etsi_en/319400_319499/319403/02.02.02_60/en_319403v020202p.pdf

Thanks,
M.D.

On 10/31/2017 2:13 AM, Kathleen Wilson via dev-security-policy wrote:
> On Monday, October 30, 2017 at 5:02:08 PM UTC-7, Buschart, Rufus wrote:

>> Our ETSI audit report (https://www.siemens.com/corp/pool/pki/siemens_etsi.pdf) states:
>>
>>> An audit of the certification service, documented in a report, provided evidence that the requirements of the following
>>> specification have been fulfilled. The audit was conducted on 22th - 24th February 2017 covering the timeframe
>>> 27th February 2016 to 21st February 2017. It was a full audit covering all aspects of the standard performed.
>>> A second and third audit was performed on 19th and 20th June 2017 to implement further Issuing CAs and in the time
>>> between 23rd to 30th August.
>> We repeat this full audit annually. From what I understand out of this discussion, this will meet your requirements, correct?
>

> Yes, that meets our requirement regarding stating the audit period and if it is a period-of-time/full audit. The problem is that most ETSI audit statements that we get do not say this. And it has been an uphill battle for me to get ETSI audit statements to say this.
>
> Please note that there is still information missing from the audit statement, such as SHA-256 fingerprints. See:
> https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#public-audit-information
>
>
> But your audit statement is much better than most ETSI audit statements I get.
>
>

>> If you want us to move from ETSI to Webtrust we, and probably every other CA relying on ETSI, would highly appreciate a reasonable grace period to do so, since we are already in the middle of the preparation of our next audit in February 2018.
>

> I understand.

[Moudrick M. Dadashov]

You might want to add one more:

REGULATION (EC) No 765/2008 OF THE EUROPEAN PARLIAMENT AND OF THE
COUNCIL of 9 July 2008 setting out the requirements for accreditation
and market surveillance relating to the marketing of products and
repealing Regulation (EEC) No 339/93

see also eIDAS  recital (44).

Thanks,
M.D.

On 10/30/2017 11:18 PM, Ryan Sleevi via dev-security-policy wrote:

> testing or inspection, along with periodic surveillance. Importantly,

> within 17065 and 17021, the way of ensuring continued compliance is done
> based on contracts and reporting - that is, the client is responsible for
> reporting changes to their conformity assessment body, and the CAB may
> determine to revoke the certification or indicate corrective actions should
> be taken (see ISO/IEC 17065:2012(E) 4.1.2.2, ISO/IEC 17021:2006(E) 8.6.3).
> ISO/IEC 17065:2012(E), Section 7.7 describes the general reporting: the
> name of the CAB, the date the certification is granted, the name and
> address of the client, the scope of the certification, the validity
> period/expiration date of the certificate, and anything else required by
> the certification scheme (319 411-1).
>

> Within the WebTrust scheme, the reports are based on either US (AICPA)
> Standards of AT101, Canadian (CPA Canada) Standards of Section 5025, or
> IFAC's ISAE3000 standards. What is important and notable about these is
> that they are reports over information, with a scope and norm (e.g.
> WebTrust for CAs) applied. This is why there's a consistent period of time
> - information is collected and the auditor evaluates, on the basis of that
> information, the level of assurance that is being met.
>

> I'm still working to have a better understanding here (and this is all in a
> personal capacity), but my conclusion is this:
> - "ETSI" audits reflect an engagement at a particular point in time, where
> a series of system controls are evaluated, and the result is a certificate
> indicating the process and products comply with the relevant criteria (319
> 411-1). This is similar to a "Point in Time Readiness Assessment". After
> the certification is granted, the TSP is responsible for notifying the CAB
> of changes (in the future), and the CAB evaluates whether those changes
> should cause it to revoke certification or institute some corrective action
> (which may not be noted in the certification). An ETSI audit is thus meant
> to indicate a CA "is" or "will be" conforming.
> - "WebTrust" audits reflect an engagement looking over the past, by the CA
> providing documentation and evidence that their policies were consistent
> with the requirements. That is, a WebTrust audit is meant to assess whether
> or not a CA is being accurate when it says it "was" conforming, but makes
> no guarantees about whether or not a CA "is" conforming, nor whether or not
> the CA disclosed all relevant information.
>
>

> The challenge is in determining whether this is a correct understanding of

> the respective differences, and then determining whether or not the

> certification-based approach provides the desired or sufficient level of
> assurance - in particular, its 'forward-looking' approach with
> self-reporting, rather than the WebTrust-style assurance engagement that is
> more 'backward-looking' without future promises.

[Arno Fiedler]

Hello,
there is a problem with the auditor qualification and the national accreditation of the auditing body.
We´ll ask ACABc to suggest a solution to take care about proper education of "qualified" auditors and "good practise" audit statements as suggested by Mozilla.
Maybe a "white listing" of qualified audit bodies is the best way.

Best regards Arno Fiedler

[Dimitris Zacharopoulos]

On 31/10/2017 11:12 πμ, Arno Fiedler via dev-security-policy wrote:
> Am Montag, 30. Oktober 2017 22:19:31 UTC+1 schrieb Ryan Sleevi:

> Hello,
> there is a problem with the auditor qualification and the national accreditation of the auditing body.
> We´ll ask ACABc to suggest a solution to take care about proper education of "qualified" auditors and "good practise" audit statements as suggested by Mozilla.
> Maybe a "white listing" of qualified audit bodies is the best way.
>
> Best regards Arno Fiedler

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

According to the list of accredited CABs
<https://ec.europa.eu/futurium/en/system/files/ged/list_of_eidas_accredited_cabs-2017-10-27.pdf>
posted by the eIDAS observatory, AENOR (CAB) is accredited by ENAC (NAB
of Spain) under "ISO/IEC 17065 + ETSI EN 319 403 + eIDAS Art.3.18 scope
of accreditation".

Here is the URL of their Accreditation Certificate
(http://www.enac.es/documents/7020/5ae31445-73fa-4e16-acc4-78e079375c4f)


Dimitris.

[Gervase Markham]

Hi Arno,

On 31/10/17 08:46, Arno Fiedler wrote:
> there is a problem with the auditor qualification and the national accreditation of some auditing bodies.

Can you help us understand what about the discussion so far leads you to
that conclusion? It seems to me that the problem being raised is with
the very nature of ETSI audits, not with the qualifications of the
auditor or with national accreditation.

Gerv

[Gervase Markham]

On 31/10/17 00:13, Kathleen Wilson wrote:
> Yes, that meets our requirement regarding stating the audit period
> and if it is a period-of-time/full audit. The problem is that most
> ETSI audit statements that we get do not say this. And it has been an
> uphill battle for me to get ETSI audit statements to say this.

But is the problem that ETSI audit statements don't _say_ this, or is
the problem that ETSI audits don't _do_ this?

It seems to me the problem is the latter - i.e. it's not a problem with
correctly describing what was done, it's a problem with what was
actually done (or not done).

Gerv

[Arno Fiedler]

Am Montag, 30. Oktober 2017 22:19:31 UTC+1 schrieb Ryan Sleevi:

Hello,
please give us some days to prepare a proper answer about the European Audit scheme based on ETSI CPs. The next CA-Day takes place on November 28th, we will change the agenda to find a solution that covers the concerns.
Best regards
Arno

[m.wied...@tuvit.de]

TÜViT as a conformity assessment body would like to add some explanations to clear up some misunderstandings about ETSI auditing.
First of all, we would like to give one preliminary remark. ETSI has separated the TSP technical requirements (ETSI EN 319 411-1, ETSI EN 319 401) from the CAB auditing requirements (ETSI EN 319 403). This is an ISO international standardization best practice (principle 1 of ISO / IEC 17007) not to mix technical requirements (“what to audit”) with auditing requirements (“how to perform such audits”).

Given that, the main issue of this thread seems to be the question about the point-in-time vs. period-of-time audit approach.
Of course, every properly conducted audit has to include and examine the point-in-time situation at the time of the current audit. But beyond that, every properly conducted audit must cover the operations performed since the last audit. The latter is defined by ETSI EN 319 403. In its section 7.9 it deals with surveillance and re-assessment. It has already been agreed upon without objections (during the CABF F2F Bilbao meeting, if we remember correctly) that only annual full system audits are eligible in the context of CABF BR and/or EVG. As a consequence, every audit (aside from the very first initial audit => point-in-time / pre-issuance readiness assessment) must be performed as re-assessment and the relaxations for surveillance audits introduced in this section do not apply in the BR / EVG context. In its last sentence this section declares, that auditors must cover a sample of records on the operations performed since the previous audit.
It is our firm opinion, shared by our national accreditation body, that this defines an audit period ranging from the date of the last audit up to the date of the current audit and that the complete period needs to be examined, not only certain bits and pieces of it. However, if the current phrase is causing problems, we are absolutely willing to approach ETSI in order to request an explicit clarification of this in a future version of the standard.
As you correctly pointed out, the ETSI EN 319 403 in section 7.10 includes a notification scheme, where the CA’s are required to notify changes affecting the certification to the conformity assessment body. But as stated above, this is not a replacement for the backward-looking period-of-time audit approach. It is a supplement in order to perform auditing of security-relevant changes (and identify changes that would be non-conformant!) in a timely manner and not only at the next annual full audit.
Summarizing this, a properly conducted ETSI audit includes both point-in-time and period-of-time (and the latter both backward- as well as forward-).

Another issue obviously is the proper format of a publicly facing audit report / audit attestation to be presented to the root programs.
With this we are totally on your side, an acceptable report / attestation must clearly state the required facts in order to enable to deal with it. These reports are not and cannot be completely defined in the ETSI EN 319 403 standard, as different use cases (“consumers”) might require different sets of information.
However, in our opinion the CABF and browser root programs give sufficient information about the required contents. We are willing to readopt the idea of creating a mandatory template for such audit attestation. This idea had been in the field since some CABF F2F-Meetings but obviously has never been finalized. Root programs could then simply decline all attestations that do not conform to the template.
We are open to further discussions, e.g. at the next CA-Day (https://www.tuvit.de/en/news/events/events-detail/details/9-ca-day-2017/) or at any upcoming CABF F2F meeting.

Best regards
Matthias Wiedenhorst, TÜViT

[Moudrick M. Dadashov]

Thank you for clarification.

Do you think the terms "/approval scheme/", "/supervision scheme/",
"/accreditation//scheme/" etc. (used in some ETSI TSs or the Commission
Decisions) have the same meaning and ETSI EN 319 403 is just one of
possible "/certification scheme/s"?

Thanks,
M.D.

[Jakob Bohm]

(This is my interpretation, I don't have authority on this):

The biggest issue (in this thread at least), is that a number of "public
audit reports" (the portion of an audit that are signed by the auditor
and published by the audited CA/TSP) lack the following 3 pieces of
basic information:

1. The specific date of the previous audit (the start of the period
covered by a period-in-time audit), thus making it impossible for
readers of such reports to determine if they did not receive or
otherwise missed a period-in-time audit covering a previous period
(the requirements for the Mozilla root program allow, and in rare
cases require, period-in-time audits more frequently than 1/year).

2. A simple statement that this is a period-in-time audit covering the
period from the stated date (see #1) to the "audit date" and not a
point-in-time audit for the "audit date" only.

3. For audits referring to the older ETSI audit regime (not the current
regime it appears), a statement that the audit was an actual full
audit with period-in-time checking etc. and not a reaffirmation of one
of the weaker schemes permitted by those old rules (forwarding
looking, notification based, not actually checked every year etc.)

Possibly the following are also missing from some audits:

4. A list of the audited CA certificates, stated in full or as hashes.

5. Compliance with any CAB/F BRs introduced after ETSI took its snapshot
of the CAB/F BRs.

6. Compliance with any additional Mozilla requirements for TLS server
("WebPKI") certificates above and beyond the CAB/F BRs. Mozilla need
not be mentioned as long as each item is somehow described as being in
the required way and audited as such.

7. Compliance with the various BR-like requirements for e-mail
certificates as currently listed in the Mozilla policy. Mozilla need
not be mentioned as long as each item is somehow described as being in
the required way and audited as such. Some of these may, for example,
already be audited ETSI requirements under a referenced ETSI standard.



Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded