Re: 46 Certificates issued with BR violations (KIR)

Wayne Thayer

unread,

Oct 8, 2018, 1:14:09 PM10/8/18

to piotr.g...@kir.pl, mozilla-dev-security-policy

Thank you for the incident report. I have posted it to the bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1495497

On Mon, Oct 8, 2018 at 8:25 AM piotr.grabowski--- via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Here's the incident report:
>
> 1. How your CA first became aware of the problem (e.g. via a problem
> report submitted to your Problem Reporting Mechanism, via a
>
> discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the
> date.
>
> Email from Wayne Thayer Oct 1, 2018
>
> 2. A timeline of the actions your CA took in response.
>
> A. Oct 2, 2018 - Investigation began.
> B. Oct 4, 2018 - Found impacted certificate policy templates.
> C Oct 4, 2018 - All the certificates owners were contacted and agreed on
> issuance new BR compliant certificates in time convenient for them,
> preferably not later than by the end of this year and revocation
> current ones.
> D. Oct 8, 2018 - Fixed impacted certificate policy templates.
> E. Oct 8, 2018 - This disclosure.
>
> Ongoing:
> F. Replacement of impacted certificates
> G. Training of periodic certificate policy templates validation.
>
> 3. Confirmation that your CA has stopped issuing TLS/SSL certificates
> with the problem.
>
> Confirmed.
>
> 4. A summary of the problematic certificates. For each problem: number
> of certs, and the date the first and last certs with that problem
>
> were issued.
>
> There are 46 certificates. The certificates were issued between Feb 20,
> 2017 and Sep 25, 2018.
>
> 5. The complete certificate data for the problematic certificates. The
> recommended way to provide this is to ensure each certificate is
>
> logged to CT and then list the fingerprints or crt.sh IDs, either in the
> report or as an attached spreadsheet, with one list per distinct
> problem.
>
> Added as attachment
>
> https://crt.sh/?caid=15985&opt=cablint,zlint,x509lint&minNotBefore=2017-01-01
>
> 6. Explanation about how and why the mistakes were made or bugs
> introduced, and how they avoided detection until now.
>
> The the incident concerns 46 certificates in the vast majority issued on
> KIR S.A. internal system purposes.
> The root cause of this issue was human error in certificate policy
> templates.
>
> "Human error" is not a sufficient response to this questions. Please
describe the process in place for modifying policy templates and how it
failed to catch this problem.

>
> Remediation items:
>
> 1. Reviewed all certificate policy templates for ensuring that all of them
> are BR comliant.
> 2. All the certificates owners were contacted and agreed on issuance new
> BR compliant certificates in time convenient for them, preferably not later
> than by the end of this year and revocation current ones.
> 3. Added procedural step for periodic certificate policy templates
> validation.
>
> None of these remediation items will prevent future problems of this
nature. How will you improve your processes to prevent future misissuance?

I assume that KIR S.A. has not implemented pre-issuance linting. Why not?
Is there a plan to implement pre-issuance linting? When?

>
> We have by the way question about error: ERROR: The 'Organization Name'
> field of the subject MUST be less than 64 characters.
> According to https://www.ietf.org/rfc/rfc5280.txt and the note from this
> RFC 'ub-organization-name INTEGER ::= 64. For UTF8String or UniversalString
> at least four times the upper bound should be allowed. So what is the max
> length of this field for UTF8String?
>
> Page 113 of RFC 5280 limits the length of the field to 64 characters
regardless of the data type:

-- Naming attributes of type X520OrganizationName:
-- X520OrganizationName ::=
-- DirectoryName (SIZE (1..ub-organization-name))
--
-- Expanded to avoid parameterized type:
X520OrganizationName ::= CHOICE {
teletexString TeletexString
(SIZE (1..ub-organization-name)),
printableString PrintableString
(SIZE (1..ub-organization-name)),
universalString UniversalString
(SIZE (1..ub-organization-name)),
utf8String UTF8String
(SIZE (1..ub-organization-name)),
bmpString BMPString
(SIZE (1..ub-organization-name)) }

Ryan Sleevi

unread,

Oct 8, 2018, 8:25:48 PM10/8/18

to piotr.g...@kir.pl, mozilla-dev-security-policy

On Mon, Oct 8, 2018 at 11:25 AM piotr.grabowski--- via dev-security-policy <

dev-secur...@lists.mozilla.org> wrote:

> Here's the incident report:
>
> 1. How your CA first became aware of the problem (e.g. via a problem
> report submitted to your Problem Reporting Mechanism, via a
>
> discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the
> date.
>
> Email from Wayne Thayer Oct 1, 2018
>
> 2. A timeline of the actions your CA took in response.
>
> A. Oct 2, 2018 - Investigation began.
> B. Oct 4, 2018 - Found impacted certificate policy templates.
> C Oct 4, 2018 - All the certificates owners were contacted and agreed on
> issuance new BR compliant certificates in time convenient for them,
> preferably not later than by the end of this year and revocation
> current ones.
> D. Oct 8, 2018 - Fixed impacted certificate policy templates.
> E. Oct 8, 2018 - This disclosure.

Can you please re-review
https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report ?

In the design of this template, one of the concerns was about understanding
*how* a problem happened, not just how a CA responded. This is why it
includes text such as "This may include events before the incident was
reported, such as when a particular requirement became applicable, or a
document changed, or a bug was introduced, or an audit was done."

1) When were the policy templates introduced
2) When were the policy templates reviewed
3) What are the templates review practices
4) What controls, if any, exist to ensure that all templates are
appropriate to the controls?

The misconfiguration of certificate policy templates is a significant
incident, precisely because there have been significant CA misissuances as
a result of it. In this regard, a CA that is misconfiguring policy
templates is arguably as negligent as one failing to perform domain
validation - this is an incredibly significant mistake by a CA. A
responsible CA seeking continued trust in their certificates would thus
want to demonstrate that they understood how significant this was, and
provide detailed descriptions about the timeline of events and the controls
and practices they have in place to mitigate the risk of template
misconfiguration. Anything short of that is gross negligence on behalf of a
CA.

Grabowski Piotr

unread,

Oct 9, 2018, 11:44:18 AM10/9/18

to Wayne Thayer, mozilla-dev-security-policy

Hello Wayne,

Please find our comments below:

So far the process for modifying policy templates was controlled by only one person at the moment. Although these persons
have an extensive experience in PKI and preparing certificate templates and in common daily duties they work with serveral CA's and with hundreds of templates and it was the first (and we hope the last) time that something was misconfigured . This incident showed us that this process should be definitely changed and that's why practicing due care we are updating remediation steps or plan for:

1. Reviewed all certificate policy templates for ensuring that all of them are BR comliant. /done
2. All the certificates owners were contacted and agreed on issuance new BR compliant certificates in time convenient for them, preferably not later than by the end of this year and revocation current ones. /done
3. Added procedural step for periodic certificate policy templates validation. /done

4. Implement dual CAO control for modifying policy template which requires the presence of 2 CAO's (Certification Authority Operators) /done
5. CAO audit logs will be daily scanned by system Auditor for policy change events and comapared if they match events from point 4) /done
6. Implement pre-issuance linting - we have sent request, urgent need to our PKI software vendor about delivery this functionality.
If we know the timeframe we will update this report.

Do you think it is a sufficient response to incident?

Best Reagrds
Piotr Grabowski
________________________________
Od: Wayne Thayer <wth...@mozilla.com>
Wysłane: poniedziałek, 8 października 2018 19:13:46
Do: Grabowski Piotr
DW: mozilla-dev-security-policy
Temat: Re: 46 Certificates issued with BR violations (KIR)

Thank you for the incident report. I have posted it to the bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1495497

On Mon, Oct 8, 2018 at 8:25 AM piotr.grabowski--- via dev-security-policy <dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>> wrote:
Here's the incident report:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a

discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date.

Email from Wayne Thayer Oct 1, 2018

2. A timeline of the actions your CA took in response.

A. Oct 2, 2018 - Investigation began.
B. Oct 4, 2018 - Found impacted certificate policy templates.
C Oct 4, 2018 - All the certificates owners were contacted and agreed on issuance new BR compliant certificates in time convenient for them, preferably not later than by the end of this year and revocation current ones.
D. Oct 8, 2018 - Fixed impacted certificate policy templates.
E. Oct 8, 2018 - This disclosure.

[logo]

Krajowa Izba Rozliczeniowa S.A., zarejestrowana w Sądzie Rejonowym dla m. st. Warszawy,
XIII Wydział Gospodarczy Krajowego Rejestru Sądowego pod nr KRS 0000113064,
NIP 526-030-05-17, REGON 012105474, kapitał zakładowy i wpłacony 5.445.000 zł

Informacja zawarta w tej korespondencji jest przeznaczona tylko dla osoby lub jednostki, do której jest adresowana. Może ona zawierać zastrzeżone i poufne informacje i jeżeli to nie Państwo są wskazanym odbiorcą, nie można kopiować, rozpowszechniać lub podejmować żadnych czynności w oparciu o nią. W przypadku otrzymania tej korespondencji przez pomyłkę, proszę powiadomić nadawcę za pomocą emaila zwrotnego i usunąć tę korespondencję (wraz z załącznikami) z Państwa systemu.

The information contained in this transmission is intended only for the individual or entity to whom it is addressed. It may contain privileged and confidential information and if you are not an indicated recipient, you must not copy, distribute or take any action in reliance on it. If received in error, please notify the sender by return email and delete his transmission (and any attachments) from your system.

Wayne Thayer

unread,

Oct 9, 2018, 5:46:02 PM10/9/18

to Piotr.G...@kir.pl, mozilla-dev-security-policy

On Tue, Oct 9, 2018 at 5:30 AM Grabowski Piotr <Piotr.G...@kir.pl>
wrote:

> Hello Wayne,
>
> Please find our comments below:
>
>
> So far the process for modifying policy templates was controlled by only
> one person at the moment. Although these persons
> have an extensive experience in PKI and preparing certificate templates
> and in common daily duties they work with serveral CA's and with hundreds
> of templates and it was the first (and we hope the last) time
> that something was misconfigured . This incident showed us that this
> process should be definitely changed and that's why practicing due care
> we are updating remediation steps or plan for:
>
> 1. Reviewed all certificate policy templates for ensuring that all of them
> are BR comliant. /done
> 2. All the certificates owners were contacted and agreed on issuance new
> BR compliant certificates in time convenient for them, preferably not later
> than by the end of this year and revocation current ones. /done
> 3. Added procedural step for periodic certificate policy templates
> validation. /done
>
> 4. Implement dual CAO control for modifying policy template which requires
> the presence of 2 CAO's (Certification Authority Operators) /done
> 5. CAO audit logs will be daily scanned by system Auditor for policy
> change events and comapared if they match events from point 4) /done

> 6. Implement pre-issuance linting - we have sent request, *urgent need*

> to our PKI software vendor about delivery this functionality.
> If we know the timeframe we will update this report.
>
> Do you think it is a sufficient response to incident?
>

I do appreciate your prompt response, but #4 in particular strikes me as an
inadequate quick fix lacking any plan to make a concerted effort to learn
from this mistake.

- Is the new dual control process documented in a manner that will be
auditable by your external auditors?
- How is version control maintained for policy templates?
- Despite the review, is it possible for one malicious employee to modify a
policy template by themselves? If not, why not?
- Have you conducted an overall review of your practices looking for other
areas where a human error can result in misissuance? If so, what did you
find and how are you addressing it?
- Why, despite the numerous misissuances documented on this list, has KIR
not even begun the process of implementing pre-issuance linting until now?
- Why is KIR not performing post-issuance linting? This problem had been
occurring for over a year and there are readily available tools (
https://crt.sh) that allow anyone to identify these problems.

Also, please respond to Ryan's email questioning how this happened.

- Wayne

>
>
>
>
Best Reagrds
> Piotr Grabowski
> ------------------------------
> *Od:* Wayne Thayer <wth...@mozilla.com>
> *Wysłane:* poniedziałek, 8 października 2018 19:13:46
> *Do:* Grabowski Piotr
> *DW:* mozilla-dev-security-policy
> *Temat:* Re: 46 Certificates issued with BR violations (KIR)

>
> Thank you for the incident report. I have posted it to the bug:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1495497
>
> On Mon, Oct 8, 2018 at 8:25 AM piotr.grabowski--- via dev-security-policy <

> [image: logo]

Grabowski Piotr

unread,

Oct 10, 2018, 4:33:14 PM10/10/18

to Wayne Thayer, mozilla-dev-security-policy

Hello Wayne,

- Is the new dual control process documented in a manner that will be auditable by your external auditors?

Yes, the new dual control process is already included in the document called instruction of the security of system Szafir (internal name of the PKI system) and it is one of the document that is presented to internal and external auditors.

- How is version control maintained for policy templates?

We are using Verizon UniCERT PKI software so version control from the software side is controlled intrenally in CAO
database components. Every policy (template) has its own number, status, date of creation and modification. Every change in
this template/policy is noted in read-only audit log that policy/temaplte with given id/name was modified and by who. Audit log is secured from tampering and can be accessed only by person with role Auditor by using multifactor authentication including physical access to datacenter room, dedicated domain credentials, smartcard (PIN) with certificate to login to Auditor application module. Every modification to policy/template enforces new version of the policy/template with new seqential number, dates and so on. Older version can not be deleted, only can be marked as "retired".

- Despite the review, is it possible for one malicious employee to modify a policy template by themselves? If not, why not?

It is impossible. CAO role is one of the most trusted role so it has to have physical access to datacenter room, dedicated domain credentials, smartcard (PIN) with certificate to login to CAO application module.

- Have you conducted an overall review of your practices looking for other areas where a human error can result in misissuance? If so, what did you find and how are you addressing it?

Yes, we have conducted an overall review and have not found any other areas where a human error can result in misissuance.

- Why, despite the numerous misissuances documented on this list, has KIR not even begun the process of implementing pre-issuance linting until now?

We have started process of implementing pre-issuance linting just after your email pointing our misissuance. We have requested pre-issuance linting functionality/patch with high priority from Verizon.

- Why is KIR not performing post-issuance linting? This problem had been occurring for over a year and there are readily available tools (https://crt.sh) that allow anyone to identify these problems.

We will implement post-issuance linting as well.

Best regards,

Piotr Grabowski

________________________________
Od: Wayne Thayer <wth...@mozilla.com>

Wysłane: wtorek, 9 października 2018 23:45:39

Do: Grabowski Piotr
DW: mozilla-dev-security-policy

Temat: Re: Odp.: 46 Certificates issued with BR violations (KIR)

On Tue, Oct 9, 2018 at 5:30 AM Grabowski Piotr <Piotr.G...@kir.pl<mailto:Piotr.G...@kir.pl>> wrote:

Hello Wayne,

Please find our comments below:

So far the process for modifying policy templates was controlled by only one person at the moment. Although these persons
have an extensive experience in PKI and preparing certificate templates and in common daily duties they work with serveral CA's and with hundreds of templates and it was the first (and we hope the last) time that something was misconfigured . This incident showed us that this process should be definitely changed and that's why practicing due care we are updating remediation steps or plan for:

1. Reviewed all certificate policy templates for ensuring that all of them are BR comliant. /done
2. All the certificates owners were contacted and agreed on issuance new BR compliant certificates in time convenient for them, preferably not later than by the end of this year and revocation current ones. /done
3. Added procedural step for periodic certificate policy templates validation. /done

4. Implement dual CAO control for modifying policy template which requires the presence of 2 CAO's (Certification Authority Operators) /done
5. CAO audit logs will be daily scanned by system Auditor for policy change events and comapared if they match events from point 4) /done

6. Implement pre-issuance linting - we have sent request, urgent need to our PKI software vendor about delivery this functionality.

If we know the timeframe we will update this report.

Do you think it is a sufficient response to incident?

I do appreciate your prompt response, but #4 in particular strikes me as an inadequate quick fix lacking any plan to make a concerted effort to learn from this mistake.

- Is the new dual control process documented in a manner that will be auditable by your external auditors?
- How is version control maintained for policy templates?
- Despite the review, is it possible for one malicious employee to modify a policy template by themselves? If not, why not?
- Have you conducted an overall review of your practices looking for other areas where a human error can result in misissuance? If so, what did you find and how are you addressing it?
- Why, despite the numerous misissuances documented on this list, has KIR not even begun the process of implementing pre-issuance linting until now?

- Why is KIR not performing post-issuance linting? This problem had been occurring for over a year and there are readily available tools (https://crt.sh) that allow anyone to identify these problems.

Also, please respond to Ryan's email questioning how this happened.

- Wayne

Best Reagrds
Piotr Grabowski

________________________________
Od: Wayne Thayer <wth...@mozilla.com<mailto:wth...@mozilla.com>>

Wysłane: poniedziałek, 8 października 2018 19:13:46
Do: Grabowski Piotr
DW: mozilla-dev-security-policy

Temat: Re: 46 Certificates issued with BR violations (KIR)

Thank you for the incident report. I have posted it to the bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1495497

[logo]

Krajowa Izba Rozliczeniowa S.A., zarejestrowana w Sądzie Rejonowym dla m. st. Warszawy,
XIII Wydział Gospodarczy Krajowego Rejestru Sądowego pod nr KRS 0000113064,
NIP 526-030-05-17, REGON 012105474, kapitał zakładowy i wpłacony 5.445.000 zł

Informacja zawarta w tej korespondencji jest przeznaczona tylko dla osoby lub jednostki, do której jest adresowana. Może ona zawierać zastrzeżone i poufne informacje i jeżeli to nie Państwo są wskazanym odbiorcą, nie można kopiować, rozpowszechniać lub podejmować żadnych czynności w oparciu o nią. W przypadku otrzymania tej korespondencji przez pomyłkę, proszę powiadomić nadawcę za pomocą emaila zwrotnego i usunąć tę korespondencję (wraz z załącznikami) z Państwa systemu.

The information contained in this transmission is intended only for the individual or entity to whom it is addressed. It may contain privileged and confidential information and if you are not an indicated recipient, you must not copy, distribute or take any action in reliance on it. If received in error, please notify the sender by return email and delete his transmission (and any attachments) from your system.

[logo]

Grabowski Piotr

unread,

Oct 10, 2018, 4:58:41 PM10/10/18

to ry...@sleevi.com, mozilla-dev-security-policy

Hello Ryan,

In the design of this template, one of the concerns was about understanding *how* a problem happened, not just how a CA responded. This is why it includes text such as "This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done."

1) When were the policy templates introduced

We are using Verizon UniCERT PKI software. Policy or templates are integral part of the software and they exists there all along.

2) When were the policy templates reviewed

All policies/templates were reviewed right after the incident occurred. We have also added procedural step for periodic certificate policy templates validation.

3) What are the templates review practices.

We have added dual CAO control for modifying policy template which requires the presence of 2 CAO's (Certification Authority Operators)
All policies/templates are reviewed against the purpose of given policy and CP/CPS.

4) What controls, if any, exist to ensure that all templates are appropriate to the controls?

We have started process of implementing pre-issuance linting just after email pointing our misissuance. We have requested pre-issuance linting functionality/patch with high priority from Verizon UniCERT. We will implement post-issuance linting with crt.sh as well.

Best regards,

Piotr Grabowski

________________________________

Od: Ryan Sleevi <ry...@sleevi.com>
Wysłane: wtorek, 9 października 2018 02:25:27

Do: Grabowski Piotr
DW: mozilla-dev-security-policy
Temat: Re: 46 Certificates issued with BR violations (KIR)

On Mon, Oct 8, 2018 at 11:25 AM piotr.grabowski--- via dev-security-policy <dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>> wrote:
Here's the incident report:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a

discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date.

Email from Wayne Thayer Oct 1, 2018

2. A timeline of the actions your CA took in response.

A. Oct 2, 2018 - Investigation began.
B. Oct 4, 2018 - Found impacted certificate policy templates.
C Oct 4, 2018 - All the certificates owners were contacted and agreed on issuance new BR compliant certificates in time convenient for them, preferably not later than by the end of this year and revocation current ones.
D. Oct 8, 2018 - Fixed impacted certificate policy templates.
E. Oct 8, 2018 - This disclosure.

Can you please re-review https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report ?

In the design of this template, one of the concerns was about understanding *how* a problem happened, not just how a CA responded. This is why it includes text such as "This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done."

1) When were the policy templates introduced
2) When were the policy templates reviewed
3) What are the templates review practices
4) What controls, if any, exist to ensure that all templates are appropriate to the controls?

The misconfiguration of certificate policy templates is a significant incident, precisely because there have been significant CA misissuances as a result of it. In this regard, a CA that is misconfiguring policy templates is arguably as negligent as one failing to perform domain validation - this is an incredibly significant mistake by a CA. A responsible CA seeking continued trust in their certificates would thus want to demonstrate that they understood how significant this was, and provide detailed descriptions about the timeline of events and the controls and practices they have in place to mitigate the risk of template misconfiguration. Anything short of that is gross negligence on behalf of a CA.

Ryan Sleevi

unread,

Oct 10, 2018, 10:53:28 PM10/10/18

to Piotr.G...@kir.pl, Wayne Thayer, mozilla-dev-security-policy

On Wed, Oct 10, 2018 at 4:33 PM Grabowski Piotr via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Hello Wayne,
>
> - Is the new dual control process documented in a manner that will be
> auditable by your external auditors?
>
> Yes, the new dual control process is already included in the document
> called instruction of the security of system Szafir (internal name of the
> PKI system) and it is one of the document that is presented to
> internal and external auditors.
>

Has this been added to your CP/CPS? If not, why not?

Can you please detail the additional controls that were specified?

> - Despite the review, is it possible for one malicious employee to modify
> a policy template by themselves? If not, why not?
>
> It is impossible. CAO role is one of the most trusted role so it has to
> have physical access to datacenter room, dedicated domain
> credentials, smartcard (PIN) with certificate to login to CAO application
> module.
>

This does not seem to describe why it is impossible. The description of
controls here reads that it is possible for a CAO to do so, if malicious,
and you merely trust the CAO to not be malicious.

> - Have you conducted an overall review of your practices looking for other
> areas where a human error can result in misissuance? If so, what did you
> find and how are you addressing it?
>
> Yes, we have conducted an overall review and have not found any other
> areas where a human error can result in misissuance.
>

Put differently: Have you completed an examination of the controls in place
to ensure that any and all configuration changes that may result in a
change to the operation of the CA undergoes multi-party review prior to and
following implementation, to ensure consistency with the CP and CPS?

Are there any operations that may modify any state within the CA software,
hardware, or operating environment that does not undergo multi-party review
prior and following?

If so, what are those operations.
If not, what are the operations that you have considered and enumerated as
requiring multi-party review prior to and following the modification?

> - Why, despite the numerous misissuances documented on this list, has KIR

> not even begun the process of implementing pre-issuance linting until now?

>
> We have started process of implementing pre-issuance linting just after

> your email pointing our misissuance. We have requested pre-issuance
> linting functionality/patch with high priority from Verizon.
>

This does not answer the question. It states that you have begun the
request, but it does not provide insight as to why you had not previously
done so.

> - Why is KIR not performing post-issuance linting? This problem had been
> occurring for over a year and there are readily available tools (
> https://crt.sh) that allow anyone to identify these problems.
>

> We will implement post-issuance linting as well.

This indicates what you will do, but does not answer why you didn't do.
Part of the post-mortem process is to understand what issues may have
existed, given the readily available nature of the tool and the discussions
on m.d.s.p. regarding other CAs.

For example, perhaps the CA did not have adequate staffing to ensure
participation in m.d.s.p. Perhaps the CA team did not have adequate
training to recognize the similarities and/or value in such.

The expectations upon CAs will continue to increase, and the question is
why did KIR S.A. not increase operational oversight in line with those
increased expectations, which would have allowed better detection and
prevention. It is positive to hear steps are being taken now to address it,
but it's reasonable to question why steps weren't taken then, when this was
a knowable and identified best practice and minimum expectation of CAs.

Ryan Sleevi

unread,

Oct 10, 2018, 10:58:19 PM10/10/18

to Piotr.G...@kir.pl, Ryan Sleevi, mozilla-dev-security-policy

On Wed, Oct 10, 2018 at 4:58 PM Grabowski Piotr <Piotr.G...@kir.pl>
wrote:

> Hello Ryan,
>
>
> In the design of this template, one of the concerns was about
> understanding *how* a problem happened, not just how a CA responded. This
> is why it includes text such as "This may include events before the
> incident was reported, such as when a particular requirement became
> applicable, or a document changed, or a bug was introduced, or an audit was
> done."
>
> 1) When were the policy templates introduced
>
> We are using Verizon UniCERT PKI software. Policy or templates are
> integral part of the software and they exists there all along.
>

I'm uncertain how to interpret your answer.

Are you saying that, until this incident, KIR S.A. operated the UniCERT PKI
software without any modifications whatsoever to the default policy
templates? Did KIR S.A. review these policy templates to ensure compliance
with the Baseline Requirements are met? If you created policy templates
yourselves, when were they created, reviewed, updated, etc.

>From your reply with Wayne, it's clear that the software maintains an audit
log for these operations. Based on your reply, the understanding is that
there are only two versions of the policy templates - the default
configuration as shipped, and now the updated one to mitigate this issue.
Is that a correct understanding?

> 2) When were the policy templates reviewed
>
> All policies/templates were reviewed right after the incident occurred.
> We have also added procedural step for periodic certificate policy
> templates validation.
>

Again, this misunderstands the question. These questions are about
understanding the events *before* the incident, not the events *after*. A
root cause analysis must necessarily trace how the incident happened, which
means understanding what events happened before.

In light of this explanation, please review this question again. When were
the policy templates operating prior to this incident reviewed? When were
they last modified? When were they introduced? Working through the steps of
how the incident happened is an essential part in demonstrating that the
mitigations are appropriate.

> 3) What are the templates review practices.
>
> We have added dual CAO control for modifying policy template which
> requires the presence of 2 CAO's (Certification Authority Operators)
> All policies/templates are reviewed against the purpose of given policy
> and CP/CPS.
>

Similarly, this discusses what has been done, but what was done prior to
this incident? What were the review practices beforehand?

>

Grabowski Piotr

unread,

Oct 12, 2018, 11:16:31 AM10/12/18

to ry...@sleevi.com, Wayne Thayer, mozilla-dev-security-policy

Hello,

My comments in blue.

________________________________
Od: Ryan Sleevi <ry...@sleevi.com>

Wysłane: czwartek, 11 października 2018 04:53
Do: Grabowski Piotr
DW: Wayne Thayer; mozilla-dev-security-policy
Temat: Re: Odp.: Odp.: 46 Certificates issued with BR violations (KIR)

On Wed, Oct 10, 2018 at 4:33 PM Grabowski Piotr via dev-security-policy <dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>> wrote:
Hello Wayne,

- Is the new dual control process documented in a manner that will be auditable by your external auditors?

Yes, the new dual control process is already included in the document called instruction of the security of system Szafir (internal name of the PKI system) and it is one of the document that is presented to internal and external auditors.

Has this been added to your CP/CPS? If not, why not?

-in our CP/CPS we have already the statement that key functions require dual control

Can you please detail the additional controls that were specified?

- Despite the review, is it possible for one malicious employee to modify a policy template by themselves? If not, why not?

It is impossible. CAO role is one of the most trusted role so it has to have physical access to datacenter room, dedicated domain credentials, smartcard (PIN) with certificate to login to CAO application module.

This does not seem to describe why it is impossible. The description of controls here reads that it is possible for a CAO to do so, if malicious, and you merely trust the CAO to not be malicious.

Yes, but you have in mind that the CAO roles are playing only by very big experience and experience (about 20 years) - these persons were building the system, so
we can trust them.

- Have you conducted an overall review of your practices looking for other areas where a human error can result in misissuance? If so, what did you find and how are you addressing it?

Yes, we have conducted an overall review and have not found any other areas where a human error can result in misissuance.

Put differently: Have you completed an examination of the controls in place to ensure that any and all configuration changes that may result in a change to the operation of the CA undergoes multi-party review prior to and following implementation, to ensure consistency with the CP and CPS?

Are there any operations that may modify any state within the CA software, hardware, or operating environment that does not undergo multi-party review prior and following?

If so, what are those operations.

Every other key functions/operations require dual control. Here I mean all staring/ reconfiguration/ upgrade and so on.

If not, what are the operations that you have considered and enumerated as requiring multi-party review prior to and following the modification?

- Why, despite the numerous misissuances documented on this list, has KIR not even begun the process of implementing pre-issuance linting until now?

We have started process of implementing pre-issuance linting just after your email pointing our misissuance. We have requested pre-issuance linting functionality/patch with high priority from Verizon.

This does not answer the question. It states that you have begun the request, but it does not provide insight as to why you had not previously done so.

- Why is KIR not performing post-issuance linting? This problem had been occurring for over a year and there are readily available tools (https://crt.sh) that allow anyone to identify these problems.

We will implement post-issuance linting as well.

We were not aware about this misissuance. We have received the notification about misissuance in October this year and immediately started
fixing the problem. So far no one was reporting to us that there is something wrong with any of our certificates.

This indicates what you will do, but does not answer why you didn't do. Part of the post-mortem process is to understand what issues may have existed, given the readily available nature of the tool and the discussions on m.d.s.p. regarding other CAs.

The same as above. Additionally we have contacted the customers and we are in the proccess of replacement of these certificates.

For example, perhaps the CA did not have adequate staffing to ensure participation in m.d.s.p. Perhaps the CA team did not have adequate training to recognize the similarities and/or value in such.

The expectations upon CAs will continue to increase, and the question is why did KIR S.A. not increase operational oversight in line with those increased expectations, which would have allowed better detection and prevention. It is positive to hear steps are being taken now to address it, but it's reasonable to question why steps weren't taken then, when this was a knowable and identified best practice and minimum expectation of CAs.

KIR is subject to an annual audit of WebTrust compliance and so far the audit has not revealed any irregularities in the management of the CA.

Wayne Thayer

unread,

Jan 9, 2019, 3:53:08 PM1/9/19

to Grabowski Piotr, ry...@sleevi.com, mozilla-dev-security-policy

KIR recently misissued another (pre-)certificate with an organizationName
field containing too many characters [1]. This is despite being given
specific guidance earlier in this thread on the organizationName attribute
[2]. I have requested a new incident report in the bug [3].

A pre-certificate was logged but the OCSP status is reported as "unknown",
so I assume that KIR detected this prior to signing the certificate. If so,
I find it particularly troubling that KIR decided not to report this, and
that after 3 months they still have no commitment from their vendor to
implement pre-issuance linting [4].

- Wayne

[1] https://crt.sh/?id=1036631881&opt=zlint
[2]
https://groups.google.com/d/msg/mozilla.dev.security.policy/IRLlR1AJ0vA/daHgd-IXBAAJ
[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1495497
[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1495497#c5

On Fri, Oct 12, 2018 at 8:16 AM Grabowski Piotr <Piotr.G...@kir.pl>
wrote:

> Hello,
>
> My comments in blue.
>
>

> ------------------------------
> *Od:* Ryan Sleevi <ry...@sleevi.com>
> *Wysłane:* czwartek, 11 października 2018 04:53
> *Do:* Grabowski Piotr
> *DW:* Wayne Thayer; mozilla-dev-security-policy
> *Temat:* Re: Odp.: Odp.: 46 Certificates issued with BR violations (KIR)

>
>
>
> On Wed, Oct 10, 2018 at 4:33 PM Grabowski Piotr via dev-security-policy <

> [image: logo]

Grabowski Piotr

unread,

Jan 11, 2019, 3:50:51 AM1/11/19

to Wayne Thayer, ry...@sleevi.com, mozilla-dev-security-policy

Hello Wayne,

Thank you for your email.

I think it is still the same incident. In my opinion there is no need to create another thread. Let me explain how it happened.

As I said in previous emails we have requested urgent need for pre-linting feature from Verizon just after the incident was reported. I have sent a few reminders

and finally on 18th of December 2018 got a response :

"The feature request of implementing pre-issuance linting was discussed with PM and development today. Our feeling is that UniCERT is policy-centric software and therefore the burden of ensuring that “misissuance” does not happen is on the implementer of the policies. To help with this, we provide registration policy templates in the Registration Policy Wizard and they ensure most of x509lint test. ", but the keyword here is most.

We can not for example restrict the length of the field like organizationName to have no more than 64 characters. We can set it as mandatory or not.

It is the only gap we have now in our policy templates. We tried to mitigate this risk putting the informational text on operators website and in the dedicated

procedure to double check the orgranizationName text size when operator is filling up the form of certificate request but but this time he did not perform this check. The post-linting procedure officially came into a force on the 1st of January 2019 as a procedural step to check the certificate that was just issued so unfortunately this certificate was not encompassed by the procedure. The OCSP status is 'unknown' because the customer have not picked up the certificate yet.

Anyway, I sent another email to Verizon that we need specific x509lint or zlint feature with fileld length validation so the case is in progress.

Please belive me, we try to practice due care as much as we can but we are just one among many clients of Verizon and we don't have such a clout to

force them to implement new feature in the timeline we propose. I have described in details our need and belive they will deliver the feature as soon as they can.

We have already reinforced and made live our post-linting procedure to check not only certificate that was just issued but check wider range like this

https://crt.sh/?caid=15985&opt=cablint,zlint,x509lint&minNotBefore=2019-01-01

TODO:

- Keep exerting pressure on Verizon to deliver:

o Policy field size validation – in our opinion it is simple change request and should be delivered ASAP.

o native x509lint or zlint feature

Piotr Grabowski
Linia biznesowa podpis elektroniczny
Krajowa Izba Rozliczeniowa S.A.
ul. rtm. W. Pileckiego 65
02-781 Warszawa
Tel. +48 22 545 56 76
Tel. +48 507 024 083

From: Wayne Thayer <wth...@mozilla.com>
Sent: Wednesday, January 09, 2019 9:52 PM
To: Grabowski Piotr <Piotr.G...@kir.pl>
Cc: ry...@sleevi.com; mozilla-dev-security-policy <mozilla-dev-s...@lists.mozilla.org>
Subject: Re: Odp.: Odp.: Odp.: 46 Certificates issued with BR violations (KIR)

KIR recently misissued another (pre-)certificate with an organizationName field containing too many characters [1]. This is despite being given specific guidance earlier in this thread on the organizationName attribute [2]. I have requested a new incident report in the bug [3].

A pre-certificate was logged but the OCSP status is reported as "unknown", so I assume that KIR detected this prior to signing the certificate. If so, I find it particularly troubling that KIR decided not to report this, and that after 3 months they still have no commitment from their vendor to implement pre-issuance linting [4].

- Wayne

[1] https://crt.sh/?id=1036631881&opt=zlint
[2] https://groups.google.com/d/msg/mozilla.dev.security.policy/IRLlR1AJ0vA/daHgd-IXBAAJ
[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1495497
[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1495497#c5

On Fri, Oct 12, 2018 at 8:16 AM Grabowski Piotr <Piotr.G...@kir.pl<mailto:Piotr.G...@kir.pl>> wrote:

Hello,

My comments in blue.

________________________________
Od: Ryan Sleevi <ry...@sleevi.com<mailto:ry...@sleevi.com>>

Wysłane: czwartek, 11 października 2018 04:53
Do: Grabowski Piotr
DW: Wayne Thayer; mozilla-dev-security-policy

Temat: Re: Odp.: Odp.: 46 Certificates issued with BR violations (KIR)

On Wed, Oct 10, 2018 at 4:33 PM Grabowski Piotr via dev-security-policy <dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>> wrote:
Hello Wayne,

- Is the new dual control process documented in a manner that will be auditable by your external auditors?

Yes, the new dual control process is already included in the document called instruction of the security of system Szafir (internal name of the PKI system) and it is one of the document that is presented to internal and external auditors.

Has this been added to your CP/CPS? If not, why not?
-in our CP/CPS we have already the statement that key functions require dual control

Can you please detail the additional controls that were specified?

- Despite the review, is it possible for one malicious employee to modify a policy template by themselves? If not, why not?

It is impossible. CAO role is one of the most trusted role so it has to have physical access to datacenter room, dedicated domain credentials, smartcard (PIN) with certificate to login to CAO application module.

This does not seem to describe why it is impossible. The description of controls here reads that it is possible for a CAO to do so, if malicious, and you merely trust the CAO to not be malicious.
Yes, but you have in mind that the CAO roles are playing only by very big experience and experience (about 20 years) - these persons were building the system, so
we can trust them.

- Have you conducted an overall review of your practices looking for other areas where a human error can result in misissuance? If so, what did you find and how are you addressing it?

Yes, we have conducted an overall review and have not found any other areas where a human error can result in misissuance.

Put differently: Have you completed an examination of the controls in place to ensure that any and all configuration changes that may result in a change to the operation of the CA undergoes multi-party review prior to and following implementation, to ensure consistency with the CP and CPS?

Are there any operations that may modify any state within the CA software, hardware, or operating environment that does not undergo multi-party review prior and following?

If so, what are those operations.
Every other key functions/operations require dual control. Here I mean all staring/ reconfiguration/ upgrade and so on.
If not, what are the operations that you have considered and enumerated as requiring multi-party review prior to and following the modification?

- Why, despite the numerous misissuances documented on this list, has KIR not even begun the process of implementing pre-issuance linting until now?

We have started process of implementing pre-issuance linting just after your email pointing our misissuance. We have requested pre-issuance linting functionality/patch with high priority from Verizon.

This does not answer the question. It states that you have begun the request, but it does not provide insight as to why you had not previously done so.

- Why is KIR not performing post-issuance linting? This problem had been occurring for over a year and there are readily available tools (https://crt.sh) that allow anyone to identify these problems.

We will implement post-issuance linting as well.
We were not aware about this misissuance. We have received the notification about misissuance in October this year and immediately started
fixing the problem. So far no one was reporting to us that there is something wrong with any of our certificates.

This indicates what you will do, but does not answer why you didn't do. Part of the post-mortem process is to understand what issues may have existed, given the readily available nature of the tool and the discussions on m.d.s.p. regarding other CAs.
The same as above. Additionally we have contacted the customers and we are in the proccess of replacement of these certificates.

For example, perhaps the CA did not have adequate staffing to ensure participation in m.d.s.p. Perhaps the CA team did not have adequate training to recognize the similarities and/or value in such.

The expectations upon CAs will continue to increase, and the question is why did KIR S.A. not increase operational oversight in line with those increased expectations, which would have allowed better detection and prevention. It is positive to hear steps are being taken now to address it, but it's reasonable to question why steps weren't taken then, when this was a knowable and identified best practice and minimum expectation of CAs.
KIR is subject to an annual audit of WebTrust compliance and so far the audit has not revealed any irregularities in the management of the CA.

[logo]

Krajowa Izba Rozliczeniowa S.A., zarejestrowana w Sądzie Rejonowym dla m. st. Warszawy,
XIII Wydział Gospodarczy Krajowego Rejestru Sądowego pod nr KRS 0000113064,
NIP 526-030-05-17, REGON 012105474, kapitał zakładowy i wpłacony 5.445.000 zł

Informacja zawarta w tej korespondencji jest przeznaczona tylko dla osoby lub jednostki, do której jest adresowana. Może ona zawierać zastrzeżone i poufne informacje i jeżeli to nie Państwo są wskazanym odbiorcą, nie można kopiować, rozpowszechniać lub podejmować żadnych czynności w oparciu o nią. W przypadku otrzymania tej korespondencji przez pomyłkę, proszę powiadomić nadawcę za pomocą emaila zwrotnego i usunąć tę korespondencję (wraz z załącznikami) z Państwa systemu.

The information contained in this transmission is intended only for the individual or entity to whom it is addressed. It may contain privileged and confidential information and if you are not an indicated recipient, you must not copy, distribute or take any action in reliance on it. If received in error, please notify the sender by return email and delete his transmission (and any attachments) from your system.

[logo]

Ryan Sleevi

unread,

Jan 11, 2019, 12:28:29 PM1/11/19

to Grabowski Piotr, Wayne Thayer, ry...@sleevi.com, mozilla-dev-security-policy

I don't think it's reasonable to push the problem to your CA software
vendor.

If Verizon does not provide this support, what steps will your CA take? If
you know what those steps are, is there reason not to take them now? If you
do not know what those steps are, when will you know?

Your software is producing invalid and improper certificates. The
responsibility for that ultimately falls on the CA, and understanding what
steps the CA is taking to prevent that is critical. It appears that the
steps, today, rely on human factors. As the past year of incident reports
have shown, relying solely on human factors is not a reasonable practice.

On Fri, Jan 11, 2019 at 3:50 AM Grabowski Piotr <Piotr.G...@kir.pl>
wrote:

> Hello Wayne,
>
> Thank you for your email.
>
> I think it is still the same incident. In my opinion there is no need to create another thread. Let me explain how it happened.
>
>
>
> As I said in previous emails we have requested urgent need for pre-linting feature from Verizon just after the incident was reported. I have sent a few reminders
>
> and finally on 18th of December 2018 got a response :
>

> "The feature request of implementing pre-issuance linting was discussed with PM and development today. Our feeling is that UniCERT is policy-centric software and therefore the burden of ensuring that “misissuance” does not happen is on the implementer of the policies. To help with this, we provide registration policy templates in the Registration Policy Wizard and they ensure most of x509lint test. ", but the keyword here is *most.*

>
> We can not for example restrict the length of the field like organizationName to have no more than 64 characters. We can set it as mandatory or not.
>
> It is the only gap we have now in our policy templates. We tried to mitigate this risk putting the informational text on operators website and in the dedicated
>
> procedure to double check the orgranizationName text size when operator is filling up the form of certificate request but but this time he did not perform this check. The post-linting procedure officially came into a force on the 1st of January 2019 as a procedural step to check the certificate that was just issued so unfortunately this certificate was not encompassed by the procedure. The OCSP status is 'unknown' because the customer have not picked up the certificate yet.
>
>
>
> Anyway, I sent another email to Verizon that we need specific x509lint or zlint feature with fileld length validation so the case is in progress.
>
> Please belive me, we try to practice due care as much as we can but we are just one among many clients of Verizon and we don't have such a clout to
>
> force them to implement new feature in the timeline we propose. I have described in details our need and belive they will deliver the feature as soon as they can.
>
>
>
> We have already reinforced and made live our post-linting procedure to check not only certificate that was just issued but check wider range like this
>
> https://crt.sh/?caid=15985&opt=cablint,zlint,x509lint&minNotBefore=2019-01-01
>
>
>
> TODO:
>
> - Keep exerting pressure on Verizon to deliver:
>
> o Policy field size validation – in our opinion it is simple change request and should be delivered ASAP.
>
> o native x509lint or zlint feature
>
>
>
>
>
> Piotr Grabowski
> Linia biznesowa podpis elektroniczny
> Krajowa Izba Rozliczeniowa S.A.
> ul. rtm. W. Pileckiego 65
> 02-781 Warszawa
>
> Tel. +48 22 545 56 76
>
> Tel. +48 507 024 083
>
>
>

> *From:* Wayne Thayer <wth...@mozilla.com>
> *Sent:* Wednesday, January 09, 2019 9:52 PM
> *To:* Grabowski Piotr <Piotr.G...@kir.pl>
> *Cc:* ry...@sleevi.com; mozilla-dev-security-policy <
> mozilla-dev-s...@lists.mozilla.org>
> *Subject:* Re: Odp.: Odp.: Odp.: 46 Certificates issued with BR

> violations (KIR)
>
>
>
> KIR recently misissued another (pre-)certificate with an organizationName
> field containing too many characters [1]. This is despite being given
> specific guidance earlier in this thread on the organizationName attribute
> [2]. I have requested a new incident report in the bug [3].
>
>
>
> A pre-certificate was logged but the OCSP status is reported as "unknown",
> so I assume that KIR detected this prior to signing the certificate. If so,
> I find it particularly troubling that KIR decided not to report this, and
> that after 3 months they still have no commitment from their vendor to
> implement pre-issuance linting [4].
>
>
>
> - Wayne
>
>
>
> [1] https://crt.sh/?id=1036631881&opt=zlint
>
> [2]
> https://groups.google.com/d/msg/mozilla.dev.security.policy/IRLlR1AJ0vA/daHgd-IXBAAJ
>
> [3] https://bugzilla.mozilla.org/show_bug.cgi?id=1495497
>
> [4] https://bugzilla.mozilla.org/show_bug.cgi?id=1495497#c5
>
>
>
> On Fri, Oct 12, 2018 at 8:16 AM Grabowski Piotr <Piotr.G...@kir.pl>

> wrote:
>
> Hello,
>
> My comments in blue.
>
>

> ------------------------------
>
> *Od:* Ryan Sleevi <ry...@sleevi.com>
> *Wysłane:* czwartek, 11 października 2018 04:53
> *Do:* Grabowski Piotr
> *DW:* Wayne Thayer; mozilla-dev-security-policy

> *Temat:* Re: Odp.: Odp.: 46 Certificates issued with BR violations (KIR)

>
>
>
>
>
> On Wed, Oct 10, 2018 at 4:33 PM Grabowski Piotr via dev-security-policy <

> [image: logo]

>
> Krajowa Izba Rozliczeniowa S.A., zarejestrowana w Sądzie Rejonowym dla m.
> st. Warszawy,
> XIII Wydział Gospodarczy Krajowego Rejestru Sądowego pod nr KRS 0000113064,
> NIP 526-030-05-17, REGON 012105474, kapitał zakładowy i wpłacony 5.445.000
> zł
>
>
>
> Informacja zawarta w tej korespondencji jest przeznaczona tylko dla osoby
> lub jednostki, do której jest adresowana. Może ona zawierać zastrzeżone i
> poufne informacje i jeżeli to nie Państwo są wskazanym odbiorcą, nie można
> kopiować, rozpowszechniać lub podejmować żadnych czynności w oparciu o nią.
> W przypadku otrzymania tej korespondencji przez pomyłkę, proszę powiadomić
> nadawcę za pomocą emaila zwrotnego i usunąć tę korespondencję (wraz z
> załącznikami) z Państwa systemu.
>
> The information contained in this transmission is intended only for the
> individual or entity to whom it is addressed. It may contain privileged and
> confidential information and if you are not an indicated recipient, you
> must not copy, distribute or take any action in reliance on it. If received
> in error, please notify the sender by return email and delete his
> transmission (and any attachments) from your system.
>
>

> [image: logo]

Wayne Thayer

unread,

Jan 16, 2019, 6:55:28 PM1/16/19

to Ryan Sleevi, Grabowski Piotr, mozilla-dev-security-policy

Piotr,

I agree with Ryan and am awaiting your response to Ryan's questions. I am
also awaiting an answer to why KIR did not report this misissuance.

- Wayne

Grabowski Piotr

unread,

Jan 17, 2019, 8:23:35 AM1/17/19

to Wayne Thayer, Ryan Sleevi, mozilla-dev-security-policy

Hello Wayne,

I am very sorry for the delay. Please find below our answers to Ryan's questions. Regarding the question why we didn't report this misissuance of this 1 certificate as separate incident in my opinion it is still the same incident. I can create new incident if you want me to. Taking into account my email and Ryan's response I assumed it is not required to create separate incident for misissuance of this 1 certificate.

So our comments in blue:

I don't think it's reasonable to push the problem to your CA software vendor.

- We are not pushing the problem to CA software vendor. I have just tried to explain how it happened.

If Verizon does not provide this support, what steps will your CA take?

- We are almost sure that Verizon will provide at least policy field validation for maximum field size which can be sufficient to eliminate the last gap in our policy templates which in turn led us to misissuance of this certificate. If Verizon will not provide this feature we will be considering usage of another UniCERT component - ARM plug-in - which analyzes requests but this means custom development for us. It would be a big change in many processes and the challange to preserve current security state as well so this should be an absolute last resort.

If you know what those steps are, is there reason not to take them now? If you do not know what those steps are, when will you know?

The main reason why we are not taking these steps now (changing processes and custom development) is absolute conviction that the cost and the risk of implementing them is much, much higher that the risk related with waiting for that feature being delivered by vendor. Just to recall, now we have the only gap which in the worst case can give us specific field in certificate longer than RFC specifies. Of course we are practicing due care and have put as much counter-measures as we can (procedure, labels above the fields).

Your software is producing invalid and improper certificates. The responsibility for that ultimately falls on the CA, and understanding what steps the CA is taking to prevent that is critical. It appears that the steps, today, rely on human factors. As the past year of incident reports have shown, relying solely on human factors is not a reasonable practice.

-I agree entirely with you, that's why we will keep exerting pressure on Verizon to deliver:

o Policy field size validation – in our opinion it is simple change request and should be delivered ASAP.

o native x509lint or zlint feature

Regards ,

Piotr Grabowski
Linia biznesowa podpis elektroniczny
Krajowa Izba Rozliczeniowa S.A.
ul. rtm. W. Pileckiego 65
02-781 Warszawa

Tel. +48 22 545 56 76

Tel. +48 507 024 083

From: Wayne Thayer <wth...@mozilla.com>
Sent: Thursday, January 17, 2019 12:55 AM
To: Ryan Sleevi <ry...@sleevi.com>
Cc: Grabowski Piotr <Piotr.G...@kir.pl>; mozilla-dev-security-policy <mozilla-dev-s...@lists.mozilla.org>
Subject: Re: Odp.: Odp.: Odp.: 46 Certificates issued with BR violations (KIR)

Piotr,

I agree with Ryan and am awaiting your response to Ryan's questions. I am also awaiting an answer to why KIR did not report this misissuance.

- Wayne

On Fri, Jan 11, 2019 at 10:28 AM Ryan Sleevi <ry...@sleevi.com<mailto:ry...@sleevi.com>> wrote:

I don't think it's reasonable to push the problem to your CA software vendor.

If Verizon does not provide this support, what steps will your CA take? If you know what those steps are, is there reason not to take them now? If you do not know what those steps are, when will you know?

Your software is producing invalid and improper certificates. The responsibility for that ultimately falls on the CA, and understanding what steps the CA is taking to prevent that is critical. It appears that the steps, today, rely on human factors. As the past year of incident reports have shown, relying solely on human factors is not a reasonable practice.

On Fri, Jan 11, 2019 at 3:50 AM Grabowski Piotr <Piotr.G...@kir.pl<mailto:Piotr.G...@kir.pl>> wrote:

Hello Wayne,

Thank you for your email.

I think it is still the same incident. In my opinion there is no need to create another thread. Let me explain how it happened.

As I said in previous emails we have requested urgent need for pre-linting feature from Verizon just after the incident was reported. I have sent a few reminders

and finally on 18th of December 2018 got a response :

"The feature request of implementing pre-issuance linting was discussed with PM and development today. Our feeling is that UniCERT is policy-centric software and therefore the burden of ensuring that “misissuance” does not happen is on the implementer of the policies. To help with this, we provide registration policy templates in the Registration Policy Wizard and they ensure most of x509lint test. ", but the keyword here is most.

We can not for example restrict the length of the field like organizationName to have no more than 64 characters. We can set it as mandatory or not.

It is the only gap we have now in our policy templates. We tried to mitigate this risk putting the informational text on operators website and in the dedicated

procedure to double check the orgranizationName text size when operator is filling up the form of certificate request but but this time he did not perform this check. The post-linting procedure officially came into a force on the 1st of January 2019 as a procedural step to check the certificate that was just issued so unfortunately this certificate was not encompassed by the procedure. The OCSP status is 'unknown' because the customer have not picked up the certificate yet.

Anyway, I sent another email to Verizon that we need specific x509lint or zlint feature with fileld length validation so the case is in progress.

Please belive me, we try to practice due care as much as we can but we are just one among many clients of Verizon and we don't have such a clout to

force them to implement new feature in the timeline we propose. I have described in details our need and belive they will deliver the feature as soon as they can.

We have already reinforced and made live our post-linting procedure to check not only certificate that was just issued but check wider range like this

https://crt.sh/?caid=15985&opt=cablint,zlint,x509lint&minNotBefore=2019-01-01

TODO:

- Keep exerting pressure on Verizon to deliver:

o Policy field size validation – in our opinion it is simple change request and should be delivered ASAP.

o native x509lint or zlint feature

Piotr Grabowski
Linia biznesowa podpis elektroniczny
Krajowa Izba Rozliczeniowa S.A.
ul. rtm. W. Pileckiego 65
02-781 Warszawa

Tel. +48 22 545 56 76

Tel. +48 507 024 083

From: Wayne Thayer <wth...@mozilla.com<mailto:wth...@mozilla.com>>
Sent: Wednesday, January 09, 2019 9:52 PM
To: Grabowski Piotr <Piotr.G...@kir.pl<mailto:Piotr.G...@kir.pl>>
Cc: ry...@sleevi.com<mailto:ry...@sleevi.com>; mozilla-dev-security-policy <mozilla-dev-s...@lists.mozilla.org<mailto:mozilla-dev-s...@lists.mozilla.org>>
Subject: Re: Odp.: Odp.: Odp.: 46 Certificates issued with BR violations (KIR)

KIR recently misissued another (pre-)certificate with an organizationName field containing too many characters [1]. This is despite being given specific guidance earlier in this thread on the organizationName attribute [2]. I have requested a new incident report in the bug [3].

A pre-certificate was logged but the OCSP status is reported as "unknown", so I assume that KIR detected this prior to signing the certificate. If so, I find it particularly troubling that KIR decided not to report this, and that after 3 months they still have no commitment from their vendor to implement pre-issuance linting [4].

- Wayne

[1] https://crt.sh/?id=1036631881&opt=zlint

[2] https://groups.google.com/d/msg/mozilla.dev.security.policy/IRLlR1AJ0vA/daHgd-IXBAAJ

[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1495497

[4] https://bugzilla.mozilla.org/show_bug.cgi?id=1495497#c5

On Fri, Oct 12, 2018 at 8:16 AM Grabowski Piotr <Piotr.G...@kir.pl<mailto:Piotr.G...@kir.pl>> wrote:

Hello,

My comments in blue.

________________________________

Od: Ryan Sleevi <ry...@sleevi.com<mailto:ry...@sleevi.com>>
Wysłane: czwartek, 11 października 2018 04:53
Do: Grabowski Piotr
DW: Wayne Thayer; mozilla-dev-security-policy

Temat: Re: Odp.: Odp.: 46 Certificates issued with BR violations (KIR)

On Wed, Oct 10, 2018 at 4:33 PM Grabowski Piotr via dev-security-policy <dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>> wrote:

Hello Wayne,

- Is the new dual control process documented in a manner that will be auditable by your external auditors?

Yes, the new dual control process is already included in the document called instruction of the security of system Szafir (internal name of the PKI system) and it is one of the document that is presented to internal and external auditors.

Has this been added to your CP/CPS? If not, why not?

-in our CP/CPS we have already the statement that key functions require dual control

Can you please detail the additional controls that were specified?

- Despite the review, is it possible for one malicious employee to modify a policy template by themselves? If not, why not?

It is impossible. CAO role is one of the most trusted role so it has to have physical access to datacenter room, dedicated domain credentials, smartcard (PIN) with certificate to login to CAO application module.

This does not seem to describe why it is impossible. The description of controls here reads that it is possible for a CAO to do so, if malicious, and you merely trust the CAO to not be malicious.

Yes, but you have in mind that the CAO roles are playing only by very big experience and experience (about 20 years) - these persons were building the system, so

we can trust them.

- Have you conducted an overall review of your practices looking for other areas where a human error can result in misissuance? If so, what did you find and how are you addressing it?

Yes, we have conducted an overall review and have not found any other areas where a human error can result in misissuance.

Put differently: Have you completed an examination of the controls in place to ensure that any and all configuration changes that may result in a change to the operation of the CA undergoes multi-party review prior to and following implementation, to ensure consistency with the CP and CPS?

Are there any operations that may modify any state within the CA software, hardware, or operating environment that does not undergo multi-party review prior and following?

If so, what are those operations.

Every other key functions/operations require dual control. Here I mean all staring/ reconfiguration/ upgrade and so on.

If not, what are the operations that you have considered and enumerated as requiring multi-party review prior to and following the modification?

- Why, despite the numerous misissuances documented on this list, has KIR not even begun the process of implementing pre-issuance linting until now?

We have started process of implementing pre-issuance linting just after your email pointing our misissuance. We have requested pre-issuance linting functionality/patch with high priority from Verizon.

This does not answer the question. It states that you have begun the request, but it does not provide insight as to why you had not previously done so.

- Why is KIR not performing post-issuance linting? This problem had been occurring for over a year and there are readily available tools (https://crt.sh) that allow anyone to identify these problems.

We will implement post-issuance linting as well.

We were not aware about this misissuance. We have received the notification about misissuance in October this year and immediately started

fixing the problem. So far no one was reporting to us that there is something wrong with any of our certificates.

This indicates what you will do, but does not answer why you didn't do. Part of the post-mortem process is to understand what issues may have existed, given the readily available nature of the tool and the discussions on m.d.s.p. regarding other CAs.

The same as above. Additionally we have contacted the customers and we are in the proccess of replacement of these certificates.

For example, perhaps the CA did not have adequate staffing to ensure participation in m.d.s.p. Perhaps the CA team did not have adequate training to recognize the similarities and/or value in such.

The expectations upon CAs will continue to increase, and the question is why did KIR S.A. not increase operational oversight in line with those increased expectations, which would have allowed better detection and prevention. It is positive to hear steps are being taken now to address it, but it's reasonable to question why steps weren't taken then, when this was a knowable and identified best practice and minimum expectation of CAs.

KIR is subject to an annual audit of WebTrust compliance and so far the audit has not revealed any irregularities in the management of the CA.

[logo]

Krajowa Izba Rozliczeniowa S.A., zarejestrowana w Sądzie Rejonowym dla m. st. Warszawy,
XIII Wydział Gospodarczy Krajowego Rejestru Sądowego pod nr KRS 0000113064,
NIP 526-030-05-17, REGON 012105474, kapitał zakładowy i wpłacony 5.445.000 zł

Informacja zawarta w tej korespondencji jest przeznaczona tylko dla osoby lub jednostki, do której jest adresowana. Może ona zawierać zastrzeżone i poufne informacje i jeżeli to nie Państwo są wskazanym odbiorcą, nie można kopiować, rozpowszechniać lub podejmować żadnych czynności w oparciu o nią. W przypadku otrzymania tej korespondencji przez pomyłkę, proszę powiadomić nadawcę za pomocą emaila zwrotnego i usunąć tę korespondencję (wraz z załącznikami) z Państwa systemu.

The information contained in this transmission is intended only for the individual or entity to whom it is addressed. It may contain privileged and confidential information and if you are not an indicated recipient, you must not copy, distribute or take any action in reliance on it. If received in error, please notify the sender by return email and delete his transmission (and any attachments) from your system.

[logo]

Krajowa Izba Rozliczeniowa S.A., zarejestrowana w Sądzie Rejonowym dla m. st. Warszawy,
XIII Wydział Gospodarczy Krajowego Rejestru Sądowego pod nr KRS 0000113064,
NIP 526-030-05-17, REGON 012105474, kapitał zakładowy i wpłacony 5.445.000 zł

Informacja zawarta w tej korespondencji jest przeznaczona tylko dla osoby lub jednostki, do której jest adresowana. Może ona zawierać zastrzeżone i poufne informacje i jeżeli to nie Państwo są wskazanym odbiorcą, nie można kopiować, rozpowszechniać lub podejmować żadnych czynności w oparciu o nią. W przypadku otrzymania tej korespondencji przez pomyłkę, proszę powiadomić nadawcę za pomocą emaila zwrotnego i usunąć tę korespondencję (wraz z załącznikami) z Państwa systemu.

The information contained in this transmission is intended only for the individual or entity to whom it is addressed. It may contain privileged and confidential information and if you are not an indicated recipient, you must not copy, distribute or take any action in reliance on it. If received in error, please notify the sender by return email and delete his transmission (and any attachments) from your system.

[logo]

Wayne Thayer

unread,

Jan 17, 2019, 3:12:58 PM1/17/19

> I have response from Verizon that they will send us a patch containing some
> pre-issuance linting features on 03/02. By the way I kindly inform you
> that I’m out of the office from 28/01 to 07/02 and will be responding to
> my emails when I return on 11/02.
>
> Regards
> Piotr Grabowski
>

> > > Regards ,
> > >
> > >
> > > Piotr Grabowski
> > > Linia biznesowa podpis elektroniczny
> > > Krajowa Izba Rozliczeniowa S.A.
> > > ul. rtm. W. Pileckiego 65
> > > 02-781 Warszawa
> > >
> > > Tel. +48 22 545 56 76
> > >
> > > Tel. +48 507 024 083
> > >
> > >
> > >
>

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

Kurt Roeckx

unread,

Jan 29, 2019, 4:10:49 AM1/29/19

to mozilla-dev-s...@lists.mozilla.org

On 2019-01-29 1:29, Wayne Thayer wrote:
> Piotr just filed an incident report on the misissuance that was reported on
> 18-January: https://bugzilla.mozilla.org/show_bug.cgi?id=1523186

I guess this part is not very clear to me:

> We identified and removed from system the registration policy that
> issued the problematic certificate. The problematic policy template
> was not listed in policies allowed for Certificate Transparency
> logging but contained Signed Certificate Timestamp extension. The
> usage of such policy template should be blocked by the CT
> functionality. We had only one policy in such state.

I could read that as:
1) This certificate was not supposed to be logged in CT
2) The issuing should have been prevented

I assume 2) was meant.

Kurt

Wayne Thayer

unread,

Feb 1, 2019, 5:03:07 PM2/1/19

to Kurt Roeckx, mozilla-dev-security-policy

It was pointed out to me that the OCSP status of the misissued certificate
that is valid for over 5 years is still "unknown" despite having been
revoked a week ago. I asked KIR about this in the bug [1] and am surprised
by their response:

This certificate is revoked on CRL. Because the certificate has been never
> received by the customer its status on OCSP is "unknown". To make the
> certificate "revoked" on OCSP first we should make it "valid" what makes no
> sense. I know there is inconsistency between CRL and OCSP but there are
> some scenarios when it can be insecure to make it valid just in order to
> make it revoked.
>

Upon further questioning KIR states:

Of course I can mark it as revoked after I make it valid, but I think it is
> more secure practice not to change its status at all when the certificate
> is not received by the customer. Let's suppose the scenario when your CA
> generate certificate and the customer wants you to deliver it to its
> office. What OCSP status the certificate should have when you are on your
> way to the customer office? valid - I do not think so. When the certificate
> is stolen you are in trouble. So the only option is "unknown" but then we
> have different statuses on CRL and OCSP - but we are still safe. It is not
> only my opinion, we had a big discuss with our auditors about that.
>

Does anyone other then KIR and their auditor (Ernst & Young) think this is
currently permitted? At the very least, I believe that returning "unknown"
for a revoked certificate is misleading to Firefox users who will receive
the "SEC_ERROR_OCSP_UNKNOWN_CERT" error instead of
"SEC_ERROR_REVOKED_CERTIFICATE".

Does anyone other than KIR and Ernst & Young believe that this meets
WebTrust for CAs control 6.8.12? [2]

- Wayne

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1523186
[2] http://www.webtrust.org/principles-and-criteria/docs/item85228.pdf

On Tue, Jan 29, 2019 at 2:10 AM Kurt Roeckx via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On 2019-01-29 1:29, Wayne Thayer wrote:

> > Piotr just filed an incident report on the misissuance that was reported
> on
> > 18-January: https://bugzilla.mozilla.org/show_bug.cgi?id=1523186
>

> I guess this part is not very clear to me:
>
> > We identified and removed from system the registration policy that
> > issued the problematic certificate. The problematic policy template
> > was not listed in policies allowed for Certificate Transparency
> > logging but contained Signed Certificate Timestamp extension. The
> > usage of such policy template should be blocked by the CT
> > functionality. We had only one policy in such state.
>
> I could read that as:
> 1) This certificate was not supposed to be logged in CT
> 2) The issuing should have been prevented
>
> I assume 2) was meant.
>
>
> Kurt

Kurt Roeckx

unread,

Feb 1, 2019, 5:38:40 PM2/1/19

to Wayne Thayer, mozilla-dev-security-policy

If you follow the RFC, the "unknown" answer can mean that it
doesn't know, and that an other option like a CRL can be tried.
With "unknown", it doesn't say anything about being valid or not.

I don't think that interpretation is very useful. I think that the
OCSP server should know about the certificate before the customer
has the certificate. I think that if you have a properly signed
certificate within it's validity period, the OCSP should always
return either "good" or "revoked", never "unknown". Once a
certificate is generated and it's not revoked it's valid.

Would it be useful to have a requirement in the BRs that the OCSP
server should not answer with "unknown" for an issued certificate
within it's validity period?

Kurt

Buschart, Rufus

unread,

Feb 2, 2019, 4:31:51 AM2/2/19

to Kurt Roeckx, Wayne Thayer, mozilla-dev-security-policy

Personally I think it would be better, if the revoke reason "Certificate hold" on the CRL would be allowed for TLS certificates, as this state would exactly cover the described scenario. The OCSP responder could in such a case reply with "bad" and deliver the reason "certificate hold". But I fully understand that browser developers had a lot of issues with this state, so it is still forbidden.

With best regards,
Rufus Buschart

Siemens AG
Information Technology
Human Resources
PKI / Trustcenter
GS IT HR 7 4
Hugo-Junkers-Str. 9
90411 Nuernberg, Germany
Tel.: +49 1522 2894134
mailto:rufus.b...@siemens.com
www.twitter.com/siemens

www.siemens.com/ingenuityforlife

Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Jim Hagemann Snabe; Managing Board: Joe Kaeser, Chairman, President and Chief Executive Officer; Roland Busch, Lisa Davis, Klaus Helmrich, Janina Kugel, Cedrik Neike, Michael Sen, Ralf P. Thomas; Registered offices: Berlin and Munich, Germany; Commercial registries: Berlin Charlottenburg, HRB 12300, Munich, HRB 6684; WEEE-Reg.-No. DE 23691322

> -----Ursprüngliche Nachricht-----
> Von: dev-security-policy <dev-security-...@lists.mozilla.org> Im Auftrag von Kurt Roeckx via dev-security-policy
> Gesendet: Freitag, 1. Februar 2019 23:38
> An: Wayne Thayer <wth...@mozilla.com>
> Cc: mozilla-dev-security-policy <mozilla-dev-s...@lists.mozilla.org>
> Betreff: Re: Odp.: Odp.: Odp.: 46 Certificates issued with BR violations (KIR)

Eric Mill

unread,

Feb 2, 2019, 9:16:46 AM2/2/19

to Buschart, Rufus, Kurt Roeckx, Wayne Thayer, mozilla-dev-security-policy

The BRs and Mozilla program policies don't support the idea of just
trusting a CA to issue certs for "internal" use or to keep them secret.
This is why CAs issuing "test certificates" on production CAs for domains
they don't own is clearly forbidden.

Given that, I don't see how it can be acceptable to provide an "unknown"
status over OCSP for a revoked certificate, on the premise that the CA
asserts they never actually shipped the cert to a customer.

The fact that they would have to mark the cert "valid" before marking it
"revoked" is a limitation of the implementation of the OCSP responder. It's
not a reason to ignore policy that is grounded in the very reasonable
desire to ensure that the certificate's revoked status is known to any
client which checks OCSP instead of CRL.

-- Eric

> > _______________________________________________
> > dev-security-policy mailing list
> > dev-secur...@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-security-policy
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

--
Eric Mill
617-314-0966 | konklone.com | @konklone <https://twitter.com/konklone>

ji...@it.auth.gr

unread,

Feb 2, 2019, 2:13:20 PM2/2/19

to Buschart, Rufus, Eric Mill, mozilla-dev-security-policy, Kurt Roeckx, Wayne Thayer

+1. Of course there must be consistency between CRLs and OCSP.

Dimitris.

bif

unread,

Feb 3, 2019, 4:26:08 PM2/3/19

to mozilla-dev-s...@lists.mozilla.org

On Friday, February 1, 2019 at 11:38:40 PM UTC+1, Kurt Roeckx wrote:
> On Fri, Feb 01, 2019 at 03:02:17PM -0700, Wayne Thayer wrote:
> > It was pointed out to me that the OCSP status of the misissued certificate
> > that is valid for over 5 years is still "unknown" despite having been

> > revoked a week ago.ntrol 6.8.12? [2]

>
> If you follow the RFC, the "unknown" answer can mean that it
> doesn't know, and that an other option like a CRL can be tried.
> With "unknown", it doesn't say anything about being valid or not.
>
> I don't think that interpretation is very useful. I think that the
> OCSP server should know about the certificate before the customer
> has the certificate.

FWIW, with ACME and automated instant certificates this may be an interesting challenge for big CAs. While you can design to try to achieve this, there will always be a case of some update not getting through in time, and some members of the high availability OCSP responders pool not having 100% of issued certificates from the last minute (obviously the longer the time from issuance, the sharply lower probability of such an event should be, and after a day it is unwise to not have all answers).

Wayne Thayer

unread,

Feb 4, 2019, 2:21:35 PM2/4/19

to bif, mozilla-dev-security-policy

While a certain amount of latency in OCSP updates is expected when a
certificate is first issued or revoked, KIR intended this to be a permanent
"unknown" status for a revoked certificate. My conclusion from this
discussion is that such a policy is not permitted, and the existing
requirements are enough to make that clear.

I created another bug to track the issue with KIR's auditor, Ernst & Young
Poland: https://bugzilla.mozilla.org/show_bug.cgi?id=1525082

KIR has commented in the original bug, and I have confirmed that the
certificate is now revoked via OCSP as well as their CRL.

On Sun, Feb 3, 2019 at 2:26 PM bif via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On Friday, February 1, 2019 at 11:38:40 PM UTC+1, Kurt Roeckx wrote:

> > On Fri, Feb 01, 2019 at 03:02:17PM -0700, Wayne Thayer wrote:
> > > It was pointed out to me that the OCSP status of the misissued
> certificate
> > > that is valid for over 5 years is still "unknown" despite having been

> > > revoked a week ago.ntrol 6.8.12? [2]

> >
> > If you follow the RFC, the "unknown" answer can mean that it
> > doesn't know, and that an other option like a CRL can be tried.
> > With "unknown", it doesn't say anything about being valid or not.
> >
> > I don't think that interpretation is very useful. I think that the
> > OCSP server should know about the certificate before the customer
> > has the certificate.
>

> FWIW, with ACME and automated instant certificates this may be an
> interesting challenge for big CAs. While you can design to try to achieve
> this, there will always be a case of some update not getting through in
> time, and some members of the high availability OCSP responders pool not
> having 100% of issued certificates from the last minute (obviously the
> longer the time from issuance, the sharply lower probability of such an
> event should be, and after a day it is unwise to not have all answers).
>
>

> > I think that if you have a properly signed
> > certificate within it's validity period, the OCSP should always
> > return either "good" or "revoked", never "unknown". Once a
> > certificate is generated and it's not revoked it's valid.
> >
> > Would it be useful to have a requirement in the BRs that the OCSP
> > server should not answer with "unknown" for an issued certificate
> > within it's validity period?
> >
> >
> > Kurt
>